{"id":352723,"date":"2021-09-29T07:51:25","date_gmt":"2021-09-29T14:51:25","guid":{"rendered":"https:\/\/css-tricks.com\/?p=352723"},"modified":"2021-09-29T07:51:27","modified_gmt":"2021-09-29T14:51:27","slug":"web-streams-everywhere-and-fetch-for-node-js","status":"publish","type":"post","link":"https:\/\/css-tricks.com\/web-streams-everywhere-and-fetch-for-node-js\/","title":{"rendered":"Web Streams Everywhere (and Fetch for Node.js)"},"content":{"rendered":"\n

Chrome developer advocate Jake Archibald called 2016 “the year of web streams<\/a>.” Clearly, his prediction was somewhat premature. The Streams Standard was announced<\/a> back in 2014. It\u2019s taken a while, but there\u2019s now a consistent streaming API implemented in modern browsers<\/a> (still waiting on Firefox\u2026) and in Node (and Deno).<\/p>\n\n\n\n\n\n\n

What are streams?<\/h3>\n\n\n

Streaming involves splitting a resource into smaller pieces called chunks and processing each chunk one at a time. Rather than needing to wait to complete the download of all the data, with streams you can process data progressively as soon as the first chunk is available.<\/p>\n\n\n\n

There are three kinds of streams: readable streams, writable streams, and transform streams. Readable streams<\/strong><\/dfn> are where the chunks of data come from. The underlying data sources could be a file or HTTP connection, for example. The data can then (optionally) be modified by a transform stream<\/strong>. The chunks of data can then be piped to a writable stream<\/strong>.<\/p>\n\n\n

Web streams everywhere<\/h3>\n\n\n

Node has always had it\u2019s own type of streams. They are generally considered to be difficult to work with<\/a>. The Web Hypertext Application Technology Working Group (WHATWG) web standard for streams came later, and are largely considered an improvement. The Node docs calls them \u201cweb streams” which sounds a bit less cumbersome. The original Node streams aren\u2019t being deprecated or removed but they will now co-exist with the web standard stream API. This makes it easier to write cross-platform code and means developers only need to learn one way of doing things.<\/p>\n\n\n\n

Deno<\/a>, another attempt at server-side JavaScript by Node\u2019s original creator, has always closely aligned with browser APIs and has full support for web streams. Cloudflare workers<\/a> (which are a bit like service workers but running on CDN edge locations) and Deno Deploy<\/a> (a serverless offering from Deno) also support streams. <\/a><\/p>\n\n\n

fetch()<\/code> response as a readable stream<\/h3>\n\n\n

There are multiple ways to create a readable stream, but calling fetch()<\/code> is bound to be the most common. The response body of fetch()<\/code> is a readable stream.<\/p>\n\n\n\n

fetch('data.txt')\n.then(response => console.log(response.body));<\/code><\/pre>\n\n\n\n
If you look at the console log you can see that a readable stream has several useful methods. As the spec says<\/a>, A readable stream can be piped directly to a writable stream, using its pipeTo()<\/code> method, or it can be piped through one or more transform streams first, using its pipeThrough()<\/code> method.<\/q><\/p>\n\n\n\n
Unlike browsers, Node core doesn\u2019t currently implement fetch. node-fetch<\/a>, a popular dependency that tries to match the API of the browser standard, returns a node stream<\/a>, not a WHATWG stream. Undici<\/a>, an improved HTTP\/1.1 client from the Node.js team, is a modern alternative to the Node.js core<\/a> http.request<\/code><\/a> (which things like node-fetch and Axios<\/a> are built on top of). Undici has implemented fetch<\/code> \u2014 and response.body<\/code> does return a web stream<\/em>. 🎉<\/p>\n\n\n\n
Undici might end up in Node.js core eventually, and it looks set to become the recommended way to handle HTTP requests in Node<\/a>. Once you npm install<\/code> undici and import fetch<\/code>, it works the same as in the browser. In the following example, we pipe the stream through a transform stream. Each chunk of the stream is a Uint8Array<\/code>. Node core provides a TextDecoderStream<\/code> to decode binary data.<\/p>\n\n\n\n
import { fetch } from 'undici';\nimport { TextDecoderStream } from 'node:stream\/web';\n\nasync function fetchStream() {\n  const response = await fetch('https:\/\/example.com')\n  const stream = response.body;\n  const textStream = stream.pipeThrough(new TextDecoderStream());\n}<\/code><\/pre>\n\n\n\n
response.body<\/code> is synchronous so you don\u2019t need to await<\/code> it. In the browser, fetch<\/code> and TextDecoderStream<\/code> are available on the global object so you wouldn\u2019t include any import statements. Other than that, the code is exactly the same for Node and web browsers. Deno also has built-in support for fetch<\/code><\/a> and TextDecoderStream<\/code><\/a>.<\/p>\n\n\n
Async iteration<\/h3>\n\n\n
The for-await-of loop is an asynchronous version of the for-of loop. A regular for-of loop is used to loop over arrays and other iterables. A for-await-of loop can be used to iterate over an array of promises, for example.<\/p>\n\n\n\n
const promiseArray = [Promise.resolve(\"thing 1\"), Promise.resolve(\"thing 2\")];\nfor await (const thing of promiseArray) { console.log(thing); }<\/code><\/pre>\n\n\n\n
Importantly for us, this can also be used to iterate streams.<\/p>\n\n\n\n
async function fetchAndLogStream() {\n  const response = await fetch('https:\/\/example.com')\n  const stream = response.body;\n  const textStream = stream.pipeThrough(new TextDecoderStream());\n\n  for await (const chunk of textStream) {\n    console.log(chunk);\n  }\n}\n\nfetchAndLogStream();<\/code><\/pre>\n\n\n\n
Async iteration of streams works in Node and Deno. All modern browsers have shipped for-await-of loops<\/a> but they don\u2019t work on streams just yet.<\/p>\n\n\n
Some other ways to get a readable stream<\/h3>\n\n\n
Fetch will be one of the most common ways to get hold of a stream, but there are other ways. Blob<\/code> and File<\/code> both have a .stream()<\/code> method that returns a readable stream. The following code works in modern browsers as well as in Node<\/a> and in Deno \u2014 although, in Node, you will need to import { Blob } from 'buffer';<\/code> before you can use it:<\/p>\n\n\n\n
const blobStream = new Blob(['Lorem ipsum'], { type: 'text\/plain' }).stream();<\/code><\/pre>\n\n\n\n
Here is a front-end browser-based example: If you have a <input type=\"file\"><\/code> in your markup, it\u2019s easy to get the user-selected file as a stream.<\/p>\n\n\n\n
const fileStream = document.querySelector('input').files[0].stream();<\/code><\/pre>\n\n\n\n
Shipping in Node 17, the FileHandle object returned by the fs\/promises open()<\/code> function has a .readableWebStream()<\/code> method.<\/p>\n\n\n\n
import {\n  open,\n} from 'node:fs\/promises';\n\nconst file = await open('.\/some\/file\/to\/read');\n\nfor await (const chunk of file.readableWebStream())\n  console.log(chunk);\n\nawait file.close();<\/code><\/pre>\n\n\n
Streams work nicely with promises<\/h3>\n\n\n
If you need to do something after the stream has completed, you can use promises.<\/p>\n\n\n\n
someReadableStream\n.pipeTo(someWritableStream)\n.then(() => console.log(\"all data successfully written\"))\n.catch(error => console.error(\"something went wrong\", error))<\/code><\/pre>\n\n\n\n
Or, you can optionally await the result:<\/p>\n\n\n\n
await someReadableStream.pipeTo(someWritableStream)<\/code><\/pre>\n\n\n
Creating your own transform stream<\/h3>\n\n\n
We already saw TextDecoderStream<\/code> (there\u2019s also a TextEncoderStream<\/code>). You can also create your own transform stream from scratch. The TransformStream<\/code> constructor can accept an object. You can specify three methods in the object: start<\/code>, transform<\/code> and flush<\/code>. They\u2019re all optional, but transform<\/code> is what actually does the transformation.<\/p>\n\n\n\n
As an example, let\u2019s pretend that TextDecoderStream()<\/code> doesn\u2019t exist and implement the same functionality (be sure to use TextDecoderStream<\/code> in production though as the following is an over-simplified example):<\/p>\n\n\n\n
const decoder = new TextDecoder();\nconst decodeStream = new TransformStream({\n  transform(chunk, controller) {\n    controller.enqueue(decoder.decode(chunk, {stream: true}));\n  }\n});<\/code><\/pre>\n\n\n\n
Each received chunk is modified and then forwarded on by the controller. In the above example, each chunk is some encoded text that gets decoded and then forwarded. Let\u2019s take a quick look at the other two methods:<\/p>\n\n\n\n
const transformStream = new TransformStream({\n  start(controller) {\n    \/\/ Called immediately when the TransformStream is created\n  },\n\n  flush(controller) {\n    \/\/ Called when chunks are no longer being forwarded to the transformer\n  }\n});<\/code><\/pre>\n\n\n\n
A transform stream is a readable stream and a writable stream working together, usually to transform some data. Every object made with new TransformStream()<\/code> has a property called readable<\/code>, which is a ReadableStream<\/code>, and a property called writable<\/code>, which is a writable stream. Calling someReadableStream.pipeThrough()<\/code> writes the data from someReadableStream<\/code> to transformStream.writable<\/code>, possibly transforms the data, then pushes the data to transformStream.readable<\/code>.<\/p>\n\n\n\n
Some people find it helpful to create a transform stream that doesn\u2019t actually transform data. This is known as an \u201cidentity transform stream” \u2014 created by calling new TransformStream()<\/code> without passing in any object argument, or by leaving off the transform method. It forwards all chunks written to its writable side to its readable side, without any changes. As a simple example of the concept, \u201chello” is logged by the following code:<\/p>\n\n\n\n
const {readable, writable} = new TransformStream();\nwritable.getWriter().write('hello');\nreadable.getReader().read().then(({value, done}) => console.log(value))<\/code><\/pre>\n\n\n
Creating your own readable stream<\/h3>\n\n\n
It\u2019s possible to create a custom stream and populate it with your own chunks. The new ReadableStream()<\/code> constructor takes an object that can contain a start<\/code> function, a pull<\/code> function, and a cancel<\/code> function. This function is invoked immediately when the ReadableStream<\/code> is created. Inside the start<\/code> function, use controller.enqueue<\/code> to add chunks to the stream.<\/p>\n\n\n\n
Here\u2019s a basic \u201chello world” example:<\/p>\n\n\n\n
import { ReadableStream } from \"node:stream\/web\";\nconst readable = new ReadableStream({\n  start(controller) {\n    controller.enqueue(\"hello\");\n    controller.enqueue(\"world\");\n    controller.close();\n  },\n});\n\nconst allChunks = [];\nfor await (const chunk of readable) {\n  allChunks.push(chunk);\n}\nconsole.log(allChunks.join(\" \"));<\/code><\/pre>\n\n\n\n
Here\u2019s an more real-world example taken from the streams specification<\/a> that turns a web socket into a readable stream:<\/p>\n\n\n\n
function makeReadableWebSocketStream(url, protocols) {\n  let websocket = new WebSocket(url, protocols);\n  websocket.binaryType = \"arraybuffer\";\n\n  return new ReadableStream({\n    start(controller) {\n      websocket.onmessage = event => controller.enqueue(event.data);\n      websocket.onclose = () => controller.close();\n      websocket.onerror = () => controller.error(new Error(\"The WebSocket errored\"));\n    }\n  });\n}<\/code><\/pre>\n\n\n
Node streams interoperability<\/h3>\n\n\n
In Node, the old Node-specific way of working with streams isn\u2019t being removed. The old node streams API and the web streams API will coexist. It might therefore sometimes be necessary to turn a Node stream into a web stream, and vice versa, using .fromWeb()<\/code> and .toWeb()<\/code> methods, which are being added in Node 17.<\/p>\n\n\n\n
import {Readable} from 'node:stream';\nimport {fetch} from 'undici';\n\nconst response = await fetch(url);\nconst readableNodeStream = Readable.fromWeb(response.body);<\/code><\/pre>\n\n\n
Conclusion<\/h3>\n\n\n
ES modules, EventTarget<\/code>, AbortController<\/code>, URL parser<\/a>, Web Crypto<\/a>, Blob<\/code>, TextEncoder\/Decoder<\/code>: increasingly more browser APIs are ending up in Node.js. The knowledge and skills are transferable. Fetch and streams are an important part of that convergence.<\/p>\n\n\n\n
Domenic Denicola<\/a>, a co-author of the streams spec, has written<\/a> that the goal of the streams API is to provide an efficient abstraction and unifying primitive<\/em> for I\/O, like promises have become for asynchronicity. To become truly useful on the front end, more APIs need to actually support streams. At the moment a MediaStream, despite its name, is not a readable stream. If you’re working with video or audio (at least at the moment), a readable stream can\u2019t be assigned to srcObject<\/code><\/a>. Or let\u2019s say you want to get an image and pass it through a transform stream, then insert it onto the page. At the time of writing, the code for using a stream as the src of an image element is somewhat verbose:<\/p>\n\n\n\n
const response = await fetch('cute-cat.png');\nconst bodyStream = response.body;\nconst newResponse = new Response(bodyStream);\nconst blob = await newResponse.blob();\nconst url = URL.createObjectURL(blob);\ndocument.querySelector('img').src = url;    <\/code><\/pre>\n\n\n\n