Skip to main content
Dublin Library

The Publishing Project

JavaScript and TypeScript Streams (2026 Edition)

 

Streams are the safest default when data can be large, continuous, or unpredictable. Instead of buffering everything in memory first, streams process data chunk by chunk.

The goal is not to memorize every stream API detail. It is to build a reliable mental model and use a small set of battle-tested patterns in production.

This post focuses on:

  • Choosing between Web Streams and Node streams
  • Bridging both APIs in Node.js
  • Applying backpressure correctly
  • Building transform pipelines for common tasks
  • Streaming with fetch()
  • Handling cancellation and failures

Who This Is For #

If you already use async/await, work with APIs/files, and write TypeScript regularly, you are the intended audience.

If you need deep protocol recovery strategies, browser internals, or experimental stream proposals, treat those as advanced follow-up topics.

Node.js Streams vs. Web Streams API #

Today, both models are relevant:

  • Web Streams API is the cross-runtime standard (browser, Deno, Workers, modern Node).
  • Node streams remain common in existing Node libraries and filesystem tooling.

A practical rule:

  • Prefer Web Streams for new isomorphic code and browser-aligned pipelines.
  • Use Node streams when integrating with existing Node APIs that already speak node:stream.

Interop Essentials: Node <-> Web #

In modern Node, use built-in adapters instead of custom wrappers.

Node Readable -> Web ReadableStream #

Use this when legacy Node sources (filesystem streams, older libraries) need to feed modern Web Stream-based pipelines. Think of it as a compatibility adapter at the edge of your system.

import { Readable } from 'node:stream';
import { createReadStream } from 'node:fs';

function nodeToWeb(filePath: string): ReadableStream<Uint8Array> {
  const nodeReadable = createReadStream(filePath);
  return Readable.toWeb(nodeReadable) as ReadableStream<Uint8Array>;
}

After conversion, keep the rest of the flow in Web Streams rather than switching back and forth. Repeated conversion inside one pipeline adds both mental and runtime overhead.

Web ReadableStream -> Node Readable #

This is the opposite boundary: you receive a Web stream, but downstream tooling expects classic Node streams.

import { Readable } from 'node:stream';

function webToNode(webReadable: ReadableStream<Uint8Array>): Readable {
  return Readable.fromWeb(webReadable);
}

In practice, this shows up when using fetch() in Node and then handing results to existing Node-based compressors, parsers, or file writers.

Bridge fetch() to Node pipeline() #

This pattern combines modern source semantics (fetch) with mature Node teardown behavior (pipeline). It is a strong default for download-and-save tasks.

import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { createWriteStream } from 'node:fs';

async function downloadViaPipeline(url: string, dest: string): Promise<void> {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Request failed (${response.status} ${response.statusText})`);
  }
  if (!response.body) {
    throw new Error('Response body is null.');
  }

  await pipeline(
    Readable.fromWeb(response.body),
    createWriteStream(dest)
  );
}

The key point is not just interoperability. It is safety: pipeline() centralizes failure handling so partial writes and hung streams are less likely.

With interop covered, the next step is data shaping, which is where transform streams do most of the practical work.

Transform Streams in Day-to-Day Work #

Transform streams are where streaming becomes truly useful. Most production pipelines are not just "read then write"; they decode, filter, normalize, and re-encode data in flight.

Use transforms when you need to:

  • Process NDJSON or line-based logs incrementally
  • Redact sensitive fields before persisting data
  • Convert text encodings or compression formats

This browser-compatible example converts a byte stream into line-delimited text and handles partial lines safely. Notice the buffered variable in the transform: it captures incomplete trailing data from one chunk and prepends it to the next. Without that, line-based parsing breaks whenever a newline boundary splits across chunks.

function createLineSplitter(): TransformStream<string, string> {
  let buffered = '';

  return new TransformStream<string, string>({
    transform(chunk, controller) {
      buffered += chunk;
      const lines = buffered.split('\n');
      buffered = lines.pop() ?? '';

      for (const line of lines) {
        controller.enqueue(line);
      }
    },
    flush(controller) {
      if (buffered) {
        controller.enqueue(buffered);
      }
    }
  });
}

async function streamLines(url: string): Promise<void> {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Request failed (${response.status} ${response.statusText})`);
  }
  if (!response.body) {
    throw new Error('Response body is null.');
  }

  const textStream = response.body.pipeThrough(new TextDecoderStream());
  const lineStream = textStream.pipeThrough(createLineSplitter());

  for await (const line of lineStream) {
    console.log(`Line: ${line}`);
  }
}

This example is intentionally different from write-to-file flows. In a file pipeline, you usually terminate the stream with a writable destination (pipeTo() in Web Streams, or pipeline() in Node). Here, the terminal sink is the for await...of loop itself. That means each parsed line is handled immediately in application code instead of being forwarded to a file writer. Use this shape when your destination is business logic (validation, filtering, metrics, UI updates) rather than persistent storage.

This line-splitting pattern generalizes well: replace controller.enqueue(line) with your own record parser, validation step, or redaction logic.

Once transforms are in place, backpressure is the next concept that keeps these pipelines stable under load.

Backpressure: The One Concept You Must Get Right #

Backpressure means the producer must slow down when the consumer is full.

Web Streams pattern (writer.ready) #

For intermediate developers, this is the most important writable pattern in the Web Streams API. writer.ready is your pacing signal from downstream.

async function writeChunksSafely(
  writable: WritableStream<Uint8Array>,
  chunks: AsyncIterable<Uint8Array>
): Promise<void> {
  const writer = writable.getWriter();

  try {
    for await (const chunk of chunks) {
      await writer.ready;
      await writer.write(chunk);
    }
    await writer.close();
  } finally {
    writer.releaseLock();
  }
}

This helper is only the writable side of the flow. It does not create a reader because chunks is passed in as an AsyncIterable<Uint8Array> source. In other words, the reader (or generator) lives upstream and this function just applies backpressure-aware writing.

Upstream can be any source that yields Uint8Array chunks as an AsyncIterable (for example, a manual reader loop, an async generator around a transform pipeline, or an experimental iterable stream API). The key point is that backpressure handling is centralized in one place, so you do not need to sprinkle await writer.ready throughout your codebase.

A generator function might look like this:

async function* chunksFromFetchBody(url: string): AsyncIterable<Uint8Array> {
  const response = await fetch(url);
  if (!response.ok || !response.body) {
    throw new Error(`Request failed (${response.status} ${response.statusText})`);
  }

  const reader = response.body.getReader();
  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      yield value;
    }
  } finally {
    reader.releaseLock();
  }
}

await writeChunksSafely(destinationWritable, chunksFromFetchBody(url));

If you skip await writer.ready, fast producers can overwhelm slow destinations and push memory usage much higher than expected.

Node streams pattern (write() + drain) #

Node exposes the same concept with a different shape. A false return from write() means "pause and wait for drain before continuing."

import { once } from 'node:events';
import { createWriteStream } from 'node:fs';

async function writeNodeSafely(data: AsyncIterable<Uint8Array>, filePath: string): Promise<void> {
  const writable = createWriteStream(filePath, { highWaterMark: 64 * 1024 });

  for await (const chunk of data) {
    if (!writable.write(chunk)) {
      await once(writable, 'drain');
    }
  }

  writable.end();
  await once(writable, 'finish');
}

When teams report "streaming still used too much memory," this missing drain handling is one of the first places to investigate.

With flow control established, we can move to network streaming, where these same ideas appear in a slightly different form.

Streaming with fetch() #

response.body is a ReadableStream, so you can process bytes as they arrive instead of waiting for a full payload.

The next example uses a manual reader because it makes progress accounting explicit and easy to instrument.

async function fetchWithProgress(url: string): Promise<void> {
  const response = await fetch(url);

  if (!response.ok) {
    const errorBody = await response.text().catch(() => '');
    const details = errorBody ? `: ${errorBody.slice(0, 200)}` : '';
    throw new Error(
      `Request failed (${response.status} ${response.statusText})${details}`
    );
  }

  if (!response.body) {
    throw new Error('Response body is null.');
  }

  const totalBytes = Number(response.headers.get('content-length')) || 0;
  let receivedBytes = 0;
  const reader = response.body.getReader();

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      receivedBytes += value.length;

      if (totalBytes) {
        const percentage = Math.round((receivedBytes / totalBytes) * 100);
        console.log(`Progress: ${percentage}%`);
      } else {
        console.log(`Received: ${receivedBytes} bytes`);
      }
    }
  } finally {
    reader.releaseLock();
  }
}

This approach is especially useful for dashboards or UI flows where users expect visibility into long-running downloads.

A More Realistic fetch Flow #

In many apps, you want all of the following in one place: status checks, cancellation, progress, and safe cleanup.

The following helper is intentionally application-shaped rather than minimal. It shows how to pass in an external AbortSignal and report progress without coupling to UI components.

async function downloadWithProgressAndAbort(
  url: string,
  onProgress: (received: number, total: number) => void,
  signal: AbortSignal
): Promise<Uint8Array[]> {
  const response = await fetch(url, { signal });

  if (!response.ok) {
    const errorBody = await response.text().catch(() => '');
    const details = errorBody ? `: ${errorBody.slice(0, 200)}` : '';
    throw new Error(
      `Request failed (${response.status} ${response.statusText})${details}`
    );
  }

  if (!response.body) {
    throw new Error('Response body is null.');
  }

  const total = Number(response.headers.get('content-length')) || 0;
  let received = 0;
  const chunks: Uint8Array[] = [];
  const reader = response.body.getReader();

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      chunks.push(value);
      received += value.length;
      onProgress(received, total);
    }
  } finally {
    reader.releaseLock();
  }

  return chunks;
}

If final memory footprint matters, stream chunks directly into a writable destination instead of returning Uint8Array[]. The control flow remains mostly the same.

Cancellation and Timeouts with AbortController #

Wire cancellation into both fetch and stream piping.

This is easy to miss: cancellation is only reliable when the same signal flows through each stage that supports it.

async function downloadWithTimeout(
  url: string,
  writable: WritableStream<Uint8Array>
): Promise<void> {
  const controller = new AbortController();
  const { signal } = controller;

  const timeoutId = setTimeout(() => controller.abort(), 10000);

  try {
    const response = await fetch(url, { signal });
    if (!response.ok) {
      throw new Error(`Request failed (${response.status} ${response.statusText})`);
    }
    if (!response.body) {
      throw new Error('Response body is null.');
    }

    await response.body.pipeTo(writable, { signal });
  } finally {
    clearTimeout(timeoutId);
  }
}

That single design habit prevents a common production bug: one layer aborts while another continues running in the background.

On the Node side, the equivalent reliability choice is whether you chain streams with .pipe() or use pipeline().

Node: Prefer pipeline() Over .pipe() #

For anything beyond trivial scripts, use pipeline() from node:stream/promises.

Why:

  • It propagates errors correctly.
  • It tears down the whole chain when one stage fails.
  • It fits naturally with async/await.
import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

async function compressFile(source: string, destination: string): Promise<void> {
  await pipeline(
    createReadStream(source),
    createGzip(),
    createWriteStream(destination)
  );
}

In other words, pipeline() gives you predictable teardown semantics so you do not need to hand-wire complex error forwarding across every stream stage.

For intermediate teams, this one choice avoids many silent failures in long-running services.

Before wrapping up, the next two sections give you a quick guardrail list for real project decisions.

Common Mistakes (and Quick Fixes) #

  • Mixing APIs repeatedly inside one flow: Convert once at the boundary, then stay in one model.
  • Ignoring backpressure signals: Always honor writer.ready or drain.
  • Forgetting cancellation wiring: Thread one AbortSignal through fetch and downstream pipe operations.
  • Assuming chunk boundaries are record boundaries: Reconstruct lines/messages explicitly in a transform.
  • Using streams for tiny payloads by default: Prefer simple buffering when data is small and bounded.

A Practical Decision Checklist #

Choose streams if most answers are "yes":

  • Could payload size exceed comfortable memory limits?
  • Do you need early processing before full download completes?
  • Will users benefit from progress updates or cancellation?
  • Is this path performance-sensitive enough to justify extra complexity?

Choose buffering if most answers are "no".

When Not to Use Streams #

Use regular buffering (json(), text(), reading full files) when:

  • Payloads are small.
  • You need the whole dataset anyway (for example, global sort).
  • Simpler code is more valuable than streaming throughput.

Streams are about predictable memory behavior and throughput under load, not mandatory complexity for every request.

Summary #

For intermediate developers, most production wins come from these habits:

  • Pick one stream model per pipeline and convert only at boundaries.
  • Respect backpressure (writer.ready, drain).
  • Use cancellation (AbortController) for user actions and timeouts.
  • Prefer Node pipeline() for robust teardown and error handling.

Master those patterns first, and you can add advanced techniques later only when your workload proves you need them.

Edit on Github