Appearance
Streams
47. What are the different types of streams in Node.js?
The short answer
A stream is data handled piece by piece, in chunks, instead of all at once. Node.js has four kinds. A readable stream is a source you read from, such as a file being read or an incoming HTTP request. A writable stream is a destination you write to, such as a file being written or an HTTP response. A duplex stream is both at once, independently, such as a TCP socket. A transform stream is a duplex stream that changes data as it passes through, such as gzip compression or encryption.
| Kind | You | Examples |
|---|---|---|
| Readable | Read from it | fs.createReadStream, req in an HTTP server, process.stdin |
| Writable | Write to it | fs.createWriteStream, res in an HTTP server, process.stdout |
| Duplex | Both, independently | net.Socket, a WebSocket |
| Transform | Write in, read out changed | zlib.createGzip, crypto.createCipheriv, a CSV parser |
Streams are everywhere in Node.js, often without you noticing: every HTTP request body is a readable stream, and every response is a writable stream. Express's res.json simply writes to that stream for you.
48. What is the difference between a Buffer and a Stream?
The short answer
A Buffer holds all of the data in memory at once, as one block of bytes. A stream delivers data a chunk at a time, often as Buffers of a few kilobytes, so you can start processing before the whole thing has arrived and never hold all of it in memory. For small data, buffering is simpler. For large or unknown-size data, such as file uploads, exports, logs or proxying, streaming keeps memory low and constant and lets the first bytes go out sooner.
| Buffer the whole thing | Stream it | |
|---|---|---|
| Memory | Size of the data | Size of a few chunks |
| First byte out | After everything is read | Almost immediately |
| Code | Simpler | A little more involved |
| Good for | Small JSON, small files | Large files, uploads, exports, proxies |
The real risk with buffering: ten users downloading a 500 MB export at once means 5 GB of memory. With streaming, the same ten downloads use a few megabytes.
49. What are readable, writable, duplex, and transform streams?
The short answer
A readable stream emits chunks of data you consume, by listening for 'data', piping it, or with for await. A writable stream accepts chunks through write() and is finished with end(). A duplex stream has a readable side and a writable side that work independently, such as a network socket. A transform stream is a duplex stream whose output is computed from its input, chunk by chunk, such as compression, encryption or turning CSV lines into JSON.
Reading with for await, the clearest modern way:
ts
import { createReadStream } from 'node:fs'
let bytes = 0
for await (const chunk of createReadStream('access.log')) { // each chunk is a Buffer
bytes += chunk.length
}
console.log(`read ${bytes} bytes`)Writing, and respecting the return value:
ts
import { createWriteStream } from 'node:fs'
import { once } from 'node:events'
const out = createWriteStream('orders.csv')
for (const order of orders) {
if (!out.write(`${order.id},${order.total}\n`)) { // (1)
await once(out, 'drain') // (2)
}
}
out.end() // (3)
await once(out, 'finish')writereturnsfalsewhen the stream's internal buffer is full.- Wait for
'drain'before writing more. This is backpressure done by hand (question 51). end()says no more data is coming;'finish'fires when everything has been flushed.
A transform stream, masking card numbers in a log file:
ts
import { Transform } from 'node:stream'
const maskCards = new Transform({
transform(chunk: Buffer, _encoding, callback) {
const masked = chunk.toString('utf8').replace(/\b\d{12}(\d{4})\b/g, '************$1')
callback(null, masked) // pass on the changed chunk
},
})A real version should handle a card number split across two chunks, for example by processing line by line (question 53).
50. What is stream piping?
The short answer
Piping connects a readable stream to a writable stream, so data flows from one to the other automatically, chunk by chunk: source.pipe(destination). Pipes can be chained through transforms, such as file.pipe(gzip).pipe(res). pipe handles backpressure for you, pausing the source when the destination is full. But it does not forward errors or clean up the other streams when one fails, which is why stream.pipeline is preferred.
ts
import { createReadStream } from 'node:fs'
import { createGzip } from 'node:zlib'
app.get('/exports/orders.csv.gz', (req, res) => {
res.setHeader('Content-Type', 'application/gzip')
createReadStream('exports/orders.csv')
.pipe(createGzip()) // (1)
.pipe(res) // (2)
})- Each chunk read from disk is compressed as it passes through.
- Compressed chunks go straight to the client. Memory stays small whatever the file size.
The problem with pipe alone: if the file does not exist, the read stream emits an error, but the gzip stream and the response are left open, the client hangs, and the error may crash the process if nothing listens for it. pipeline fixes all of that (question 52).
51. What is backpressure, and how do streams handle it?
The short answer
Backpressure is what happens when data arrives faster than the destination can handle it, such as reading from a fast disk and sending to a slow mobile client. Without control, the excess piles up in memory until the process runs out. Streams handle it with a buffer limit called the high-water mark: when a writable's buffer is full, write() returns false, and the source should pause until the writable emits 'drain'. pipe and pipeline do this pausing and resuming automatically.
write returns false and the source pauses. ③ When the client catches up, 'drain' fires and the source resumes.What ignoring backpressure looks like:
ts
readable.on('data', (chunk) => res.write(chunk)) // ignores write()'s return valueWith a slow client, every chunk read from disk is queued in the response's memory. Download a 2 GB file this way over a slow connection and the process may hold most of it in memory.
Respecting it: use pipeline, or check write() and wait for 'drain' yourself (question 49).
The same idea beyond streams: whenever a producer can outrun a consumer, such as a queue consumer pulling messages faster than it can process them, you need a limit and a way to slow the producer down.
52. How does stream.pipeline() help manage stream errors?
The short answer
pipeline connects a chain of streams like pipe does, with backpressure, but also handles failure: if any stream in the chain errors, it destroys all the others, so files are closed and connections released, and it reports the error once, through a callback or, with node:stream/promises, a promise you can await inside try/catch. It also accepts async generator functions as transform steps. Use it instead of chaining pipe calls.
ts
import { pipeline } from 'node:stream/promises'
import { createReadStream, createWriteStream } from 'node:fs'
import { createGzip } from 'node:zlib'
async function archiveLog(name: string) {
try {
await pipeline(
createReadStream(`logs/${name}.log`),
async function* (source) { // (1)
for await (const chunk of source) yield chunk.toString().replaceAll(/token=\w+/g, 'token=***')
},
createGzip(),
createWriteStream(`archive/${name}.log.gz`),
) // (2)
console.log('archived', name)
} catch (err) {
logger.error({ err, name }, 'archiving failed') // (3)
throw err
}
}- An async generator as a transform step: no
Transformclass needed. - The promise resolves when the last stream has finished writing everything.
- Any failure lands here: a missing file, a full disk, a gzip error. Every stream in the chain has already been closed.
In an HTTP handler, if the client disconnects mid-download, the response stream errors, and pipeline stops reading the file too, instead of reading the rest of a large file for nobody.
53. How would you process a very large file efficiently in Node.js?
The short answer
Stream it, never read it all into memory. Read it with fs.createReadStream, split it into records as it flows, such as lines with readline or rows with a streaming CSV parser, and handle each record as it arrives, writing results out through another stream or in batches to the database. Keep backpressure, by awaiting each batch before reading more, so a slow database slows the reading instead of filling memory. Memory then stays roughly constant whatever the file size.
Importing a 5 GB CSV of orders, line by line, in batches of 1,000:
ts
import { createReadStream } from 'node:fs'
import { createInterface } from 'node:readline'
async function importOrders(path: string) {
const lines = createInterface({ input: createReadStream(path), crlfDelay: Infinity }) // (1)
let batch: Order[] = []
let count = 0
for await (const line of lines) { // (2)
if (count++ === 0) continue // skip header
batch.push(parseOrderLine(line))
if (batch.length === 1000) {
await db.orders.insertMany(batch) // (3)
batch = []
}
}
if (batch.length) await db.orders.insertMany(batch) // (4)
return count - 1
}readlineturns the stream of chunks into whole lines, handling lines split across chunks.for awaitpulls one line at a time.- Awaiting the insert pauses the loop, which pauses reading the file: backpressure for free.
- Write the last partial batch.
Options for different shapes of work:
| Need | Approach |
|---|---|
| Line-based text: logs, CSV | readline, or a CSV parser such as csv-parse in streaming mode |
| Huge JSON | A streaming JSON parser, or ask for NDJSON, one JSON object per line |
| CPU-heavy work per record | Hand batches to worker threads (question 23) |
| A job that takes minutes | Run it as a background job, report progress, allow resuming from a checkpoint |
Make it restartable: record how far you got, such as the last line number or ID, so a crash halfway through a 5 GB file does not mean starting again from zero.
54. What is the difference between fs.readFile() and fs.createReadStream()?
The short answer
fs.readFile reads the entire file into memory and gives it to you at once, as one Buffer or string, which is simple and fine for small files such as configuration. fs.createReadStream returns a readable stream that delivers the file in chunks, so memory stays small and constant, you can start processing straight away, and you can pipe it to a response or a transform. Use readFile for small files you need whole, and streams for large files, uploads and downloads.
readFile | createReadStream | |
|---|---|---|
| Memory | Whole file | A few chunks, tens of KB each |
| Result | One Buffer or string | A stream of chunks |
| Start processing | After the whole file is read | Immediately |
| Can pipe to a response | No | Yes (question 50) |
| Ranges | No | start and end options, for partial downloads |
| Good for | Config, templates, small JSON | Logs, exports, media, anything large |
ts
import { readFile } from 'node:fs/promises'
import { createReadStream } from 'node:fs'
import { pipeline } from 'node:stream/promises'
const config = JSON.parse(await readFile('config.json', 'utf8')) // small: read whole
app.get('/videos/:id', async (req, res) => {
res.setHeader('Content-Type', 'video/mp4')
await pipeline(createReadStream(`media/${videoFileFor(req.params.id)}`), res) // large: stream
})A size limit to know: readFile refuses files larger than 2 GiB with the error ERR_FS_FILE_TOO_LARGE, and long before that size it puts heavy pressure on memory. Streams have no such limit.