Node.js Buffer vs streams
- Last updated
- Prerequisites:
- Node.js fs and http basics
- Understanding of memory allocation in JS
- nodejs
- buffer
- streams
- memory
- binary-data
- backpressure
Read at your depth
The practical view
A Buffer is a fixed-size chunk of memory outside the V8 heap: Buffer.alloc(1024) reserves 1KB of raw memory, and Buffer.from(str) copies a string into binary. Reading a whole file with fs.readFileSync returns one Buffer for the entire content; for a 4GB file that is 4GB in memory. Streams process data incrementally: fs.createReadStream(path) emits 'data' chunks; pipe() connects readable → writable (fs.createWriteStream, res (response)). The practical rule: buffers for small, known-size binary work; streams for anything that can be processed as it arrives — files, network responses, logs, CSV parsing.
Legacy vs modern
Whole-file Buffer read vs streamed processing
readFileSync loads the entire payload into memory before any processing; a read stream bounds memory and starts processing on the first chunk.
const data = fs.readFileSync('big.log', 'utf8');
processLines(data);fs.createReadStream('big.log')
.pipe(createLineSplitter())
.on('data', handleLine);Interview gotchas
Context
This is the single most important streams question — the interviewer wants the mechanism (highWaterMark, write() return value, drain event, pause/resume).
The mechanical answer
Backpressure is the flow-control mechanism that prevents a fast producer from flooding a slow consumer. In Node, the readable side pauses (stops reading from the source) when the writable side reports that its internal buffer is full. pipe() wires this automatically: it listens to 'data' on the source (flowing mode), calls dest.write(chunk), and if write() returns false (buffer above highWaterMark), pipe pauses the source until the dest emits 'drain', then resumes. Without pipe, a manual 'data' handler that ignores write()'s return value causes unbounded buffering and memory growth. pipeline() additionally forwards errors and closes streams correctly — pipe() alone does not.
Trap
The trap answer is 'backpressure means the consumer slows down the producer by throttling' — it is a pull-based pause, not a rate limit. Another trap: 'writing to a stream is synchronous' — writes are buffered and flushed asynchronously; the return value of write() is the backpressure signal. Candidates who name 'drain' and highWaterMark concretely, and who know pipeline() for error handling, demonstrate production experience.
Context
The interviewer wants to verify understanding of off-heap memory and the pooling/lifetime implications of Buffers.
The mechanical answer
A Buffer is a Uint8Array backed by an ArrayBuffer allocated in native (C++) memory rather than V8's managed heap objects — the bytes live outside the JS heap, so large binary data does not count against V8's heap limits and avoids GC pressure from copying. Buffer.allocUnsafe(size) returns a Buffer whose memory is not zero-filled (faster), drawn from a pool of pre-allocated 8KB slabs for small sizes; this is why it must be written to before being read — uninitialized memory can contain stale data from earlier allocations. Buffer.alloc(size) zero-fills. The danger: holding Buffers for a long time pins native memory that the allocator may not return promptly, so stream consumers should let chunks be GC'd rather than accumulating them.
Trap
The naive answer is 'Buffer is an array of bytes' — it is a Uint8Array backed by an external ArrayBuffer, with different memory behavior and methods (write, toString, copy). Another trap: 'allocUnsafe is always dangerous' — it is a documented performance choice whose only risk is reading unwritten bytes, so the guidance is 'write before read', not 'never use'. Mentioning the 8KB slab pool shows you know the pooling mechanism, not just the API.