Python generators: lazy memory efficiency
- Last updated
- Prerequisites:
- Python functions and iteration basics
- Understanding of lists vs iterators
- python
- generators
- memory
- lazy-evaluation
- iterators
- yield
Read at your depth
The practical view
A generator is a function with yield instead of return: def evens(n): for i in range(n): if i % 2 == 0: yield i. Calling it returns a generator object (an iterator), not the values. Each next(gen) resumes execution from the previous yield until the next yield or StopIteration. for x in evens(10) drives it automatically. The memory win: a list of a million numbers holds a million int objects; a generator computes them on demand. Use generators for streaming large data (files, logs, API pagination), infinite sequences, and pipelines (gen1 | gen2 via yield from or chained loops).
Legacy vs modern
Building a list vs streaming with a generator
A list materializes every value in memory; a generator computes one value at a time, bounding memory to the working set.
lines = [line.strip() for line in open('big.log')]
unique = {line for line in lines}unique = set()
total = 0
for line in open('big.log'): # iterating a file IS a generator
unique.add(line.strip())
total += 1Interview gotchas
Context
This targets the frame-retention mechanism — the interviewer wants the PyFrameObject and the retention footgun, not just 'generators are lazy'.
The mechanical answer
A suspended generator holds its entire execution state in a heap-allocated frame: locals, the operand stack, and the instruction pointer (f_lasti). That frame stays alive as long as the generator object is referenced, which means a generator can pin memory unexpectedly — e.g., a generator that yielded one value but whose frame still references a large list keeps that list reachable. The fix is to scope the generator tightly (del gen, or let it go out of scope) and avoid long-lived references into a generator that captures big state. Also, the generator's try/finally blocks run on close(), so an abandoned generator may never run cleanup until it is collected.
Trap
The trap answer is 'generators use no memory'. They hold the frame (locals + state) — that is precisely how laziness works, and it can retain large referenced objects. Another trap: 'iterating a generator twice works like a list' — generators are single-pass iterators; the second loop yields nothing. Candidates who mention f_lasti and the frame's referenced-object retention demonstrate the actual mechanism rather than the slogan.
Context
The interviewer wants the performance trade-off analyzed honestly — frame-resume overhead versus memory and streaming wins.
The mechanical answer
Each next() call must resume the frame: restore the eval state, run until the next yield, and suspend again — a function-call-like cost per value, plus the iterator protocol dispatch (PEP 380 yield-from adds another layer). A list comprehension runs a tight loop in C (LIST_APPEND) and is often faster per element for small, fully-consumed data. Generators win on the axes that dominate real systems: memory boundedness for large/unbounded data (a 10GB stream fits), early termination (stop after the first match without computing the rest), and pipeline composition where intermediate lists would multiply memory. So the rule is: materialize small data, stream large or potentially-infinite data.
Trap
The naive answer is 'generators are always more efficient'. For small, always-consumed inputs a list is typically faster and simpler. Another trap: 'generators save memory on everything' — the frame-retention case (holding a generator referencing a big object) can pin MORE memory than the list would have used if the list were consumed and dropped. Mentioning early termination and pipeline memory as the real wins is what makes the answer senior.