Appearance
3.13.9 — Iterators, Generators & Context Managers
A nightly job exports every order from the last five years into a CSV file: forty million rows. The first version loads all the rows into a list, then writes them. It runs out of memory at row twelve million and crashes. The second version changes three lines. It reads one page of rows, writes them, and asks for the next page. It finishes using less memory than a browser tab. The difference is the subject of the first half of this page: iterators and generators, which hand out values one at a time instead of building everything up front. The second half covers the with statement, which makes sure files, database transactions and locks are always cleaned up, even when something crashes in the middle. Both features are built on small protocols you can implement yourself.
1. What a for loop really does
JavaScript's for…of uses an iteration protocol (3.6.6). Python's for uses one too, and it is even smaller. There are two roles.
- An iterable is anything you can loop over. It has an
__iter__method that returns an iterator. Lists, strings, dicts, sets, files and ranges are all iterables. - An iterator is the object that does the looping. It has a
__next__method that returns the next value, and raisesStopIterationwhen there are no more values.
The built-in functions iter() and next() call those two methods. Let us drive a loop by hand.
python
skus = ["mug", "pen"]
it = iter(skus) # (1)
print(it)
print(next(it), next(it)) # (2)
try:
next(it) # (3)
except StopIteration:
print("StopIteration")text
<list_iterator object at 0x000002F426E106A0>
mug pen
StopIteration(1)iter(skus)callsskus.__iter__()and returns a separate iterator object.(2)Eachnext()hands out one value and moves an internal bookmark forward.(3)When the values run out, the iterator raisesStopIteration. In Python, raising an exception is the normal way to say "finished".
Now you can see what for sku in skus: actually runs.
python
it = iter(skus)
while True:
try:
sku = next(it)
except StopIteration:
break
print(sku) # the loop bodyEvery for loop, list comprehension, sum(), list(), sorted(), zip(), unpacking assignment and in check on an iterator runs this same loop. Implement the protocol once, and all of them work with your object.
Iterable versus iterator
Why two roles instead of one? Because a collection can be looped over many times, and each loop needs its own bookmark. The list is the iterable: it holds the data. Each call to iter() creates a fresh iterator with its own position. Two nested loops over the same list each get their own iterator, so they do not disturb each other.
An iterator is also iterable. Its __iter__ simply returns itself, so you can pass an iterator anywhere a for loop is expected. But it can only be used once.
python
remaining = iter([1, 2, 3, 4])
print(4 in remaining, list(remaining)) # (1)text
True [](1)inpulled values from the iterator until it found4, which used up every value. Nothing is left forlist(). This "exhausted iterator" surprise catches everyone once.
Writing an iterable class by hand
Here is the export job's source of orders, which arrive from the database in pages. A class can make itself loopable with the two methods.
python
class OrderPages:
def __init__(self, pages):
self.pages = pages
def __iter__(self): # (1)
return OrderPagesIterator(self.pages)
class OrderPagesIterator:
def __init__(self, pages):
self.pages = pages
self.index = 0 # (2)
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.pages):
raise StopIteration # (3)
page = self.pages[self.index]
self.index += 1
return page
pages = OrderPages([["ORD-1", "ORD-2"], ["ORD-3"]])
for page in pages:
print(page)
print(list(pages)) # (4)text
['ORD-1', 'ORD-2']
['ORD-3']
[['ORD-1', 'ORD-2'], ['ORD-3']](1)The iterable returns a brand-new iterator every time.(2)The iterator keeps the bookmark.(3)It signals the end withStopIteration.(4)Because each loop gets a new iterator, the second loop starts from the beginning again.
It works, but it is a lot of code for "hand out these pages in order". Python has a far shorter way.
2. Generators: functions that pause
A generator function is a function that contains the keyword yield. Calling it does not run its body. It returns a generator object, which is an iterator. Each next() runs the body until the next yield, hands out that value, and pauses, keeping every local variable exactly as it was. The next next() resumes from the line after the yield.
python
def read_orders(pages):
print(" generator started")
for page in pages:
for order in page:
yield order # (1)
print(" generator finished")
gen = read_orders([["ORD-1", "ORD-2"], ["ORD-3"]])
print(type(gen)) # (2)
print(next(gen)) # (3)
print(next(gen))
print(next(gen))
try:
next(gen) # (4)
except StopIteration:
print("StopIteration")text
<class 'generator'>
generator started
ORD-1
ORD-2
ORD-3
generator finished
StopIteration(1)yieldhands out one value and pauses the function right here.(2)Calling the function created a generator object. Notice that "generator started" has not printed yet. No line of the body has run.(3)The firstnext()runs the body from the top until the firstyield.(4)When the body reaches its end, the generator raisesStopIterationfor you.
The whole OrderPages class from section 1, with its separate iterator class and bookmark, becomes a few lines. The bookmark is simply where the function is paused, and the loop variables are ordinary locals.
How pausing works: the frame lives on the heap
A normal function call creates a frame: an object holding the function's local variables, its value stack and its current position in the bytecode. When the function returns, the frame is thrown away. A generator keeps its frame alive inside the generator object. yield saves the current position and steps out. next() steps back in and continues from the saved position. You can look inside.
python
import inspect
def read_orders(pages):
for page in pages:
for order in page:
yield order
gen = read_orders([["ORD-1"]])
print(inspect.getgeneratorstate(gen)) # (1)
next(gen)
print(inspect.getgeneratorstate(gen)) # (2)
print(list(gen.gi_frame.f_locals)) # (3)text
GEN_CREATED
GEN_SUSPENDED
['pages', 'page', 'order'](1)Created but never started.(2)Paused at ayield.(3)The paused frame still holds every local variable.
This is the same trick as JavaScript generators, and as Python closures keeping cells alive in 3.13.4: state that would normally die is kept reachable on the heap.
Why generators save memory: laziness
A generator produces each value only when someone asks for it. That is called lazy evaluation. Compare a list comprehension with a generator expression, which is the same syntax with round brackets (3.13.3).
python
import sys
eager = [n * 2 for n in range(1_000_000)] # (1)
lazy = (n * 2 for n in range(1_000_000)) # (2)
print(sys.getsizeof(eager), sys.getsizeof(lazy))text
8448728 208(1)The list holds a million results at once: over 8 MB for the pointers alone, plus the number objects.(2)The generator holds only its paused frame: about 200 bytes, whatever the length.
That is the fix for the opening story. The export job becomes a pipeline of generators: one reads pages from the database, one turns rows into CSV lines, and the file writer pulls lines one at a time. At any moment only one page is in memory, whether the table has forty rows or forty million.
python
def fetch_pages(page_size):
for start in range(0, 7, page_size): # pretend database with 7 rows
yield [f"ORD-{n}" for n in range(start, min(start + page_size, 7))]
def rows(pages):
for page in pages:
yield from page # (1)
def csv_lines(order_ids):
for order_id in order_ids:
yield f"{order_id},paid\n"
pipeline = csv_lines(rows(fetch_pages(page_size=3))) # (2)
for line in pipeline: # (3)
print(line, end="")text
ORD-0,paid
ORD-1,paid
ORD-2,paid
ORD-3,paid
ORD-4,paid
ORD-5,paid
ORD-6,paid(1)yield fromhands out every item of another iterable. Section 2.3 explains it.(2)Building the pipeline runs nothing. It only connects three paused generators.(3)Each turn of the loop pulls one line. That pull travels backwards through the chain.csv_linesasksrowsfor one ID, androwsasksfetch_pagesfor a page only when the current page runs out.
The price of laziness is that a generator can be used once. A second loop over the same generator gets nothing. If you need the values twice, either call the generator function again or store the values in a list.
2.1 Two-way generators: send
yield is also an expression. Its value is whatever the caller passes in with gen.send(value). That turns a generator into a small machine that receives input and keeps state between inputs. Such a two-way routine is called a coroutine.
python
def running_total():
total = 0
while True:
amount = yield total # (1)
total += amount
cart_total = running_total()
print(next(cart_total)) # (2)
print(cart_total.send(450)) # (3)
print(cart_total.send(40))
print(cart_total.send(120))text
0
450
490
610(1)This line does two things. It handstotalout, then pauses. When resumed bysend(x), theyieldexpression evaluates tox, which is stored inamount.(2)A new generator must first be advanced to its firstyield. This is called priming. Sending a value into a generator that has not started raisesTypeError.(3)Eachsendresumes the generator with a value and returns the next value it yields.
This send-and-pause machinery is where Python's async/await came from. Before Python 3.5, the asyncio library ran coroutines written as generators that yielded "please wake me when this network call finishes". 3.13.15 shows how await grew out of exactly this.
2.2 throw and close: controlling a generator from outside
Two more methods complete the set.
gen.throw(exception)raises an exception inside the generator, at the pausedyield. The generator can catch it and continue.gen.close()raises a specialGeneratorExitexception at the pausedyield, so the generator can clean up and finish.
python
def order_writer():
try:
while True:
order = yield
print(" writing", order)
except GeneratorExit: # (1)
print(" closing: flushing the buffer")
writer = order_writer()
next(writer)
writer.send("ORD-1")
writer.close() # (2)text
writing ORD-1
closing: flushing the buffer(1)GeneratorExitarrives at the pausedyield. Atry/finallywould catch it just as well.(2)close()is also called automatically when a generator object is garbage-collected, and when aforloop over it ends early withbreak. So a generator'sfinallyblock reliably runs its cleanup.
2.3 yield from: handing over to another generator
yield from other hands out every value of other as if the current generator had yielded them itself. It also passes send, throw and close straight through. Its most common use is recursion over nested data, such as a comment thread with replies to replies.
python
def flatten(comment):
yield comment["id"]
for reply in comment.get("replies", []):
yield from flatten(reply) # (1)
thread = {"id": 1, "replies": [{"id": 2, "replies": [{"id": 3}]}, {"id": 4}]}
print(list(flatten(thread)))text
[1, 2, 3, 4](1)Each reply is flattened by a recursive generator, and every ID it produces flows straight out.
yield from also captures the inner generator's return value. A return x inside a generator ends it and stores x on the StopIteration exception. result = yield from inner() receives it. That detail was the foundation of the old generator-based coroutines.
3. itertools: the iterator toolkit
The itertools module is a set of fast, lazy building blocks, all written in C. They take iterables and return iterators, so they chain into pipelines without building lists. Here are the ones you will reach for most often.
python
import itertools
print(list(itertools.islice(itertools.count(1001), 3))) # (1)
print(list(itertools.batched(range(7), 3))) # (2)
print(list(itertools.pairwise([10, 12, 15]))) # (3)
print(list(itertools.accumulate([450, 40, 120]))) # (4)
print(list(itertools.chain(["mug"], ("pen", "lamp")))) # (5)
print(list(itertools.product("SM", ["red", "blue"]))) # (6)
print([(key, list(group)) for key, group in
itertools.groupby(["a1", "a2", "b1", "a3"], key=lambda s: s[0])]) # (7)text
[1001, 1002, 1003]
[(0, 1, 2), (3, 4, 5), (6,)]
[(10, 12), (12, 15)]
[450, 490, 610]
['mug', 'pen', 'lamp']
[('S', 'red'), ('S', 'blue'), ('M', 'red'), ('M', 'blue')]
[('a', ['a1', 'a2']), ('b', ['b1']), ('a', ['a3'])](1)countcounts forever.islicetakes a slice of any iterator, here the first three order numbers.(2)batched, added in 3.12, cuts a stream into chunks. It is ideal for inserting rows into a database 500 at a time.(3)pairwisegives each neighbouring pair, useful for differences between readings.(4)accumulategives running totals.(5)chainjoins several iterables into one stream.(6)productgives every combination, here every size and colour of a T-shirt.(7)groupbygroups neighbouring items with the same key. Notice thata3forms its own group, because it is not next to the otheraitems. Sort by the key first if you want one group per key. This trap appears in code reviews constantly.
Two built-ins belong here too. zip walks several iterables side by side and silently stops at the shortest, which can hide a bug where two lists should have been the same length. Since Python 3.10, zip(a, b, strict=True) raises ValueError instead. And enumerate(items, start=1) gives each item with its position, so you never need range(len(items)).
4. Context managers: guaranteed cleanup
Now the second half of the page. Some resources must always be released: a file must be closed, a database transaction must be committed or rolled back, a lock must be released. If an exception happens in the middle, a plain close() at the end never runs. The long way to be safe is try/finally. The short way is with.
python
with open("orders.csv", "w") as out: # (1)
out.write("ORD-1,paid\n")
# (2) the file is closed here, even if write() had raised an exception(1)openreturns a file object that is a context manager.as outnames it.(2)Leaving the block, by reaching its end, byreturn, bybreakor by an exception, always closes the file.
What with does underneath
A context manager is any object with two methods, __enter__ and __exit__. The statement with manager as name: body runs roughly this.
python
manager = ... # the expression after `with`
name = manager.__enter__() # (1)
try:
... # body
except BaseException as error:
if not manager.__exit__(type(error), error, error.__traceback__): # (2)
raise # (3)
else:
manager.__exit__(None, None, None) # (4)(1)__enter__runs first. Whatever it returns is bound to theasname.(2)If the body raises,__exit__receives the exception's type, the exception and its traceback.(3)If__exit__returns a false value, the exception carries on outward. That is the normal case.(4)If the body finished normally,__exit__receives threeNones.
Here is a context manager written as a class, so you can watch both methods run.
python
import time
class Timer:
def __enter__(self):
self.start = time.perf_counter()
print(" enter")
return self # (1)
def __exit__(self, exc_type, exc, tb):
self.ms = (time.perf_counter() - self.start) * 1000
print(f" exit: exc_type={exc_type.__name__ if exc_type else None}")
return False # (2)
with Timer() as timer:
total = sum(range(1000))
try:
with Timer():
raise ValueError("bad price")
except ValueError as error:
print("still raised:", error)text
enter
exit: exc_type=None
enter
exit: exc_type=ValueError
still raised: bad price(1)Returningselflets the caller readtimer.msafter the block.(2)ReturningFalselets any exception continue.__exit__ran in both cases, which is the whole point.
Swallowing an exception
If __exit__ returns True, the exception is suppressed and the program continues after the with block. Use this rarely and narrowly. The standard library's contextlib.suppress does it for named exception types.
python
from contextlib import suppress
with suppress(FileNotFoundError): # (1)
open("yesterday.csv")
print("carried on")text
carried on(1)A missing file is ignored. Any other error still propagates. Never suppress plainException, which would hide real bugs.
5. @contextmanager: a context manager from a generator
Writing a class with two methods is clumsy for something that is really "do this, run the block, then do that". The decorator contextlib.contextmanager turns a generator into a context manager. Everything before the yield is __enter__. Everything after it is __exit__.
python
import time
from contextlib import contextmanager
@contextmanager
def timed(label):
start = time.perf_counter() # (1)
try:
yield # (2)
finally:
print(f" {label}: {(time.perf_counter() - start) * 1000:.0f} ms") # (3)
with timed("import batch"):
time.sleep(0.03)text
import batch: 30 ms(1)This runs on entry.(2)Thewithblock runs while the generator is paused here. A value afteryieldwould be bound to theasname.(3)This runs on exit. Thetry/finallymatters. If the block raises, the decorator throws the exception into the generator at theyield, using thethrowmethod from section 2.2. Withoutfinally, the timing line would be skipped on errors.
This is the decorator from 3.13.5's table, and now you can see how it works: it is a small class whose __enter__ calls next() on your generator and whose __exit__ calls next() again, or throw() if there was an exception.
Here is the most important real-world shape: a database transaction that commits on success and rolls back on failure.
python
from contextlib import contextmanager
@contextmanager
def transaction(connection):
print(" BEGIN")
try:
yield connection # (1)
except Exception:
print(" ROLLBACK") # (2)
raise
else:
print(" COMMIT") # (3)
with transaction("db") as conn:
print(" insert order")
try:
with transaction("db") as conn:
raise ValueError("bad row")
except ValueError:
passtext
BEGIN
insert order
COMMIT
BEGIN
ROLLBACK(1)The block runs with the connection.(2)Any error inside the block rolls back and is then re-raised, so the caller still learns about it.(3)Only a clean exit commits.
SQLAlchemy's with session.begin(): and FastAPI's yield dependencies (3.14.4) are both built on exactly this shape. There is also asynccontextmanager and async with, for resources that need await to open or close, covered in 3.13.15.
6. Many resources at once: ExitStack
with can open several resources in one line: with open(a) as f1, open(b) as f2:. But what if the number of resources is only known at run time, such as one file per warehouse? contextlib.ExitStack collects any number of context managers and closes all of them, in reverse order, when the block ends.
python
from contextlib import ExitStack
from pathlib import Path
for name in ["north.csv", "south.csv"]:
Path(name).write_text(f"stock from {name}")
with ExitStack() as stack:
files = [stack.enter_context(open(name)) for name in ["north.csv", "south.csv"]] # (1)
print([f.read() for f in files])
print(all(f.closed for f in files)) # (2)text
['stock from north.csv', 'stock from south.csv']
True(1)Each file is opened and registered with the stack.(2)All of them were closed when the block ended. If opening the second file had failed, the first would still have been closed.
What the interviewer will push on
"What is the difference between an iterable and an iterator?" An iterable has __iter__ and can produce fresh iterators. An iterator has __next__, holds the position and is single-use. The tell of understanding is explaining why there are two roles: one collection must support several independent loops at once.
"What does yield do, and how is a generator different from returning a list?" It pauses the function, keeping its frame on the heap, and hands out one value. A generator is lazy, uses constant memory and can be infinite, but it is single-pass. Mention the 200-byte generator against the 8 MB list.
"What is yield from?" Delegation to a sub-generator, forwarding values, send, throw and close, and capturing its return value. Give the recursive tree example. Bonus: it was the bridge to async/await.
"How does the with statement work? Can it swallow exceptions?" __enter__ on entry, __exit__(type, value, traceback) on any exit, and a true return value suppresses the exception. Show @contextmanager and why its yield needs try/finally.
"How would you process a 50 GB file in Python?" Iterate over the file object, which reads one line at a time, and chain generators for parsing and filtering. Use itertools.batched for bulk writes. Never call .read() or .readlines(). The common wrong answer loads it into pandas in one go.
Something to volunteer that nobody asks for: itertools.groupby only groups adjacent items, so unsorted input silently produces duplicate groups. And zip silently truncates to the shortest input unless you pass strict=True. Both are real production bugs, and naming them shows you have been bitten.
Recall
forcallsiter()once, thennext()untilStopIteration. An iterable has__iter__and makes fresh iterators. An iterator has__next__, holds the position and is single-use.- A generator function contains
yield. Calling it runs nothing and returns a generator. Eachnext()runs to the nextyieldand pauses, keeping the frame on the heap. - Generators are lazy (lazy evaluation): constant memory, can be infinite, single-pass. Chain them into pipelines.
send(x)makesyieldevaluate tox, which turns a generator into a coroutine. Prime it withnext()first.throw()raises inside it.close()raisesGeneratorExitsofinallyruns.yield fromdelegates and captures the return value.itertools:islice,count,batched,pairwise,accumulate,chain,product, andgroupby(adjacent only).zip(..., strict=True).withcalls__enter__, then always__exit__(type, value, traceback). Returning true suppresses the exception.@contextmanagerturns a generator with a singleyieldinto a context manager. Wrap theyieldintry/finally.ExitStackhandles a variable number of resources.
Self-test: Rewrite a for loop as a while loop with iter and next. Why is "generator started" not printed when you call a generator function? What does send need before its first use? Why must @contextmanager code put cleanup in finally? What does __exit__ returning True do?
Quiz Bank
FoundationalYou loop over a generator twice and the second loop does nothing. Why?
A generator is an iterator, and iterators are single-use. The first loop pulled every value and ran the generator body to its end. The generator is now finished, and every further next() raises StopIteration immediately, so the second loop ends at once. To loop again, call the generator function again to get a fresh generator. Or, if the data is small and needed several times, store it with list(gen).
AppliedWrite a generator that reads a large log file and yields only lines containing ERROR, without loading the file into memory.
python
def error_lines(path):
with open(path, encoding="utf-8") as log:
for line in log:
if "ERROR" in line:
yield line.rstrip("\n")A file object is an iterator that reads one line at a time from a buffer, so memory stays small regardless of file size. The with block keeps the file open while the generator is paused. If the caller stops early with break, the generator is closed, GeneratorExit is raised at the yield, and the with block closes the file.
InterviewImplement a context manager that temporarily changes an environment variable and restores it afterwards, even on error.
python
import os
from contextlib import contextmanager
@contextmanager
def env_var(name, value):
old = os.environ.get(name)
os.environ[name] = value
try:
yield
finally:
if old is None:
del os.environ[name]
else:
os.environ[name] = oldThe code before yield saves the old value and sets the new one. The finally block restores it whether the with body finished normally or raised, because @contextmanager throws any exception into the generator at the yield. Handling the "was not set before" case separately matters, because setting it back to an empty string is not the same as removing it.
StaffA data pipeline built from chained generators is hard to debug: an exception deep in a parsing stage shows a confusing traceback and nobody knows which record failed. How would you improve it without giving up laziness?
Keep the generator pipeline, but make each stage report context. Wrap the per-record work in each stage with try/except and re-raise a domain error that includes the record's identifier and the stage name, using raise ... from error so the original traceback is kept (3.13.10). Add a counter or enumerate so the error names the line number. For bad records that should not stop the job, send them to a separate "rejected" output with the reason, rather than raising. Keep stages small and named, so the traceback shows parse_rows, enrich_prices and so on. Log progress every N records from a dedicated stage. Laziness is preserved because none of this builds lists.
Next: this page leaned on exceptions everywhere: StopIteration ends loops, GeneratorExit closes generators, and __exit__ receives errors. 3.13.10 covers exceptions properly: the hierarchy, else and finally, chaining with from, exception groups, and the rules for handling errors well in real services.