Appearance
3.13.15 — asyncio In Depth
A checkout page needs three things before it can show the total: the customer's saved addresses, live stock for each item, and a shipping quote. Each comes from a different service, and each takes about 200 milliseconds. Asked one after another, the page takes 600 milliseconds. Asked all at once, it takes 200. Now multiply that by ten thousand customers checking out at the same time. Ten thousand threads would need gigabytes of memory just for their stacks, and the operating system would spend much of its time switching between them. asyncio does it differently. It runs all ten thousand conversations inside one thread, switching between them only at the exact points where each one is waiting. This page explains how that works, from the generator trick it grew out of, through the event loop, to the rules that keep an async service fast. FastAPI is built on top of all of it.
1. The story: from callbacks to async and await
The problem asyncio solves is old. A server spends most of its life waiting: for a database, for another service, for the client to send more bytes. The question is what a single thread should do while it waits.
- Callbacks, 2002. The Twisted framework let you say "when this data arrives, call this function". It worked, and it scaled, but code became a pyramid of nested callbacks, the same "callback hell" JavaScript suffered (3.6.8).
- Generators that can pause, 2005. PEP 342 added
send()to generators (3.13.9). People realised that a generator couldyieldat the exact point where it needed to wait, and a scheduler could resume it with the result. Code written this way looks sequential but pauses at each wait. The Tornado framework, built at FriendFeed in 2009, used this idea. yield from, 2012. PEP 380 let one generator hand control to another, so pausable functions could call other pausable functions.- asyncio, 2014. Guido van Rossum designed a standard event loop for Python, first called "Tulip", which became the
asynciomodule in Python 3.4 (PEP 3156). Coroutines were still generators usingyield from. asyncandawait, 2015. Yury Selivanov's PEP 492, in Python 3.5, gave the pattern its own syntax.async defmakes a native coroutine, andawaitreplacesyield from. Under the hood it is the same machinery.- Making it safe, 2018 to 2022.
asyncio.run()arrived in 3.7.TaskGroupandasyncio.timeout()arrived in 3.11, inspired by the Trio library's idea of "structured concurrency".
So await is not magic. It is a generator pausing and handing control back to a scheduler. Section 3 proves it by building a scheduler from scratch.
2. Coroutines, await and asyncio.run
A function defined with async def is a coroutine function. Calling it does not run its body. It returns a coroutine object, which is a paused computation, exactly like calling a generator function.
python
import asyncio
async def fetch_stock(sku):
await asyncio.sleep(0.2) # (1)
return sku, 12
pending = fetch_stock("mug") # (2)
print(pending)
pending.close() # (3)
print(asyncio.run(fetch_stock("mug"))) # (4)text
<coroutine object fetch_stock at 0x000001FC3ED60D40>
('mug', 12)(1)awaitmeans "pause me here until this finishes, and let other work run meanwhile".asyncio.sleepstands in for a network call.(2)Calling the function only created a coroutine object. No line of the body ran.(3)We discard it cleanly. A coroutine that is created and never awaited triggers the warning "coroutine … was never awaited", one of the most common asyncio bugs.(4)asyncio.runcreates an event loop, runs the coroutine to completion on it, and closes the loop. It is the entry point of an async program. In FastAPI, the server (Uvicorn) runs the loop for you, so you never callasyncio.runyourself.
You can only use await inside an async def function. That rule spreads upward: a function that awaits must be async, so its caller must await it, and so on up to the event loop. People call this "function colouring". It is the price of making every pause point visible in the code.
3. What await really does: build an event loop
Here is the whole idea in about twenty lines. Two couriers each drive a route of several legs. Each leg is a wait. A tiny scheduler runs both couriers in one thread, using a simulated clock so nothing actually sleeps.
python
import heapq
class Sleep:
def __init__(self, minutes):
self.minutes = minutes
def __await__(self): # (1)
yield self.minutes # (2)
async def courier(name, legs):
for minutes in legs:
print(f" {name}: driving {minutes} min")
await Sleep(minutes) # (3)
return f"{name} finished"
def run_all(*coroutines):
timers = [(0, n, coro) for n, coro in enumerate(coroutines)] # (4)
heapq.heapify(timers)
while timers:
clock, n, coro = heapq.heappop(timers) # (5)
try:
minutes = coro.send(None) # (6)
except StopIteration as finished: # (7)
print(f"t={clock}: {finished.value}")
continue
heapq.heappush(timers, (clock + minutes, n, coro)) # (8)
run_all(courier("asha", [5, 10]), courier("ravi", [3, 3, 3]))text
asha: driving 5 min
ravi: driving 3 min
ravi: driving 3 min
asha: driving 10 min
ravi: driving 3 min
t=9: ravi finished
t=15: asha finished(1)An object is awaitable if it has__await__, which must return an iterator. A generator function is the easy way to write one.(2)Thisyieldis the real pause. The value travels all the way up through everyawaitin the chain, to whoever calledsend(). Here it carries a request: "wake me in this many minutes".(3)await Sleep(5)runsSleep(5).__await__()and passes its yields straight up, exactly likeyield fromin 3.13.9.(4)The scheduler keeps a heap, a structure that always gives back the smallest item first, of(wake-up time, tie-breaker, coroutine).(5)Take whichever coroutine should wake up soonest.(6)send(None)resumes it. It runs until its nextawaitpauses it, and the yielded number arrives here.(7)When a coroutine returns, Python raisesStopIterationcarrying the return value. That is the generator protocol again.(8)Schedule the coroutine to wake up after its requested wait.
Follow the output. Both couriers run "at the same time" in one thread. Ravi finishes at minute 9, after three 3-minute legs, while Asha is on her second leg. Asha finishes at 15. The total is the length of the longest route, not the sum of both.
The real asyncio event loop is this same idea with two additions.
- Coroutines do not yield numbers. They yield futures: objects representing a result that will arrive later, such as "the reply on this socket". The loop wakes the coroutine when its future is complete, and passes the result back in.
- Besides timers, the loop asks the operating system which sockets have data ready. It uses the
selectorsmodule, which wrapsepollon Linux,kqueueon macOS and I/O completion ports on Windows. Those are the same kernel mechanisms Part 2 described and Node.js's libuv uses (3.8.1).
Figure 1 shows one turn of the real loop.
Step ③ is where the loop spends its idle time, inside the kernel, using no CPU, with a timeout set to the next timer. One thread can watch tens of thousands of sockets this way.
4. Running things concurrently: tasks
Awaiting one coroutine after another runs them one after another. Each await pauses until that one finishes.
python
import asyncio
import time
async def fetch_stock(sku):
await asyncio.sleep(0.2)
return sku, 12
async def sequential():
start = time.perf_counter()
await fetch_stock("mug") # (1)
await fetch_stock("pen")
await fetch_stock("lamp")
print(f"sequential: {time.perf_counter() - start:.2f}s")
asyncio.run(sequential())text
sequential: 0.60s(1)Nothing else was scheduled, so while this waits the loop has nothing to do.
To run them at the same time, wrap each one in a task. A task is a coroutine that the event loop has been told to drive on its own, in the background. It starts running at the loop's next turn, without anyone awaiting it.
TaskGroup: the modern way
Python 3.11 added asyncio.TaskGroup, and it should be your default.
python
async def checkout_data():
start = time.perf_counter()
async with asyncio.TaskGroup() as group: # (1)
mug = group.create_task(fetch_stock("mug")) # (2)
pen = group.create_task(fetch_stock("pen"))
lamp = group.create_task(fetch_stock("lamp"))
print(f"task group: {time.perf_counter() - start:.2f}s", # (3)
mug.result(), pen.result(), lamp.result())
asyncio.run(checkout_data())text
task group: 0.20s ('mug', 12) ('pen', 12) ('lamp', 12)(1)The group owns every task created inside it.(2)create_taskschedules the coroutine and returns a task immediately.(3)Theasync withblock does not end until every task in the group has finished. So by this line all results are ready, and the total time is the longest wait, not the sum.
What happens when one task fails? This is where TaskGroup earns its place.
python
async def charge(order_id, fail=False, delay=0.1):
await asyncio.sleep(delay)
if fail:
raise ConnectionError(f"{order_id}: gateway down")
print(f" charged {order_id}")
return order_id
async def charge_all():
try:
async with asyncio.TaskGroup() as group:
group.create_task(charge("ORD-1", delay=0.5)) # (1)
group.create_task(charge("ORD-2", fail=True)) # (2)
group.create_task(charge("ORD-3", fail=True, delay=0.15))
except* ConnectionError as failures: # (3)
print("failed:", [str(e) for e in failures.exceptions])
asyncio.run(charge_all())text
failed: ['ORD-2: gateway down'](1)A slow task.(2)This one fails after 0.1 seconds.(3)As soon as one task fails, the group cancels all the others, waits for them to stop, and raises anExceptionGroup(3.13.10).ORD-1was never charged, andORD-3never got far enough to fail.
This is structured concurrency: tasks live inside a block, the block cannot end while any of them is still running, and a failure cleans up its siblings. No task is ever left running in the background after the code that started it has moved on.
gather: the older way
asyncio.gather(a(), b(), c()) runs coroutines together and returns their results as a list. It is still everywhere in existing code. Its default failure behaviour is weaker. The first exception is raised to you, but the other tasks keep running in the background with nobody watching them. With return_exceptions=True, failures come back as values in the result list instead.
python
async def charge_some():
results = await asyncio.gather(
charge("ORD-1"), charge("ORD-2", fail=True), return_exceptions=True)
print(results)
asyncio.run(charge_some())text
charged ORD-1
['ORD-1', ConnectionError('ORD-2: gateway down')]Use gather(..., return_exceptions=True) when you deliberately want "try all, report each". Use TaskGroup when the tasks succeed or fail together.
Keep a reference to every task
A task created with asyncio.create_task() outside a group can be garbage-collected while still running if nothing holds a reference to it, because the event loop keeps only a weak reference (3.13.13). A "fire and forget" task can therefore silently vanish. Keep tasks in a set, or better, use a TaskGroup.
5. Timeouts and cancellation
Every call to another service needs a time limit. Otherwise one slow dependency holds requests open until the whole service runs out of capacity.
python
import asyncio
async def slow_gateway():
await asyncio.sleep(5)
async def pay():
try:
async with asyncio.timeout(0.3): # (1)
await slow_gateway()
except TimeoutError: # (2)
print("gave up after 0.3s")
asyncio.run(pay())text
gave up after 0.3s(1)asyncio.timeout, added in 3.11, limits everything inside the block.(2)When time runs out, the work inside is cancelled, and aTimeoutErroris raised outside the block. Older code usesasyncio.wait_for(coro, timeout), which does the same thing.
How cancellation works
Cancelling a task does not kill it instantly. task.cancel() arranges for an asyncio.CancelledError to be raised inside the coroutine at the await where it is paused. That is the throw() method of generators again (3.13.9). So the coroutine gets a chance to clean up.
python
import asyncio
async def sync_inventory():
try:
print(" sync started")
await asyncio.sleep(10)
except asyncio.CancelledError:
print(" cleaning up after cancel") # (1)
raise # (2)
async def main():
task = asyncio.create_task(sync_inventory())
await asyncio.sleep(0.1)
task.cancel()
try:
await task
except asyncio.CancelledError:
print("task cancelled:", task.cancelled())
asyncio.run(main())text
sync started
cleaning up after cancel
task cancelled: True(1)Cleanup runs. Afinallyblock or anasync withwould also run here.(2)Always re-raiseCancelledError. If you swallow it, the task carries on as if nothing happened, and timeouts andTaskGroupstop working correctly.CancelledErrorinherits fromBaseException, notException, precisely so that a broadexcept Exception:does not catch it by accident.
6. The cardinal rule: never block the event loop
All tasks share one thread. A task only gives up control at an await. So a task that does slow work without awaiting freezes every other task, and every other request, until it is done. Here a heartbeat should tick every 0.1 seconds while a request handler runs.
python
import asyncio
import time
async def heartbeat(start):
for _ in range(5):
print(f" tick {time.perf_counter() - start:.2f}")
await asyncio.sleep(0.1)
async def blocking_handler():
time.sleep(0.35) # (1)
async def friendly_handler():
await asyncio.to_thread(time.sleep, 0.35) # (2)
async def run(handler):
start = time.perf_counter()
await asyncio.gather(heartbeat(start), handler())
print("blocking:")
asyncio.run(run(blocking_handler))
print("to_thread:")
asyncio.run(run(friendly_handler))text
blocking:
tick 0.00
tick 0.35
tick 0.47
tick 0.57
tick 0.68
to_thread:
tick 0.00
tick 0.12
tick 0.23
tick 0.33
tick 0.44(1)time.sleepis an ordinary blocking call. The whole thread stops, so the heartbeat could not tick for 0.35 seconds. In a web server, every request on that worker would have frozen.(2)asyncio.to_threadruns the blocking call in a separate thread and awaits its result. The loop stays free, and the heartbeat ticks on time.
Things that block the loop, and their fixes:
| Blocking | Async replacement |
|---|---|
time.sleep | await asyncio.sleep |
requests.get | httpx.AsyncClient |
| sync database driver | async driver (asyncpg) |
| big file read | asyncio.to_thread |
| CPU-heavy work | process pool |
For CPU-heavy work, a thread does not help, because of the GIL (3.13.13). Use loop.run_in_executor(process_pool, func, ...) to send it to another process. During development, asyncio.run(main(), debug=True) logs any step that holds the loop for more than 100 milliseconds, which finds these bugs quickly.
7. Limiting concurrency with a semaphore
Starting ten thousand requests at once is easy with tasks, and it will overwhelm the service you are calling. An asyncio.Semaphore limits how many tasks may be inside a block at once. It is the async version of the thread semaphore in 3.13.14.
python
import asyncio
import time
gateway = asyncio.Semaphore(2) # (1)
async def charge(n, start):
async with gateway: # (2)
print(f" t={time.perf_counter() - start:.1f} charging ORD-{n}")
await asyncio.sleep(0.2)
async def main():
start = time.perf_counter()
async with asyncio.TaskGroup() as group:
for n in range(5):
group.create_task(charge(n, start))
asyncio.run(main())text
t=0.0 charging ORD-0
t=0.0 charging ORD-1
t=0.2 charging ORD-2
t=0.2 charging ORD-3
t=0.4 charging ORD-4(1)At most two charges at once.(2)Tasks beyond the limit wait here, without blocking the loop, until a slot frees up.
8. Async iteration and async context managers
The two protocols from 3.13.9 each have an async version, for when getting the next item, or opening and closing a resource, involves waiting.
python
import asyncio
from contextlib import asynccontextmanager
async def order_pages(): # (1)
for page in range(1, 4):
await asyncio.sleep(0.01) # fetch a page
yield [f"ORD-{page}{n}" for n in range(2)]
@asynccontextmanager
async def db_connection(): # (2)
print(" open connection")
try:
yield "conn"
finally:
print(" close connection")
async def export():
async with db_connection() as conn: # (3)
async for page in order_pages(): # (4)
print(" page", page)
asyncio.run(export())text
open connection
page ['ORD-10', 'ORD-11']
page ['ORD-20', 'ORD-21']
page ['ORD-30', 'ORD-31']
close connection(1)An async generator: anasync defwithyield. It mayawaitbetween items.(2)asynccontextmanagerworks likecontextmanager, but its setup and cleanup mayawait.(3)async withcalls__aenter__and__aexit__, which are awaited.(4)async forcalls__anext__and awaits each item.
Database sessions, HTTP clients and streaming responses in FastAPI all use these two forms.
9. Context variables: per-request data
In a threaded server, per-request data such as a request ID can live in thread-local storage. In asyncio, thousands of requests share one thread, so thread-locals would mix them up. Context variables, from the contextvars module added in 3.7, solve this. Each task gets its own copy of the context, so a value set in one request is invisible to the others.
python
import asyncio
import contextvars
request_id = contextvars.ContextVar("request_id", default="-") # (1)
def log(message):
print(f" [{request_id.get()}] {message}") # (2)
async def load_order():
await asyncio.sleep(0.01)
log("loaded order")
async def handle(rid):
request_id.set(rid) # (3)
log("start")
await load_order()
log("end")
async def main():
await asyncio.gather(handle("req-1"), handle("req-2"))
asyncio.run(main())text
[req-1] start
[req-2] start
[req-1] loaded order
[req-1] end
[req-2] loaded order
[req-2] end(1)Create the variable once, at module level.(2)Any function, however deep, can read the current request's value without it being passed as a parameter.(3)Each request sets its own value. The two requests interleave, yet each log line carries the right ID.
3.14.5 uses exactly this to put a request ID on every log line of a FastAPI service.
10. asyncio compared with JavaScript
If you know JavaScript's event loop (3.6.8), the ideas map closely, with three differences worth knowing.
- Starting work. Calling a JavaScript
asyncfunction starts it immediately and returns a promise. Calling a Pythonasync deffunction starts nothing. You mustawaitit or wrap it in a task. - The loop is a library. JavaScript always has an event loop. In Python, synchronous code is the default, and the loop exists only inside
asyncio.runor a server like Uvicorn. - Swapping the loop. Python lets you replace the event loop implementation. uvloop, which is built on libuv, the library under Node.js, is a drop-in replacement that runs network code two to four times faster, and Uvicorn uses it automatically when it is installed.
What the interviewer will push on
"What happens when you call an async def function?" It returns a coroutine object and runs nothing. It runs only when awaited or wrapped in a task. The common bug is forgetting the await, which produces "coroutine was never awaited".
"How does await work under the hood?" A coroutine is a generator-like object. await passes a future up through the call chain to the event loop and pauses. The loop resumes it with send() when the future completes. Building the twenty-line scheduler, or explaining it, shows real depth.
"gather or TaskGroup?" TaskGroup gives structured concurrency: when one task fails, the others are cancelled and all failures are raised as an ExceptionGroup. gather by default leaves the other tasks running when one fails. Use gather(return_exceptions=True) when you want every result regardless of failures.
"What blocks the event loop, and how do you find it?" Any synchronous I/O or CPU work without an await: time.sleep, requests, sync database drivers, heavy JSON parsing. Fix with async libraries, asyncio.to_thread or a process pool. Find it with debug=True, which logs slow steps.
"How does cancellation work?" CancelledError is raised inside the task at its current await. Clean up and re-raise. It is a BaseException, so except Exception does not swallow it.
Something to volunteer that nobody asks for: the event loop holds only weak references to tasks, so a create_task whose result you do not store can be garbage-collected mid-flight. Keep references or use a TaskGroup. Engineers who have run async code in production know this one.
Recall
- History: Twisted callbacks → generator coroutines (PEP 342,
yield fromin PEP 380) → asyncio (PEP 3156, 3.4) →async/await(PEP 492, 3.5) →asyncio.run(3.7) →TaskGroupandtimeout(3.11). - An
async defis a coroutine function. Calling it creates a coroutine and runs nothing.asyncio.runstarts an event loop. awaitpasses a future up to the loop and pauses, the same mechanism as a generator'syield. The loop runs ready tasks, fires timers, asks the OS (epoll,kqueue, IOCP) which sockets are ready, and wakes waiting tasks.- A task runs a coroutine concurrently.
TaskGroupgives structured concurrency: it waits for all tasks and cancels the rest on failure.gatheris older. Keep references to tasks. asyncio.timeout()cancels work that takes too long. Cancellation raisesCancelledErrorinside the task. Clean up and re-raise.- Never block the loop. Use async libraries,
asyncio.to_threadfor blocking calls, and processes for CPU work.debug=Truefinds slow steps.Semaphorelimits concurrency. async for/ async generator,async with/asynccontextmanager. Context variables give each task its own request-scoped values. uvloop speeds up the loop.
Self-test: Why does calling fetch_stock("mug") print nothing? In the twenty-line scheduler, what travels up through await, and what calls send? What does a TaskGroup do when one task fails? Why does time.sleep inside a handler freeze other requests? Why must you re-raise CancelledError?
Quiz Bank
FoundationalWhat is the difference between a coroutine and a task?
A coroutine is the paused computation you get by calling an async def function. It does nothing until something drives it, either by awaiting it, which runs it inside the current task, or by giving it to the event loop. A task is a coroutine that has been scheduled on the event loop with create_task or a TaskGroup. The loop drives it independently, so it runs concurrently with other tasks, and it can be awaited later for its result, or cancelled.
AppliedAn async endpoint calls three internal services with three awaits in a row and takes 600 ms. How do you speed it up safely?
The three awaits are sequential, so the time is the sum of the three calls. If the calls do not depend on each other, start them together inside an asyncio.TaskGroup and read each task's result after the block. The time becomes that of the slowest call, about 200 ms. Wrap the group in asyncio.timeout() so a slow dependency cannot hold the request open, and decide what to do when one call fails. Either let the group fail the whole request, or catch that one failure inside its own coroutine and return a fallback, such as showing the page without a shipping estimate. Make sure the HTTP client is async and shared across requests, so connections are reused.
InterviewExplain how a single thread can serve 10,000 concurrent connections with asyncio.
Each connection is handled by a coroutine that spends almost all its time paused at an await, waiting for data on a socket. A paused coroutine is just a small object on the heap, a few kilobytes, with no thread stack and no operating-system thread. The event loop registers all 10,000 sockets with the kernel's readiness mechanism, such as epoll on Linux, and makes one system call that sleeps until any of them has data or a timer is due. It then resumes only the coroutines whose sockets are ready. Each runs until its next await, which takes microseconds. As long as no handler blocks or computes for long, one thread keeps all of them moving, and the cost of an idle connection is memory, not CPU.
StaffAfter moving a service to asyncio, p99 latency got worse even though average latency improved. What would you look for?
Long tail latency in an async service usually means something occasionally holds the event loop, and every request on that worker waits behind it. Look for synchronous calls hidden in async code paths. Common ones are a sync database or HTTP client, file I/O, time.sleep and DNS resolution. Logging handlers that write over the network can block too. So can CPU spikes, such as parsing a very large JSON body or compressing a response. Enable loop debug or a monitor that measures loop lag, and correlate slow requests with slow loop steps. Also check concurrency limits: unbounded fan-out can overload a downstream service, whose slowness then shows up as tail latency. Fix with async libraries, to_thread for unavoidable blocking calls, process pools for CPU work, semaphores for fan-out, and enough worker processes to use every core.
Next: the language and its runtime are now covered. 3.13.16 closes the chapter with the everyday tools that make code look like a senior wrote it: structural pattern matching, f-strings and the new t-strings, and the standard-library modules you will use every week.