Skip to content

Volume ITechnology

3.14.8 — Streaming, Uploads, WebSockets & Real-Time ​

So far every response was one JSON document, built completely in memory and then sent. That shape breaks in three situations the order service now faces. Finance wants a CSV export of five million orders, which does not fit in memory and takes minutes to build. Customers want to watch their order move from "packed" to "out for delivery" without refreshing the page. And the delivery app needs a live, two-way connection, so the driver's position appears on the customer's map within a second. This page covers the tools for each: streaming responses, server-sent events, uploads that never load whole files into memory, and WebSockets. Every one of them is built on the ASGI message model from 3.14.1, where a response can be many send calls instead of one.

1. Streaming a large response ​

A normal endpoint builds the whole body, then sends it. For a five-million-row export, that means holding every row in memory, and the client sees nothing until the very end. Often a proxy or load balancer gives up first, because nothing arrived for sixty seconds.

A StreamingResponse takes an iterator and sends each piece as soon as it is produced. Combined with the async generators from 3.13.15, memory stays flat whatever the size of the export.

python
import asyncio
import csv
import io
from collections.abc import AsyncIterator
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()
ORDERS = [{"id": n, "sku": "MUG" if n % 2 else "PEN", "total": 450 if n % 2 else 40}
          for n in range(1, 6)]

async def fetch_pages(page_size: int):                          # (1)
    for start in range(0, len(ORDERS), page_size):
        await asyncio.sleep(0.01)                               # stands in for a database query
        yield ORDERS[start:start + page_size]

async def csv_rows() -> AsyncIterator[str]:
    buffer = io.StringIO()
    writer = csv.writer(buffer)
    writer.writerow(["id", "sku", "total"])
    async for page in fetch_pages(page_size=2):
        for order in page:
            writer.writerow([order["id"], order["sku"], order["total"]])
        yield buffer.getvalue()                                 # (2)
        buffer.seek(0)
        buffer.truncate(0)                                      # (3)

@app.get("/exports/orders.csv")
async def export_orders():
    return StreamingResponse(
        csv_rows(), media_type="text/csv",
        headers={"Content-Disposition": 'attachment; filename="orders.csv"'})   # (4)
text
200 text/csv; charset=utf-8   transfer-encoding: chunked
chunk: 'id,sku,total\r\n1,MUG,450\r\n2,PEN,40\r\n'
chunk: '3,MUG,450\r\n4,PEN,40\r\n'
chunk: '5,MUG,450\r\n'
  • (1) Read the database one page at a time. In a real service this is a keyset-paginated query, or the database driver's server-side cursor, so the database never builds the whole result at once either.
  • (2) Each page becomes one chunk of the response, sent as soon as it is ready.
  • (3) Empty the buffer so memory does not grow.
  • (4) Content-Disposition: attachment makes the browser download the file instead of showing it.

The response has no Content-Length, because the server does not know the total size in advance. Instead HTTP/1.1 uses chunked transfer encoding: each piece is sent with its own length, and an empty piece marks the end. Under ASGI, each yield becomes one http.response.body message with more_body set to true, and the last one with more_body set to false.

Four cautions for streaming in production.

  1. The status code is sent first. Once the first chunk has gone out, the status is fixed at 200. If the database fails halfway, you cannot change it to 500. The client receives a truncated file. Validate everything you can before returning the StreamingResponse, and consider writing a final "row count" line or a checksum so the client can detect truncation.
  2. Resources must stay open while streaming. The database session must live until the last chunk. That is exactly why the default cleanup scope for yield dependencies runs after the response is sent (3.14.4).
  3. Proxies may buffer. Some reverse proxies collect a whole response before forwarding it, which removes the benefit. Disable buffering for streaming routes in the proxy, for example with Nginx's X-Accel-Buffering: no response header.
  4. Very big exports belong in a background job. The job writes the file to object storage, and the user downloads it with a link. Streaming suits exports of seconds to a few minutes, not hours.

For a file that already exists on disk, use FileResponse, which sends it efficiently and sets the length and type for you.

2. Server-sent events: the server pushes, the browser listens ​

The customer's order page should update itself as the order moves along. The browser could ask "any news?" every few seconds, which is called polling. It wastes requests when nothing changed, and is slow when something did. Server-sent events, or SSE, are a simple standard for the opposite direction: the browser opens one ordinary HTTP request, and the server keeps it open and writes events down it whenever it likes.

The format is plain text with the media type text/event-stream. Each event is a few field: value lines followed by a blank line. Browsers have a built-in EventSource object that connects, parses events and reconnects automatically if the connection drops, sending the id of the last event it saw, so the server can resume where it left off.

FastAPI supports SSE directly. An endpoint with response_class=EventSourceResponse can simply yield events.

python
import asyncio
from collections.abc import AsyncIterator
from fastapi import Request
from fastapi.sse import EventSourceResponse, ServerSentEvent

STATUSES = ["paid", "packed", "shipped", "out for delivery", "delivered"]

@app.get("/orders/{order_id}/events", response_class=EventSourceResponse)     # (1)
async def order_events(order_id: int, request: Request) -> AsyncIterator[ServerSentEvent]:
    for number, status in enumerate(STATUSES, start=1):
        if await request.is_disconnected():                                    # (2)
            break
        yield ServerSentEvent(event="status", id=str(number),
                              data={"order_id": order_id, "status": status})   # (3)
        await asyncio.sleep(0.05)                                              # in reality: wait for the next real update
text
content-type: text/event-stream; charset=utf-8

event: status
data: {"order_id": 7, "status": "paid"}
id: 1

event: status
data: {"order_id": 7, "status": "packed"}
id: 2

event: status
data: {"order_id": 7, "status": "shipped"}
id: 3
…
  • (1) The endpoint is an async generator. FastAPI turns each yielded item into the SSE wire format and also sends small keep-alive comments, : ping, during quiet periods, so proxies do not close an idle connection.
  • (2) Stop work promptly when the customer closes the tab. Without this check, the generator would keep running for nobody.
  • (3) event names the kind of event, id lets the browser resume after a reconnect, and data is JSON-encoded for you.

On the page, three lines of JavaScript consume it: const source = new EventSource("/orders/7/events"); source.addEventListener("status", e => show(JSON.parse(e.data)));. Choose SSE when updates flow one way, from server to client, as with order tracking, notifications, live dashboards and streaming text from a language model. It is plain HTTP, so it passes through proxies, works with ordinary authentication headers or cookies, and needs no special server.

3. Uploads that do not fill memory ​

Product managers upload product photos. A careless implementation reads the whole upload into one bytes object, and a few hundred simultaneous 50 MB uploads exhaust the server's memory. FastAPI's UploadFile avoids this, and you can do better still.

python
import os
import tempfile
from fastapi import HTTPException, UploadFile

MAX_BYTES = 5 * 1024 * 1024

@app.post("/products/{sku}/images")
async def upload_image(sku: str, image: UploadFile):
    if image.content_type not in {"image/png", "image/jpeg"}:              # (1)
        raise HTTPException(status_code=415, detail="only PNG or JPEG images")
    size = 0
    with tempfile.NamedTemporaryFile(delete=False, suffix=".img") as out:
        while chunk := await image.read(64 * 1024):                       # (2)
            size += len(chunk)
            if size > MAX_BYTES:                                          # (3)
                out.close()
                os.unlink(out.name)
                raise HTTPException(status_code=413, detail="image larger than 5 MB")
            out.write(chunk)
    return {"sku": sku, "filename": image.filename, "bytes": size}
text
2 MB PNG   → 200 {"sku":"MUG","filename":"mug.png","bytes":2097156}
6 MB PNG   → 413 {"detail":"image larger than 5 MB"}
GIF        → 415 {"detail":"only PNG or JPEG images"}
  • (1) Reject the wrong type early. The content_type is chosen by the client and can be a lie, so before trusting the file, check its first bytes, its "magic number", or open it with an image library. 415 means Unsupported Media Type.
  • (2) Read in 64 KB pieces. The walrus operator (3.13.3) reads and tests in one line, stopping at the empty chunk that means "end of file".
  • (3) Stop as soon as the limit is passed, instead of after receiving the whole file. 413 means Content Too Large.

What is UploadFile underneath? Starlette parses the multipart form as it arrives and writes each file part into a SpooledTemporaryFile: it is kept in memory while small and rolls over to a temporary file on disk once it passes 1 MB. So a 2 MB upload never sat in memory as one block. Your await image.read(...) calls read from that file in a worker thread, so they do not block the event loop. Never use file: bytes = File() for large uploads, because that loads the whole file into memory.

Two production notes. Put a hard limit on request size at the reverse proxy too, such as Nginx's client_max_body_size, so huge bodies are rejected before Python sees a byte. And for large files, let clients upload directly to object storage such as S3, using a short-lived pre-signed URL your API hands out. The API then only receives a small "upload finished" notification and never carries the file at all.

4. WebSockets: two-way, real-time ​

SSE sends one way. The delivery app needs both directions on one connection: the driver's phone sends its position every few seconds, and every customer watching that order receives it immediately. A WebSocket is a long-lived, two-way connection. It starts as an ordinary HTTP request with an Upgrade: websocket header. The server agrees with status 101 Switching Protocols, and from then on either side can send messages at any time, with very little overhead per message. Part 5 describes the protocol itself.

In ASGI, a WebSocket is a connection whose scope["type"] is "websocket", and the app exchanges websocket.receive and websocket.send messages for as long as it stays open (3.14.1). FastAPI wraps that in a WebSocket object.

python
from collections import defaultdict
from fastapi import WebSocket, WebSocketDisconnect

class TrackingHub:
    def __init__(self):
        self.watchers: dict[int, set[WebSocket]] = defaultdict(set)     # (1)

    async def connect(self, order_id: int, websocket: WebSocket):
        await websocket.accept()                                         # (2)
        self.watchers[order_id].add(websocket)

    def disconnect(self, order_id: int, websocket: WebSocket):
        self.watchers[order_id].discard(websocket)

    async def publish(self, order_id: int, message: dict):
        for websocket in list(self.watchers[order_id]):                  # (3)
            try:
                await websocket.send_json(message)
            except Exception:
                self.disconnect(order_id, websocket)                     # (4)

hub = TrackingHub()

@app.websocket("/ws/orders/{order_id}")
async def track(websocket: WebSocket, order_id: int):
    await hub.connect(order_id, websocket)
    try:
        while True:
            message = await websocket.receive_json()                     # (5)
            if message.get("type") == "ping":
                await websocket.send_json({"type": "pong"})
            elif message.get("type") == "driver_location":
                await hub.publish(order_id, {"type": "location",
                                             "lat": message["lat"], "lng": message["lng"]})
    except WebSocketDisconnect:                                          # (6)
        hub.disconnect(order_id, websocket)

A customer's app and the driver's app both connect to order 7.

python
import asyncio
import json
import websockets

async def main():
    async with websockets.connect("ws://127.0.0.1:8000/ws/orders/7") as customer, \
               websockets.connect("ws://127.0.0.1:8000/ws/orders/7") as driver:
        await customer.send(json.dumps({"type": "ping"}))
        print("customer got", await customer.recv())
        await driver.send(json.dumps({"type": "driver_location", "lat": 12.97, "lng": 77.59}))
        print("customer got", await customer.recv())
        print("driver got", await driver.recv())

asyncio.run(main())
text
customer got {"type":"pong"}
customer got {"type":"location","lat":12.97,"lng":77.59}
driver got {"type":"location","lat":12.97,"lng":77.59}
  • (1) For each order, the set of open connections watching it.
  • (2) A WebSocket must be accepted before messages flow. This is the moment to check authentication: read a token from the query string or a cookie, and close with code 1008, "policy violation", if it is invalid. Browsers cannot set an Authorization header on a WebSocket, which is why tokens travel differently here.
  • (3) Copy the set before looping, because a failed send removes items from it.
  • (4) A connection that died without a clean goodbye is cleaned up on the next send.
  • (5) The loop waits for the next message. While it waits, this coroutine costs nothing but a little memory, so one worker can hold thousands of these connections.
  • (6) When the client disconnects, receive_json raises WebSocketDisconnect, and we stop tracking it.

The problem with more than one process ​

TrackingHub keeps its connections in the memory of one process. In production there are several Uvicorn workers on several servers. The driver's phone may be connected to worker A, and the customer's phone to worker B on another machine. Worker A's publish never reaches worker B's sockets.

The standard fix is a shared publish-subscribe channel, such as Redis Pub/Sub or a message broker. Each worker subscribes to the channels for the orders its own clients are watching. publish sends the update to the shared channel instead of directly to sockets, and every worker forwards what it receives to its local connections. 3.14.1 raised this same issue for 5,000 connections, and Part 10 covers pub-sub systems in depth.

Two more production habits for long-lived connections:

  1. Heartbeats. Send a ping every 20 to 30 seconds, and drop connections that stop answering. Mobile networks and load balancers silently cut idle connections.
  2. Graceful deploys. Restarting a worker closes all of its sockets. Clients must reconnect with backoff, and the server should send each reconnecting client the current state, not only future changes.

5. Choosing between them ​

NeedUse
Big download, generated on the flyStreamingResponse
Existing file on diskFileResponse
Server → client live updatesServer-sent events
Both directions, low latencyWebSocket
Large uploadsUploadFile in chunks, or direct to storage
Updates every few minutesPlain polling is fine

The last row matters. A dashboard that refreshes every five minutes does not need any of this machinery. Choose the simplest tool that meets the real latency requirement.

What the interviewer will push on ​

"How would you return a CSV export of ten million rows?" Stream it: an async generator reading the database page by page, wrapped in a StreamingResponse, so memory stays flat and the client receives data immediately. The tell is naming the pitfalls: the status code is fixed once streaming starts, the database session must stay open, and proxies may buffer. For very large exports, a background job writing to object storage.

"SSE or WebSockets?" SSE for one-way server-to-client updates over plain HTTP, with automatic reconnect and resume by event ID. WebSockets for two-way, low-latency messaging. A strong answer mentions that SSE works through ordinary HTTP infrastructure and authentication, while WebSockets need upgrade support and a different way to pass tokens.

"How does FastAPI handle file uploads without exhausting memory?" UploadFile is backed by a spooled temporary file that rolls to disk after 1 MB, and reads run in a thread. Read it in chunks, enforce a size limit as you go, and check the real file type. Mention proxy body limits and pre-signed direct uploads.

"Your WebSocket messages reach some clients but not others in production. Why?" Several worker processes each hold their own connections in memory, so a publish in one process does not reach sockets in another. Use a shared pub-sub channel that all workers subscribe to.

"What happens if the database fails halfway through a streamed response?" The status was already sent as 200, so the client gets a truncated body. Detect it with a trailing count or checksum, validate as much as possible before streaming, and log the failure with the request ID.

Something to volunteer that nobody asks for: browsers cannot send an Authorization header when opening a WebSocket, so authentication usually uses a short-lived token in the query string or a cookie, checked before accept(). Tokens in URLs can end up in proxy logs, so make them single-use and short-lived.

Recall

  • StreamingResponse(async_generator) sends each yield as a chunk, using chunked transfer encoding. Memory stays flat. The status is fixed once streaming starts, the session must stay open (default yield cleanup scope), and proxies may buffer. Use FileResponse for existing files.
  • Server-sent events: text/event-stream, event/data/id lines, browser EventSource with automatic reconnect. FastAPI: response_class=EventSourceResponse and yield ServerSentEvent(...), with keep-alive pings. Check request.is_disconnected().
  • UploadFile is a spooled temporary file that moves to disk after 1 MB. Read in chunks, enforce limits as you read (413), and verify the real type (415). Use proxy limits and pre-signed direct uploads.
  • A WebSocket starts with HTTP Upgrade and a 101 response, then two-way messages. @app.websocket, accept(), receive_json, send_json, WebSocketDisconnect. Authenticate before accept.
  • Connections live in one process. Use publish-subscribe (such as Redis) across workers. Add heartbeats, and reconnect with backoff.

Self-test: Why can a streamed response not turn into a 500 halfway through? What does a browser send when an SSE connection drops and reconnects? Why did a 2 MB upload not sit in memory? Why might a driver's location never reach a customer when there are four workers?

Quiz Bank ​

FoundationalWhat is the difference between StreamingResponse and returning a list from an endpoint?

Returning a list makes FastAPI validate and serialise the whole list, build the full JSON body in memory, and send it with a Content-Length header after everything is ready. StreamingResponse takes an iterator or async iterator and sends each produced chunk immediately using chunked transfer encoding, so memory stays roughly constant and the client starts receiving data at once. The trade-offs are that a streaming response skips response-model validation, cannot change its status code after the first chunk, and needs every resource it reads from to stay open until the end.

AppliedUsers report that a live order-status page stops updating after about a minute behind the company load balancer. What is likely happening, and how do you fix it?

Load balancers and proxies close connections that are idle for longer than their timeout, often 60 seconds. An SSE stream or WebSocket with no traffic during quiet periods looks idle and gets cut. Send periodic keep-alives: SSE comment lines such as : ping every 15 to 30 seconds, which FastAPI's EventSourceResponse does, or WebSocket ping frames. Raise the idle timeout for these routes on the load balancer. Disable response buffering for the stream. On the client, rely on EventSource reconnecting with Last-Event-ID, or implement reconnect with backoff for WebSockets, and have the server resend the current status on reconnect.

InterviewDesign a FastAPI endpoint that accepts a CSV of 2 million product price updates.

Do not process it inside the request. Accept the upload as an UploadFile, stream it in chunks to object storage or a temporary file while enforcing a size limit and checking it looks like CSV. Create an import job record, and return 202 Accepted with the job ID and a status URL. A worker process reads the file line by line with the csv module, validates each row with a Pydantic model, collects invalid rows with their line numbers into an error report, and applies valid updates in batches of a few thousand per transaction. It records progress so the job can resume after a crash. The client polls the status URL or subscribes to SSE progress events. For very large files, give the client a pre-signed URL to upload directly to storage, so the API never carries the bytes.

StaffYou need live order tracking for 200,000 concurrent customers across 30 servers. Sketch the architecture.

Separate the real-time tier from the REST API, so deploys and load in one do not disturb the other. Clients connect with SSE for status changes, since the flow is one-way, and WebSockets only for driver apps that send positions. Each real-time server holds a few thousand connections per worker, with heartbeats and long idle timeouts on the load balancer. Order services publish state changes to a durable event stream such as Kafka, keyed by order ID. A fan-out layer, such as Redis Pub/Sub or a stream consumer per server, delivers each event only to the servers that hold subscribers for that order. Clients authenticate with short-lived tokens before the connection is accepted. On reconnect they send their last event ID, and the server replays recent events from a short-lived store. Throttle driver position updates to a few per second, and send customers smoothed positions rather than every raw point.

Next: the service is feature-complete. 3.14.9 covers what it takes to run it for real: how many worker processes to start, how Uvicorn and Gunicorn fit together behind a proxy, containers, health checks, structured logs, and the performance traps that only appear under real traffic.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Volume I — Technology
Part 0 — Retention & Notation
Part 1 — Foundations of Computation
Part 2 — Operating Systems
2.1 The Kernel
2.2 Processes
2.3 Threads & Scheduling
2.4 Concurrency Under the Hood
2.5 Memory Management
2.6 File Systems
2.7 Input/Output
2.8 Linux for Engineers
2.9 Windows & macOS in Practice
Part 3 — Languages, Compilers & Runtimes
3.6 JavaScript & V8
3.7 TypeScript
3.8 Node.js Internals
3.13 Python & CPython
3.14 FastAPI & Production Python
Part 4 — Data Structures & Algorithms
4.0 Classic Problems — Before NeetCode
4.4 Arrays & Hashing — Problems
4.5 Two Pointers
4.6 Sliding Window
4.8 Stack — Problems
4.9 Linked List — Problems
4.11 Binary Search — Problems
4.13 Trees
4.14 Tree — Problems
4.15 Trie — Problems
4.17 Heap — Problems
4.18 Backtracking — Problems
4.19 Graphs
4.20 Graph — Problems
4.21 Advanced Graph — Problems
4.23 1-D DP — Problems
4.24 2-D DP — Problems
4.26 Greedy — Problems
4.27 Interval — Problems
4.28 Math & Geometry — Problems
4.30 Bit Manipulation — Problems
Part 5 — Networking & the Web
5.3 IP
5.4 Transport
5.6 HTTP
Part 6 — Frontend Engineering
6.1 Browser Internals
6.2 HTML & CSS Mastery
6.3 JavaScript in the Browser
6.4 React, from First Component to Full Shop
6.5 Frameworks & Rendering
6.8 UI Engineering & Patterns
Part 7 — Databases & Storage
7.2 SQL Mastery
7.3 Storage Engine Internals
7.4 Transactions
7.5 NoSQL Taxonomy
7.8 Analytics & Big Data
Part 8 — Security, Identity & Cryptography
8.2 Cryptography
8.3 Protocols of Trust
8.4 Identity: AuthN & AuthZ
8.5 Application Security
8.6 Infrastructure Security
Part 9 — Software Design (LLD)
9.2 OOP Foundations
9.3 Design Principles
9.4 Design Patterns
9.5 Concurrency
9.6 API Design
9.7 LLD Interview Machines
9.9 Node/Express Production Playbook
Part 10 — Distributed Systems & HLD
10.7 Consistency & Consensus
10.8 Messaging & Microservices
10.14 Caching
Building blocks from the design seat
Part 11 — System Design Case Studies
Part 12 — AI & ML Engineering
12.1 What AI Engineering Is
12.2 The Math You Need
12.3 Classical Machine Learning
12.4 Data for ML
12.5 Deep Learning
12.6 Embeddings & Vector Search
12.7 Transformers & LLMs
12.8 Building with LLMs
12.9 Retrieval-Augmented Generation
12.10 Agents
12.11 Adapting Models
12.12 Production AI
12.13 Multimodal AI
12.14 AI Security & Governance
12.15 Case Studies & Capstone
Part 13 — Cloud, DevOps & Infrastructure
13.11 Git, from First Commit to Team Workflow
Part 14 — Containers: Docker & Kubernetes
14.1 Foundations
14.2 Docker
14.3 Kubernetes
14.4 Kubernetes on Azure
Part 15 — The Craft & Career
Nothing picked yet