Appearance
3.14.9 — Running FastAPI in Production
On a laptop, uvicorn main:app --reload is the whole deployment. In production the same app runs behind a load balancer, on several machines, inside containers, restarted by an orchestrator, watched by dashboards, and upgraded several times a day while customers are using it. A different set of questions appears. How many processes should run? How does the app learn the customer's real IP address and that the connection was HTTPS? What happens to requests in flight during a deploy? How does an engineer find one failed checkout among a million log lines? Why is p99 latency fine at 100 requests per second and terrible at 300? This page answers each of those questions for a FastAPI service. Part 13 covers containers, cloud and observability in general. This page is the Python and FastAPI side.
1. Processes: how many workers?
One Uvicorn process runs one event loop on one CPU core, because of the GIL (3.13.13). To use a machine with several cores, run several worker processes, each a full copy of your app with its own event loop, its own lifespan, its own connection pools and its own memory.
There are three common ways to run them.
- Uvicorn's own supervisor.
uvicorn shop.main:app --workers 4starts four workers and restarts any that crash. - Gunicorn as the supervisor. Gunicorn, the long-standing WSGI server, can manage Uvicorn workers. Its process management is battle-tested, and many teams already know its settings.
- One worker per container, with the orchestrator, such as Kubernetes, running many containers. This is the most common modern choice, because the orchestrator already restarts failed containers and spreads load across machines. Running a supervisor inside each container would duplicate that job.
How many workers? Start from what the work looks like.
- I/O-bound async services, the typical FastAPI case, need roughly one worker per CPU core. Each worker already overlaps thousands of waits, so more processes than cores only adds switching.
- Services with plain
defendpoints use the 40-thread pool per worker (3.14.3), so they may benefit from a few more workers than cores. - CPU-heavy endpoints cap throughput at the number of cores, whatever you do. Move that work out of the request path (3.13.14).
Then measure. Memory is often the real limit: each worker loads the whole app, so a service using 400 MB per worker cannot run 16 workers on a 4 GB machine. And remember the connection pool arithmetic from 3.14.6: every worker multiplies database connections.
2. Behind a reverse proxy
A FastAPI process is almost never exposed to the internet directly. A reverse proxy or load balancer, such as Nginx, a cloud load balancer, or a Kubernetes ingress, sits in front. It terminates TLS, spreads traffic across instances, limits request sizes, and protects the app from slow or malformed clients.
That creates a problem. From the app's point of view, every request now comes from the proxy, over plain HTTP. request.client.host is the proxy's address, and request.url.scheme is http, even though the customer used HTTPS. Redirects then point to http://, rate limits treat every customer as one IP, and logs lose the real client.
Proxies solve this by adding headers: X-Forwarded-For with the client's IP address, and X-Forwarded-Proto with the original scheme. Uvicorn reads them when you ask it to, but only from addresses you trust, because any client could send those headers itself.
text
uvicorn shop.main:app --host 0.0.0.0 --port 8000 \
--proxy-headers --forwarded-allow-ips "10.0.0.0/8" # trust only the internal networkIf the app is served under a path prefix, such as https://example.com/api/, pass --root-path /api so the generated documentation and redirects include the prefix.
3. A production container
Here is a container image for the service, built with uv (3.13.11) from its lockfile.
dockerfile
# ---- build stage ----
FROM python:3.14-slim AS build
# (1) the uv binary, copied from its official image
COPY --from=ghcr.io/astral-sh/uv:0.11 /uv /bin/uv
WORKDIR /app
# (2) compile .pyc files now, so containers start faster
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
COPY pyproject.toml uv.lock ./
# (3) dependencies only, cached until the lockfile changes
RUN uv sync --locked --no-dev --no-install-project
COPY src ./src
# (4) now the project itself, as a normal package
RUN uv sync --locked --no-dev --no-editable
# ---- runtime stage ----
FROM python:3.14-slim
# (5) never run as root
RUN useradd --create-home --uid 10001 app
WORKDIR /app
# (6) only the finished virtual environment crosses over
COPY --from=build --chown=app:app /app/.venv /app/.venv
# (7) logs are written immediately
ENV PATH="/app/.venv/bin:$PATH" PYTHONUNBUFFERED=1
USER app
EXPOSE 8000
# (8) exec form: Uvicorn is the main process and receives SIGTERM
CMD ["uvicorn", "order_service.main:app", "--host", "0.0.0.0", "--port", "8000", "--proxy-headers", "--forwarded-allow-ips", "10.0.0.0/8"](1)Copy the uv binary from its official image. Dockerfile comments must sit on their own lines, which is why the markers here are above each instruction.(2)Compile.pycfiles at build time (3.13.1), so containers start faster.(3)Install only the dependencies, from the lockfile, before copying the source code. Docker caches each step, so this slow step is reused untilpyproject.tomloruv.lockchanges, not on every code edit.(4)Install the project itself into the virtual environment as a normal, non-editable package.(5)Never run as root. If an attacker finds a bug in the app, they should not also own the container.(6)The runtime image receives only the finished virtual environment. Build tools and caches stay behind in the build stage, so the final image is smaller and has less to attack.(7)PYTHONUNBUFFERED=1writes log lines immediately instead of buffering them, so nothing is lost when a container is killed.(8)The exec form, a JSON list, makes Uvicorn the container's main process. The orchestrator's stop signal,SIGTERM, then reaches Uvicorn directly. With the shell form, a shell sits in between, may not pass the signal on, and every deploy kills in-flight requests abruptly.
4. Deploys without dropping requests
A graceful shutdown is what lets you deploy during the busiest hour. When Uvicorn receives SIGTERM, it does this.
- Stop accepting new connections.
- Let requests already in progress finish, for up to
--timeout-graceful-shutdownseconds. - Run the lifespan's shutdown code (3.14.5), closing pools and clients.
- Exit.
The orchestrator must cooperate: stop sending new traffic to the instance before it shuts down, and allow enough time for in-flight requests to finish. Long-lived connections, such as WebSockets and SSE streams (3.14.8), will be cut, so clients must reconnect.
Database migrations run before new code starts and must work with the old code still running, the expand-and-contract rule from 3.14.6.
5. Health checks
The orchestrator needs to ask each instance two different questions, and they need two different endpoints.
python
from fastapi import FastAPI, Response
from sqlalchemy import text
@app.get("/healthz") # (1) liveness
async def liveness():
return {"status": "ok"}
@app.get("/readyz") # (2) readiness
async def readiness(response: Response):
try:
async with SessionFactory() as session:
await session.execute(text("SELECT 1"))
except Exception:
response.status_code = 503
return {"status": "database unavailable"}
return {"status": "ready"}(1)Liveness: "is this process alive and its event loop responding?" It must be cheap and must not check dependencies. If the database is down and liveness checks it, the orchestrator restarts every instance in a loop, which fixes nothing and makes recovery slower.(2)Readiness: "should this instance receive traffic right now?" It may check critical dependencies. A failing readiness check takes the instance out of the load balancer, without restarting it.
A blocked event loop makes even the liveness check slow, so a service with a CPU-heavy async def endpoint can be killed by its own health checks under load. That is one more reason never to block the loop.
6. Logs that answer questions
print and plain text logs do not scale to many instances. Production services write structured logs: one JSON object per line, with consistent fields, which a log system can search and filter. The request ID from 3.14.5 goes into every line.
python
import contextvars
import json
import logging
import sys
from datetime import datetime, timezone
request_id_var = contextvars.ContextVar("request_id", default="-")
class JsonFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
entry = {
"time": datetime.fromtimestamp(record.created, timezone.utc).isoformat(timespec="milliseconds"),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
"request_id": request_id_var.get(), # (1)
}
entry.update(getattr(record, "fields", {})) # (2)
if record.exc_info:
entry["exception"] = self.formatException(record.exc_info) # (3)
return json.dumps(entry)
def configure_logging(level: str = "INFO") -> None: # (4)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
root = logging.getLogger()
root.handlers[:] = [handler]
root.setLevel(level)
configure_logging()
logger = logging.getLogger("shop.orders")
request_id_var.set("req-42")
logger.info("order created", extra={"fields": {"order_id": "ORD-9", "total_paise": 49900}})text
{"time": "2026-10-21T16:26:14.669+00:00", "level": "INFO", "logger": "shop.orders", "message": "order created", "request_id": "req-42", "order_id": "ORD-9", "total_paise": 49900}(1)Every line carries the current request's ID, read from the context variable, without any function having to pass it along.(2)Structured fields are passed throughextraand become searchable keys, so "show every log line for ORD-9" is one query.(3)Tracebacks are kept inside the same JSON line, so a multi-line traceback is not split into fifty unrelated log entries.(4)Configure once, in the lifespan or at start-up (3.13.16). Write to standard output and let the platform collect it.
Never log tokens, passwords, full card numbers or entire request bodies from sensitive endpoints (3.14.7).
Logs answer "what happened to this request?". Two more signals complete the picture. Metrics answer "how is the whole service doing?": request rate, error rate and latency percentiles per endpoint, plus pool usage and event-loop lag. Traces answer "where did this slow request spend its time?" by following it across services. OpenTelemetry provides ready-made instrumentation for FastAPI, SQLAlchemy and httpx. Part 13 covers all three.
7. Performance traps that only appear under load
Most FastAPI performance problems are not about FastAPI. They are one of these.
1. A blocked event loop. One synchronous call inside an async def, such as requests, a synchronous database driver, time.sleep, a big JSON parse or image resizing, serialises the whole worker (3.14.3). The symptom is that p99 latency explodes while CPU usage looks low. You can catch it with a small watcher started in the lifespan.
python
import asyncio
import logging
import time
logger = logging.getLogger("shop.loop")
async def watch_event_loop(interval: float = 0.5, warn_after: float = 0.1):
while True:
start = time.perf_counter()
await asyncio.sleep(interval) # (1)
lag = time.perf_counter() - start - interval # (2)
if lag > warn_after:
logger.warning("event loop blocked for %.0f ms", lag * 1000)text
WARNING shop.loop: event loop blocked for 416 ms ← logged while a handler called time.sleep(0.8)(1)Ask to sleep for half a second.(2)If the loop was blocked, the sleep overshoots. The overshoot is the time some other code held the loop.
Start it with asyncio.create_task(watch_event_loop()) in the lifespan, and cancel it at shutdown. Publish the lag as a metric, and alert when it rises.
2. Pool exhaustion. Every request waits for a database connection that never comes, because the pool is too small or connections are held too long, for example across a slow external API call inside an open transaction. Keep transactions short, never call other services while holding one, and watch pool wait time.
3. No timeouts on outgoing calls. A slow payment provider without a timeout holds every request open until the whole service is stuck. Every outgoing call needs a timeout: httpx.AsyncClient(timeout=5.0), database statement timeouts, and asyncio.timeout around groups of calls (3.13.15). Retries need backoff and a limit, or they multiply the load on a struggling dependency.
4. N+1 queries. Fine with test data, terrible with real customers (3.14.6).
5. Doing too much per request. Validation and serialisation of huge payloads, re-validating trusted data, and large responses without pagination all cost CPU in every worker. Declaring a return type lets FastAPI serialise through Pydantic's fast Rust path. Paginate everything. Cache what can be cached.
6. Missing fast components. Install uvicorn[standard], which brings uvloop and httptools, the faster event loop and HTTP parser from 3.14.1.
Measure before changing anything. py-spy, a sampling profiler, can attach to a running worker and show where its time goes, with no code change and almost no overhead. A load test with a tool such as Locust or k6, run against realistic data, shows how latency changes as traffic grows. Optimise the top line of the profile, never a guess.
8. A production checklist
Before the service takes real traffic:
- Workers sized by cores and memory, and connection pools sized against the database limit.
- Proxy headers trusted only from the load balancer's network.
- The container runs as a non-root user, with the exec-form command and graceful shutdown timeouts matched to the orchestrator.
- Separate liveness and readiness endpoints.
- Structured JSON logs with request IDs, and no secrets in them.
- Metrics for rate, errors, latency, pool usage and loop lag, with alerts.
- Timeouts on every outgoing call, and limits on request body size, page size and concurrency.
debug=False, documentation pages disabled or protected if the API is private, and error responses that never include tracebacks.- Settings from the environment, validated at start-up (3.14.2), with secrets in a secret manager.
- Migrations reviewed and backward-compatible.
What the interviewer will push on
"How many Uvicorn workers would you run?" Roughly one per core for an async I/O-bound service, adjusted by measurement. Memory per worker and database connections per worker are often the real limits. In Kubernetes, usually one worker per container and scale replicas. The tell is explaining why more workers than cores does not help an async service.
"What is the difference between liveness and readiness?" Liveness asks whether to restart the process and must not check dependencies. Readiness asks whether to send it traffic and may check them. Checking the database in liveness causes restart storms during a database outage.
"How does the app know the client's real IP behind a load balancer?" X-Forwarded-For and X-Forwarded-Proto, honoured by Uvicorn with --proxy-headers, only from trusted proxy addresses in --forwarded-allow-ips. Trusting everyone lets clients forge their IP, which breaks rate limits and audit logs.
"p99 latency is high but CPU is low. What do you check?" A blocked event loop, using a loop-lag metric or py-spy. Pool waits. Slow downstream calls without timeouts. Then N+1 queries and lock waits in the database.
"How do you deploy without dropping requests?" SIGTERM reaches Uvicorn through an exec-form command, the instance is removed from load balancing first, in-flight requests finish within the graceful timeout, and the lifespan closes resources. Migrations are backward-compatible.
Something to volunteer that nobody asks for: a shell-form CMD in a Dockerfile puts a shell between the orchestrator and Uvicorn, so SIGTERM may never arrive, and every deploy hard-kills requests after the grace period. The exec-form JSON list fixes it. It is a one-line bug that shows up as "random 502s during deploys".
Recall
- One Uvicorn worker = one process, one event loop, one core. Run several with
--workers, Gunicorn, or one per container scaled by the orchestrator. About one per core for async I/O services, limited by memory and database connections. - Behind a reverse proxy:
--proxy-headerswith--forwarded-allow-ipsset to the proxy network, and--root-pathfor path prefixes. - Container: multi-stage build, uv from the lockfile with dependencies cached first, a non-root user,
PYTHONUNBUFFERED=1, and an exec-formCMDsoSIGTERMreaches Uvicorn. - Graceful shutdown: stop accepting, finish in-flight requests, run lifespan shutdown. Long-lived connections must reconnect.
- Liveness stays cheap with no dependencies. Readiness may check the database and returns 503 to leave the load balancer.
- Structured logs: one JSON line, request ID from a context variable, fields via
extra, tracebacks inside the line, no secrets. Plus metrics and traces through OpenTelemetry. - Traps: blocked event loop (watch loop lag), pool exhaustion, missing timeouts, N+1, oversized payloads. Install
uvicorn[standard]. Profile withpy-spyand load-test before optimising.
Self-test: Why does adding workers beyond the number of cores not help an async service? What breaks if --forwarded-allow-ips is * on a server exposed to the internet? Why must liveness not check the database? How does a sleep-and-measure loop detect a blocked event loop?
Quiz Bank
FoundationalWhy does each Uvicorn worker have its own database connection pool, and why does that matter?
Each worker is a separate operating-system process with its own memory. Objects created in the lifespan, including the SQLAlchemy engine and its pool, exist once per process and cannot be shared across processes. So the total number of database connections is the pool size times the number of workers times the number of instances. A modest pool of 15 with 4 workers on 6 instances is 360 connections, more than many databases allow by default. Size pools from the database's limit downwards, or use a connection pooler such as PgBouncer.
AppliedEvery deploy causes a burst of 502 errors for about 30 seconds. What would you check?
Check that the container's command uses exec form, so Uvicorn receives SIGTERM and shuts down gracefully instead of being killed after the grace period. Check that the orchestrator removes the instance from the load balancer before sending SIGTERM, often with a short pre-stop delay, and that --timeout-graceful-shutdown is shorter than the orchestrator's termination grace period. Check readiness: new instances must not receive traffic until the lifespan has finished, pools are open and the readiness endpoint passes. Check that keep-alive timeouts on the proxy are shorter than Uvicorn's, so the proxy never reuses a connection the server just closed. Finally, check whether a migration ran that the old code could not handle.
InterviewA FastAPI service's p99 latency jumps from 80 ms to 4 s at peak traffic while CPU stays at 35%. How do you investigate?
Low CPU with high tail latency means requests are waiting, not computing. First check event-loop lag. A blocked loop from a synchronous call inside async def serialises every request on the worker. A lag metric or py-spy dump on a live worker shows it immediately. Next, check database pool wait time and active connections. If requests queue for connections, look for long transactions or external calls made while a session is open. Then check downstream latency and timeouts, since a slow dependency without a timeout holds requests open. Correlate slow requests by request ID in the structured logs and traces to see where the time went. Fix the specific cause: async libraries or to_thread, shorter transactions, timeouts with limited retries, and more workers only once the per-worker bottleneck is gone.
StaffDesign the observability for a fleet of FastAPI services so that on-call engineers can resolve a checkout incident in minutes.
Standardise one shared library that every service imports. It configures JSON logging with request and trace IDs from context variables, OpenTelemetry tracing for FastAPI, SQLAlchemy and httpx with trace context propagated across services, and metrics for request rate, errors and latency percentiles per route, plus pool usage, event-loop lag and downstream call latency. Define service-level objectives for key journeys such as checkout, and alert on error-budget burn rather than raw thresholds. Every error response carries the request ID, so support can hand an incident a direct link into traces and logs. Dashboards per service follow one template, starting from the user journey and drilling down to dependencies. Runbooks link from each alert. Log retention and sampling are set by cost, with full traces kept for errors and slow requests.
Next: every piece is now in place. 3.14.10 assembles them into one complete order service, walks a single checkout through every layer from the socket to the database and back, and finishes with the cross-chapter questions a senior Python interview actually asks.