Appearance
Production Scenarios
Every answer follows the same incident shape: stabilise first, then find the cause, then prevent it. Say that order out loud; interviewers listen for it.
54. The database is at 100% CPU and the site is slow. What do you do?
The short answer
Stabilise: find what is consuming the database right now with pg_stat_activity and pg_stat_statements, cancel runaway queries, pause heavy background jobs and reports, and shed or rate-limit non-critical traffic. Diagnose: look for a new query from a recent deploy, a missing or unused index, an N+1 pattern, a cache that stopped working and sent everything to the database, or a traffic spike. Fix: roll back or add the index concurrently, restore the cache, move reads to replicas. Prevent: query review, slow-query alerts, statement timeouts, and a load test for new queries.
Commands to run first, in PostgreSQL:
sql
-- what is running now, longest first
SELECT pid, now() - query_start AS running, state, left(query, 120)
FROM pg_stat_activity WHERE state <> 'idle' ORDER BY running DESC LIMIT 10;
-- which statements cost the most in total
SELECT left(query, 120), calls, round(total_exec_time) AS total_ms, round(mean_exec_time, 1) AS mean_ms
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;
-- stop one runaway query
SELECT pg_cancel_backend(12345);| Finding | Action |
|---|---|
| One new query dominates after a deploy | Roll back; add the right index with CREATE INDEX CONCURRENTLY |
| Many identical small queries | N+1; batch them; cache |
| Cache hit rate fell to zero | Cache outage or key change; restore it, add a stampede guard |
| A report or export running | Move it to a replica or off-peak |
| Real traffic growth | Read replicas, caching, then capacity |
55. API latency jumped right after a deploy. What do you do?
The short answer
Stabilise first: roll back. A deploy that correlates with a latency jump is the most likely cause, and rolling back is fast and safe if migrations are backward compatible. Then investigate on the bad version, in staging or a canary: compare traces between versions to find the slow span, check for new queries, missing indexes, N+1 loops, a new synchronous dependency, a larger payload, a lost cache, or a blocking CPU task. Fix, then redeploy as a canary with automatic rollback on latency. Prevent it with canary analysis, performance budgets and query review.
The steps, in order:
- Confirm the correlation: latency rose within minutes of the deploy, on instances running the new version.
- Roll back (rollback strategies).
- Confirm recovery on the dashboard.
- Find the cause from traces of the new version: which span grew (troubleshooting latency).
- Fix and redeploy as a canary, comparing p99 with the old version before promoting.
- Write it up: what was missed, and which check would have caught it.
56. A queue's backlog keeps growing. What do you do?
The short answer
A growing backlog means messages arrive faster than consumers finish them. Check which side changed. Producers: a traffic spike, a bug publishing duplicates, a retry storm. Consumers: crashed workers, a slow dependency they call, one poison message failing and retrying forever, a slow query, or a lowered concurrency. Stabilise by scaling consumers if the downstream can take it, moving poison messages to the dead letter queue, and pausing non-essential producers. Then fix the cause, and alert on queue age, the age of the oldest message, not just length.
| Check | Tells you |
|---|---|
| Publish rate versus consume rate | Which side changed |
| Consumer errors and retries | Poison message or failing dependency |
| Consumer processing time per message | A slow query or call inside the handler |
| Number of running consumers | Crashes or scaling limits |
| Age of the oldest message | How late the work is, which is what users feel |
| Dead letter queue size | Messages that will never succeed (dead letter queues) |
Do not scale blindly. If consumers are slow because the database is overloaded, adding consumers makes it worse. Fix the downstream bottleneck first.
57. Customers report being charged twice. What do you do?
The short answer
Stabilise: identify affected payments from the ledger and the provider, refund duplicates quickly, and communicate with affected customers. If the bug is still active, disable the path, such as automatic retries, or roll back. Find the cause: almost always a retry without an idempotency key, a client double-submit, a webhook processed twice, or a timeout treated as failure and charged again. Fix with idempotency keys stored under a unique constraint, idempotent webhook handling by event ID, and "unknown outcome" handling that queries the provider instead of charging again. Prevent with reconciliation alerts that catch duplicates within hours.
The usual chain of events:
- The checkout service calls the payment provider.
- The provider charges the card but the response is slow, and the call times out.
- The service treats the timeout as a failure and retries, or the user clicks "Pay" again.
- The second call charges the card again.
The fix in each layer:
| Layer | Fix |
|---|---|
| Client | Disable the button while paying; send an idempotency key per checkout |
| Your API | Unique constraint on the key; a repeat returns the first result (idempotency keys) |
| Provider call | Pass your idempotency key to the provider too; on timeout, look up the payment instead of retrying blindly |
| Webhooks | Store processed event IDs (idempotent consumers) |
| Detection | Reconciliation compares charges per order and alerts on duplicates (payments design) |
58. Redis, used as a cache, goes down. What happens, and what do you do?
The short answer
If the app was designed well, it falls back to the database and gets slower, not broken. If not, every request now hits the database at once, which can overload it and turn a cache outage into a full outage. Stabilise: make sure cache calls time out quickly instead of hanging, limit how many concurrent requests may go to the database, shed non-critical traffic, and fail over to the Redis replica or restart it. Recover: warm the hottest keys gradually. Prevent: short timeouts and a circuit breaker on cache calls, replicas with automatic failover, stampede protection, and load-testing the "cache cold" case.
| Design choice | Effect during the outage |
|---|---|
| Cache errors treated as misses with a 50 ms timeout | Slower, still working |
| No timeout on cache calls | Requests hang; connection pools fill; outage |
| Concurrency limit on cache-miss database reads | Database protected |
| Stale copies in a local in-memory cache | Hottest pages keep working |
| Warming after recovery, rate-limited | No second stampede (cache warming) |
59. Pods keep restarting in a crash loop. What do you do?
The short answer
Find why the container exits. kubectl describe pod shows the last state and reason, such as OOMKilled or an exit code; kubectl logs --previous shows the output of the crashed run. Common causes: out of memory for its limit, a missing config or secret making startup fail validation, a failing liveness probe killing a slow-starting or overloaded app, a dependency unreachable at startup, or a bad image. Stabilise by rolling back if it began with a deploy. Then fix: raise the memory limit or fix the leak, add the missing config, use a startup probe and a liveness check that does not depend on other services.
| Symptom | Cause | Fix |
|---|---|---|
Last State: OOMKilled | Memory above the limit | Fix the leak, or raise the limit; set Node's heap below it |
| Exit code 1, config error in logs | Missing env var or secret | Add it; config validation made it obvious (config) |
Liveness probe failed events | Probe too strict or checks the database | Startup probe; cheap liveness (probes) |
| Fails connecting to a dependency at boot | Dependency down or not yet ready | Retry with backoff at startup instead of exiting |
ImagePullBackOff | Wrong tag or registry credentials | Fix the tag or the pull secret |
60. A critical third-party API, such as a payment or shipping provider, is down. What do you do?
The short answer
Contain the blast radius: circuit breakers open so requests fail fast instead of piling up; timeouts and bulkheads keep the rest of the system healthy. Degrade gracefully: queue work that can wait, such as shipping labels; show clear messages where it cannot, such as "card payments are temporarily unavailable"; switch to a secondary provider if you have one. Communicate on a status page. Recover by draining queued work at a controlled rate once the provider returns, and reconciling anything with an unknown outcome. Prevention is designing for this from the start.
| Provider | Fallback |
|---|---|
| Shipping quote | Flat-rate estimate; correct later |
| Shipping label | Queue the label request; process when it returns |
| Email or SMS | Queue; second provider |
| Payment | Second provider if the business needs it; otherwise a clear message, no silent retries |
| Address lookup | Let users type the address manually |
Patterns behind it: dependency outages and circuit breakers.
61. Traffic suddenly rises tenfold. What do you do?
The short answer
First decide whether it is real users or an attack: look at the sources, user agents, endpoints and whether conversion rises too. For an attack, block or challenge at the CDN or WAF and tighten rate limits. For real traffic, scale the stateless tiers immediately, make sure autoscaling is not capped, protect the database with caching and by shedding non-critical work, enable load shedding and a waiting room if needed, and serve stale cached pages rather than errors. Afterwards, raise capacity limits, pre-scale for known events, and run a load test at that level.
Order of actions:
- Classify: attack, bot scraping, or real demand.
- At the edge: WAF rules, bot challenges, per-IP rate limits for attacks; CDN caching for real demand.
- Scale out: app instances and workers; check autoscaling maximums and cloud quotas.
- Protect the core: turn off expensive features, such as recommendations; shed low-priority endpoints (load shedding).
- Watch the database pool, CPU and replicas; it is usually what breaks.
- Afterwards: capacity review and a load test.
62. Data disagrees between two services. What do you do?
The short answer
In event-driven systems this means an event was lost, processed twice, processed out of order, or failed in a consumer. Contain: find the scope, which records and since when, by comparing the two systems. Repair: replay events from the broker or the outbox for the affected range, or run a one-off reconciliation that copies the truth from the owning service. Fix the cause: a dual write without an outbox, a non-idempotent consumer, missing ordering by key, or a consumer that swallowed an error. Prevent: outbox, idempotent consumers, version checks, DLQ alerts, and a scheduled reconciliation job that detects drift early.
| Root cause | Fix |
|---|---|
| Database write succeeded, publish failed | Transactional outbox (outbox) |
| Event applied twice | Processed-message table |
| Events applied out of order | Partition by entity key; ignore older versions |
| Consumer error swallowed | Fail loudly; dead letter queue with alerts |
| Nobody noticed for weeks | Nightly reconciliation comparing counts and checksums |
63. A TLS certificate expired and users see errors. What do you do?
The short answer
Stabilise: issue or install a valid certificate immediately, from your ACME client, cloud certificate manager or CA, with the full chain, and reload the proxy or load balancer. Verify from outside with a TLS check of the expiry date and chain. Then find why automation failed: a renewal job that silently errored, a DNS or HTTP challenge that broke, a manually issued certificate nobody tracked, or a service that loaded the new file but was never reloaded. Prevent: automate every certificate, alert at 30 and 14 days before expiry from independent monitoring, and keep an inventory of all certificates, including internal and partner ones.
The rotation procedure is in certificate rotation; for SAML certificates, which break every customer's SSO, see SAML certificates.