Skip to content

Interview PrepScenarios · Advanced

Production Scenarios ​

Every answer follows the same incident shape: stabilise first, then find the cause, then prevent it. Say that order out loud; interviewers listen for it.

54. The database is at 100% CPU and the site is slow. What do you do? ​

The short answer

Stabilise: find what is consuming the database right now with pg_stat_activity and pg_stat_statements, cancel runaway queries, pause heavy background jobs and reports, and shed or rate-limit non-critical traffic. Diagnose: look for a new query from a recent deploy, a missing or unused index, an N+1 pattern, a cache that stopped working and sent everything to the database, or a traffic spike. Fix: roll back or add the index concurrently, restore the cache, move reads to replicas. Prevent: query review, slow-query alerts, statement timeouts, and a load test for new queries.

Commands to run first, in PostgreSQL:

sql
-- what is running now, longest first
SELECT pid, now() - query_start AS running, state, left(query, 120)
FROM pg_stat_activity WHERE state <> 'idle' ORDER BY running DESC LIMIT 10;

-- which statements cost the most in total
SELECT left(query, 120), calls, round(total_exec_time) AS total_ms, round(mean_exec_time, 1) AS mean_ms
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;

-- stop one runaway query
SELECT pg_cancel_backend(12345);
FindingAction
One new query dominates after a deployRoll back; add the right index with CREATE INDEX CONCURRENTLY
Many identical small queriesN+1; batch them; cache
Cache hit rate fell to zeroCache outage or key change; restore it, add a stampede guard
A report or export runningMove it to a replica or off-peak
Real traffic growthRead replicas, caching, then capacity

55. API latency jumped right after a deploy. What do you do? ​

The short answer

Stabilise first: roll back. A deploy that correlates with a latency jump is the most likely cause, and rolling back is fast and safe if migrations are backward compatible. Then investigate on the bad version, in staging or a canary: compare traces between versions to find the slow span, check for new queries, missing indexes, N+1 loops, a new synchronous dependency, a larger payload, a lost cache, or a blocking CPU task. Fix, then redeploy as a canary with automatic rollback on latency. Prevent it with canary analysis, performance budgets and query review.

The steps, in order:

  1. Confirm the correlation: latency rose within minutes of the deploy, on instances running the new version.
  2. Roll back (rollback strategies).
  3. Confirm recovery on the dashboard.
  4. Find the cause from traces of the new version: which span grew (troubleshooting latency).
  5. Fix and redeploy as a canary, comparing p99 with the old version before promoting.
  6. Write it up: what was missed, and which check would have caught it.

56. A queue's backlog keeps growing. What do you do? ​

The short answer

A growing backlog means messages arrive faster than consumers finish them. Check which side changed. Producers: a traffic spike, a bug publishing duplicates, a retry storm. Consumers: crashed workers, a slow dependency they call, one poison message failing and retrying forever, a slow query, or a lowered concurrency. Stabilise by scaling consumers if the downstream can take it, moving poison messages to the dead letter queue, and pausing non-essential producers. Then fix the cause, and alert on queue age, the age of the oldest message, not just length.

CheckTells you
Publish rate versus consume rateWhich side changed
Consumer errors and retriesPoison message or failing dependency
Consumer processing time per messageA slow query or call inside the handler
Number of running consumersCrashes or scaling limits
Age of the oldest messageHow late the work is, which is what users feel
Dead letter queue sizeMessages that will never succeed (dead letter queues)

Do not scale blindly. If consumers are slow because the database is overloaded, adding consumers makes it worse. Fix the downstream bottleneck first.

57. Customers report being charged twice. What do you do? ​

The short answer

Stabilise: identify affected payments from the ledger and the provider, refund duplicates quickly, and communicate with affected customers. If the bug is still active, disable the path, such as automatic retries, or roll back. Find the cause: almost always a retry without an idempotency key, a client double-submit, a webhook processed twice, or a timeout treated as failure and charged again. Fix with idempotency keys stored under a unique constraint, idempotent webhook handling by event ID, and "unknown outcome" handling that queries the provider instead of charging again. Prevent with reconciliation alerts that catch duplicates within hours.

The usual chain of events:

  1. The checkout service calls the payment provider.
  2. The provider charges the card but the response is slow, and the call times out.
  3. The service treats the timeout as a failure and retries, or the user clicks "Pay" again.
  4. The second call charges the card again.

The fix in each layer:

LayerFix
ClientDisable the button while paying; send an idempotency key per checkout
Your APIUnique constraint on the key; a repeat returns the first result (idempotency keys)
Provider callPass your idempotency key to the provider too; on timeout, look up the payment instead of retrying blindly
WebhooksStore processed event IDs (idempotent consumers)
DetectionReconciliation compares charges per order and alerts on duplicates (payments design)

58. Redis, used as a cache, goes down. What happens, and what do you do? ​

The short answer

If the app was designed well, it falls back to the database and gets slower, not broken. If not, every request now hits the database at once, which can overload it and turn a cache outage into a full outage. Stabilise: make sure cache calls time out quickly instead of hanging, limit how many concurrent requests may go to the database, shed non-critical traffic, and fail over to the Redis replica or restart it. Recover: warm the hottest keys gradually. Prevent: short timeouts and a circuit breaker on cache calls, replicas with automatic failover, stampede protection, and load-testing the "cache cold" case.

Design choiceEffect during the outage
Cache errors treated as misses with a 50 ms timeoutSlower, still working
No timeout on cache callsRequests hang; connection pools fill; outage
Concurrency limit on cache-miss database readsDatabase protected
Stale copies in a local in-memory cacheHottest pages keep working
Warming after recovery, rate-limitedNo second stampede (cache warming)

59. Pods keep restarting in a crash loop. What do you do? ​

The short answer

Find why the container exits. kubectl describe pod shows the last state and reason, such as OOMKilled or an exit code; kubectl logs --previous shows the output of the crashed run. Common causes: out of memory for its limit, a missing config or secret making startup fail validation, a failing liveness probe killing a slow-starting or overloaded app, a dependency unreachable at startup, or a bad image. Stabilise by rolling back if it began with a deploy. Then fix: raise the memory limit or fix the leak, add the missing config, use a startup probe and a liveness check that does not depend on other services.

SymptomCauseFix
Last State: OOMKilledMemory above the limitFix the leak, or raise the limit; set Node's heap below it
Exit code 1, config error in logsMissing env var or secretAdd it; config validation made it obvious (config)
Liveness probe failed eventsProbe too strict or checks the databaseStartup probe; cheap liveness (probes)
Fails connecting to a dependency at bootDependency down or not yet readyRetry with backoff at startup instead of exiting
ImagePullBackOffWrong tag or registry credentialsFix the tag or the pull secret

60. A critical third-party API, such as a payment or shipping provider, is down. What do you do? ​

The short answer

Contain the blast radius: circuit breakers open so requests fail fast instead of piling up; timeouts and bulkheads keep the rest of the system healthy. Degrade gracefully: queue work that can wait, such as shipping labels; show clear messages where it cannot, such as "card payments are temporarily unavailable"; switch to a secondary provider if you have one. Communicate on a status page. Recover by draining queued work at a controlled rate once the provider returns, and reconciling anything with an unknown outcome. Prevention is designing for this from the start.

ProviderFallback
Shipping quoteFlat-rate estimate; correct later
Shipping labelQueue the label request; process when it returns
Email or SMSQueue; second provider
PaymentSecond provider if the business needs it; otherwise a clear message, no silent retries
Address lookupLet users type the address manually

Patterns behind it: dependency outages and circuit breakers.

61. Traffic suddenly rises tenfold. What do you do? ​

The short answer

First decide whether it is real users or an attack: look at the sources, user agents, endpoints and whether conversion rises too. For an attack, block or challenge at the CDN or WAF and tighten rate limits. For real traffic, scale the stateless tiers immediately, make sure autoscaling is not capped, protect the database with caching and by shedding non-critical work, enable load shedding and a waiting room if needed, and serve stale cached pages rather than errors. Afterwards, raise capacity limits, pre-scale for known events, and run a load test at that level.

Order of actions:

  1. Classify: attack, bot scraping, or real demand.
  2. At the edge: WAF rules, bot challenges, per-IP rate limits for attacks; CDN caching for real demand.
  3. Scale out: app instances and workers; check autoscaling maximums and cloud quotas.
  4. Protect the core: turn off expensive features, such as recommendations; shed low-priority endpoints (load shedding).
  5. Watch the database pool, CPU and replicas; it is usually what breaks.
  6. Afterwards: capacity review and a load test.

62. Data disagrees between two services. What do you do? ​

The short answer

In event-driven systems this means an event was lost, processed twice, processed out of order, or failed in a consumer. Contain: find the scope, which records and since when, by comparing the two systems. Repair: replay events from the broker or the outbox for the affected range, or run a one-off reconciliation that copies the truth from the owning service. Fix the cause: a dual write without an outbox, a non-idempotent consumer, missing ordering by key, or a consumer that swallowed an error. Prevent: outbox, idempotent consumers, version checks, DLQ alerts, and a scheduled reconciliation job that detects drift early.

Root causeFix
Database write succeeded, publish failedTransactional outbox (outbox)
Event applied twiceProcessed-message table
Events applied out of orderPartition by entity key; ignore older versions
Consumer error swallowedFail loudly; dead letter queue with alerts
Nobody noticed for weeksNightly reconciliation comparing counts and checksums

63. A TLS certificate expired and users see errors. What do you do? ​

The short answer

Stabilise: issue or install a valid certificate immediately, from your ACME client, cloud certificate manager or CA, with the full chain, and reload the proxy or load balancer. Verify from outside with a TLS check of the expiry date and chain. Then find why automation failed: a renewal job that silently errored, a DNS or HTTP challenge that broke, a manually issued certificate nobody tracked, or a service that loaded the new file but was never reloaded. Prevent: automate every certificate, alert at 30 and 14 days before expiry from independent monitoring, and keep an inventory of all certificates, including internal and partner ones.

The rotation procedure is in certificate rotation; for SAML certificates, which break every customer's SSO, see SAML certificates.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
System Design
The method
Case studies
Auth designs
Scenarios
Other topics
Nothing picked yet