Skip to content

Interview PrepCloud

Reliability, Disaster Recovery and Cost ​

583. What is the difference between high availability, fault tolerance, and disaster recovery? ​

The short answer

High availability means the system keeps working through common failures, such as a machine or a zone going down, with at most a brief interruption, because there are spare copies ready to take over. Fault tolerance goes further: users notice nothing at all, because every part runs in duplicate at the same time. Disaster recovery is the plan for big events that high availability does not cover, such as losing a whole region, data corruption or ransomware: how you restore service and data, and how fast.

High availabilityFault toleranceDisaster recovery
HandlesMachine and zone failuresAny single failure, invisiblyRegion loss, corruption, attacks
User impactSeconds to a minute, maybe some errorsNoneMinutes to hours, by design
HowSeveral copies, health checks, automatic failoverEverything doubled and running in lock-stepBackups, a standby region, runbooks
CostModerateHighDepends on how fast you need to recover
ExampleThree app copies across zones behind a load balancerAircraft control systems; some payment hardwareRestoring into another region from backups

An everyday picture. A car with a spare tyre is highly available: you stop for ten minutes and carry on. A lorry with double rear tyres is fault tolerant: one bursts and it keeps driving. Breakdown cover that brings a replacement car after a crash is disaster recovery.

Why both HA and DR are needed. High availability copies everything instantly, including mistakes. If someone runs DELETE FROM orders by accident, the standby deletes the orders too, within milliseconds. Only backups and point-in-time restore, which are part of disaster recovery, bring them back (question 577).

584. What are RTO and RPO? ​

The short answer

The recovery time objective, RTO, is how long the service may be down after a disaster before it must be working again. The recovery point objective, RPO, is how much data, measured in time, you can afford to lose: an RPO of 15 minutes means losing at most the last 15 minutes of changes. The business sets both, based on the cost of downtime and lost data, and they decide which recovery strategy, and how much spending, is needed.

timelast good copydisasterservice backdata lostmust be under the RPOdowntimemust be under the RTO
Looking back, looking forward. RPO looks back from the disaster: how much recent data may be lost. RTO looks forward: how long until the service works again.

Two separate questions:

  • RPO, looking back: "if the disaster happened now, how many minutes of orders could we lose?" It is decided by how often data is copied. Nightly backups mean an RPO of up to 24 hours. Continuous replication can bring it to seconds.
  • RTO, looking forward: "how long until customers can order again?" It is decided by how much is ready in advance, and how practised the recovery is.

Examples for the shop:

SystemRTORPOWhy
Checkout and orders15 minutesNear zeroEvery minute down loses sales; lost orders are lost money
Product catalogue1 hour24 hoursRarely changes; can be re-imported
Analytics1 day1 dayNobody loses money if reports are late
CI system4 hours1 dayDevelopers wait, customers do not (question 307)

Lower costs more. Moving from an RTO of four hours to four minutes usually means a second region running warm, permanently. Ask the business what an hour of downtime costs, and let that decide.

A measured number beats a promised one. Your real RTO is what a recovery drill achieves, not what the plan says (question 588).

585. What are the main disaster recovery strategies? ​

The short answer

There are four, from cheapest and slowest to most expensive and fastest. Backup and restore keeps only backups in another region and rebuilds everything after a disaster, taking hours. Pilot light keeps the data replicated and the core pieces ready but switched off, taking tens of minutes. Warm standby runs a smaller, working copy of everything that is scaled up on failover, taking minutes. Active-active runs full production in two or more regions at once, recovering almost instantly.

backup and restoreonly backupsin region Bhourspilot lightdata replicated,servers offtens of minuteswarm standbysmall working copy,scaled up on failoverminutesactive-activefull productionin both regionsnear zerofaster recovery, higher cost, more complexitythe times shown are typical recovery times
Pay now, or wait later. Each step to the right keeps more running in the second region, so recovery is faster and the monthly bill is higher.
StrategyRunning in region BTypical RTOTypical RPOCost
Backup and restoreBackups onlyHoursHours, up to the backup intervalLowest
Pilot lightReplicated data; infrastructure defined but offTens of minutesMinutes or lessLow
Warm standbyA scaled-down full copyMinutesSeconds to minutesMedium
Active-activeFull production, serving usersNear zeroNear zeroHighest

What makes each work:

  1. Backup and restore needs infrastructure as code, so the whole environment can be recreated in a new region from scratch, plus backups copied there (question 595).
  2. Pilot light keeps the database replicated to region B. On failover, you start the app servers from code and images that are already there.
  3. Warm standby runs a small copy, say 10% of capacity, that already works. Failover means scaling it up and moving traffic.
  4. Active-active needs the application to handle users in both regions at once, which is the hard part: data written in two places must not conflict.

Choose from the RTO and RPO, per system, not one strategy for everything (question 584). The checkout may need warm standby while analytics only needs backups.

586. When is multi-zone enough, and when do you need multiple regions? ​

The short answer

Spreading across availability zones in one region handles the failures that actually happen most: a machine, a rack or a whole data centre going down. It is cheap, simple, and the network between zones is fast enough for synchronous database copies. Multiple regions are needed when the business cannot accept a full-region outage, when laws require copies far apart, or when users around the world need low latency. Multi-region costs much more and makes data consistency hard, so do it only for a clear reason.

Multi-zoneMulti-region
Protects againstMachine, rack, data-centre failuresA whole region failing, and wide disasters
Network delay between copiesAbout a millisecond or twoTens to hundreds of milliseconds
Synchronous database copyPracticalUsually too slow; asynchronous instead
Cost increaseSmallLarge: duplicated capacity and data transfer
ComplexityLow; most managed services do it for youHigh: data, failover, testing
Also givesLower latency for distant users, data residency

Why the distance matters for data. A database that waits for a copy in another zone 2 milliseconds away barely notices. Waiting for a copy on another continent, 80 milliseconds away, on every write, makes everything slow. So multi-region designs usually copy asynchronously, which means a failover can lose the last few seconds of data. That shapes the RPO (question 584).

Signs you need multi-region:

  • Regulators or contracts require recovery from a region loss.
  • An hour of total outage would cost more than years of running a standby region.
  • Customers on several continents need fast responses.

Signs you do not, yet: a young product, one market, and no zone redundancy yet. Get multi-zone right first: it covers most real failures for a fraction of the cost.

587. How would you design a highly available web application in the cloud? ​

The short answer

Remove every single point of failure, layer by layer. Put a CDN and a global or regional load balancer in front. Run at least two, better three, copies of each stateless service across zones, with health checks and autoscaling. Keep state out of the app servers, in a zone-redundant managed database with automatic failover, a managed cache and object storage. Use queues between services so one slow part does not stop the rest. Then add backups in another region and practise failing over.

Layer by layer, for the shop:

LayerChoiceRemoves the risk of
DNSManaged DNS with health checksOne name server failing
EdgeCDN with WAFOverload, attacks, distant users
Load balancerZone-redundant managed load balancerOne balancer failing
App3 or more copies across 3 zones, autoscaledOne machine or zone failing
SessionsIn a shared cache or tokens, not in memoryUsers logged out when a copy dies
DatabaseManaged, with a standby in another zoneDatabase server or zone failing
FilesZone-redundant object storageA disk failing
Background workQueues and workersA slow service blocking checkout
RecoveryBackups and copies in a second regionRegion loss, corruption

Rules that make it work:

  1. Stateless app servers. Any copy can serve any user, so losing one is harmless.
  2. Health checks that mean something, so the load balancer stops sending traffic to broken copies quickly (question 401).
  3. Timeouts and retries with backoff on every call between services, so a slow dependency does not freeze everything.
  4. Graceful degradation. If recommendations are down, show the product page without them, rather than an error page.
  5. Enough spare capacity to lose one zone. With three zones, each must handle half the load, not a third, during an outage of one.
  6. Rolling deployments with automatic rollback, because releases cause more outages than hardware (question 280).

The same thinking in Kubernetes: replicas spread across zones with topology spread constraints, PodDisruptionBudgets, and a zone-redundant control plane (questions 396 and 400).

588. What is chaos engineering, and how do you start safely? ​

The short answer

Chaos engineering means breaking things on purpose, in a controlled way, to find weaknesses before real failures find them. You state what should happen, for example "if one zone fails, checkout keeps working", then cause that failure for real, such as stopping instances or adding network delay, and watch. Start small and safe: in a non-production environment, with one clear question, a limited blast radius, a way to stop instantly, and people watching.

The idea came from Netflix. In 2011 they built Chaos Monkey, a tool that randomly shut down production servers during working hours. Engineers then had to build services that survived it, and real failures stopped being surprises.

How to run an experiment, step by step:

  1. Define normal with a measurable signal, such as orders per minute and error rate.
  2. Make a hypothesis: "if one of three app copies dies, error rate stays below 0.1%".
  3. Limit the blast radius: one service, one environment, a small share of traffic.
  4. Prepare the stop button: how to end the experiment in seconds.
  5. Inject the failure: stop an instance, block a dependency, add latency, fill a disk.
  6. Watch and compare against normal.
  7. Fix what you learn, then repeat, gradually closer to production.

Failures worth practising:

FailureWhat it reveals
One instance or pod killedWhether health checks and replacement work
A whole zone unavailableWhether spare capacity is really enough
Database failoverWhether apps reconnect, and how long errors last
A dependency slow, not downMissing timeouts, which are worse than failures
A region failover drillThe real RTO, against the planned one

Tools: AWS Fault Injection Service, Azure Chaos Studio, and in Kubernetes, Chaos Mesh or LitmusChaos.

Game days are planned sessions where a team runs such a scenario together, often a disaster recovery drill, and practises the response. The lessons, such as a missing alert or an outdated runbook, matter as much as the technical result.

589. How do you fail over traffic and data between regions? ​

The short answer

Traffic moves with a global entry point: a global load balancer or CDN, such as Azure Front Door or AWS Global Accelerator, or DNS failover with health checks, which sends users to the healthy region. Data is the hard part: the database in the standby region must be a replica that can be promoted to primary, accepting that asynchronous copies may lose the last seconds of writes. The whole procedure, from detection to promotion and traffic shift, should be written down, automated where safe, and practised.

global entry pointhealth checks both regionsregion A, primaryapp, full capacitydatabase primaryregion B, standbyapp, warm, scales updatabase replica, promotable① normal③ on failure② async copy
Move the users, promote the data. ① Normally, all traffic goes to region A. ② The database copies changes to region B a moment later. ③ On failure, the entry point sends users to region B, and region B's replica becomes the primary.

Moving traffic, two ways:

Global load balancer or anycastDNS failover
ExamplesAzure Front Door, AWS Global Accelerator, Google global load balancerRoute 53 failover records, Azure Traffic Manager
Switch timeSecondsMinutes: TTL plus client caching (question 540)
Works forHTTP; some for any TCP or UDPAnything that uses DNS

Anycast means the same IP address is announced from many locations worldwide, so users connect to the nearest one, and traffic can be moved without any DNS change.

Moving the data, step by step:

  1. Confirm the primary region is really down, or unusable, not just a blip. Failing over by mistake is expensive.
  2. Stop writes to the old primary if it is still partly reachable, so two primaries do not accept writes at once. Two primaries accepting different writes is called split brain, and cleaning it up is painful.
  3. Promote the replica in region B to primary.
  4. Point the apps in region B at it, scale them up, and move traffic.
  5. Later, fail back in a calm, planned way: rebuild region A as a replica, catch up, then switch.

Automatic or manual? Traffic failover for stateless apps can be automatic. Database promotion is often a human decision, made in minutes, because a wrong automatic promotion can cause split brain or data loss. Managed features such as Azure SQL failover groups, Aurora Global Database and Cosmos DB multi-region support make each step faster and safer.

590. Why do data transfer and egress costs matter, and how do you reduce them? ​

The short answer

Clouds usually let data in for free, but charge for data going out to the internet, called egress, and often for data crossing between zones and regions. NAT gateways and some private links charge per gigabyte processed too. These charges grow quietly with traffic and are easy to miss in a design. Reduce them by serving users from a CDN, keeping chatty services in the same zone or region, using private endpoints or gateway endpoints for cloud services, compressing data, and watching transfer costs by tag.

Where the per-gigabyte charges usually are:

TrafficUsually charged?
From the internet into the cloudNo
Within one zone, private addressesNo
Between zones in a regionOften, a small charge each way, such as in AWS
Between regionsYes
Out to the internetYes, the largest per-GB rate
Through a NAT gatewayYes, a processing charge on top
Through private endpointsYes, a small charge per GB

Prices change and differ by provider and region, so check the current price pages before relying on a number.

A worked example. A service in zone 1 calls a database in zone 2, sending 5 TB a month each way. At $0.01 per GB in each direction, that is 10,000 GB × $0.01 = $100 a month. Small. Now a log pipeline sends 200 TB a month across zones: $2,000 a month for traffic nobody planned.

Ways to reduce it:

  1. CDN for user downloads. CDN egress is often cheaper than egress from servers, and cached content does not touch your origin (question 541).
  2. Gateway endpoints for object storage traffic, which in AWS avoid NAT processing charges completely (question 537).
  3. Keep chatty services close. Prefer same-zone routing where your platform supports it, such as Kubernetes topology-aware routing.
  4. Compress API responses and logs.
  5. Filter logs before shipping them. Debug logs from every pod, sent to another region, add up quickly.
  6. Watch it. Break the bill down by "data transfer" and by tag, so a jump is noticed within days (question 593).

591. What is FinOps, and how do you optimise cloud costs? ​

The short answer

FinOps is the practice of managing cloud spending as a shared responsibility of engineering, finance and the business, with clear visibility, ownership and regular decisions about value. In practice, tag everything so each cost has an owner, show teams their own spending, and set budgets and anomaly alerts. Then work the main savings levers. Delete waste, right-size, and scale down when idle. Use commitment discounts for steady usage and spot for interruptible work. Choose cheaper storage tiers and architectures.

The three FinOps phases, a cycle that repeats:

  1. Inform: who spends what, and on what. Tags, cost reports per team, cost per unit of business, such as cost per order.
  2. Optimise: act on the biggest items first.
  3. Operate: make it routine: budgets, alerts, reviews, and cost as part of design decisions.

The savings levers, roughly in order of ease:

LeverExampleTypical effort
Delete wasteUnattached disks, old snapshots, idle load balancers, forgotten test environmentsLow
Turn off when idleDev and test environments off at night and weekendsLow
Right-sizeA machine at 10% CPU moved to half the sizeLow to medium
Storage tiers and lifecycleOld logs to archive (question 572)Low
Commitment discountsReservations or savings plans for steady usage (question 592)Low, but a long commitment
Spot capacityCI runners and batch jobs (question 561)Medium
AutoscalingScale with real demand, including to zeroMedium
Architecture changesFewer cross-region calls, caching, a cheaper service typeHigh

Unit economics is the most useful number. "We spend $80,000 a month" means little. "Each order costs us 4 cents of cloud, down from 6 cents last quarter" tells you whether spending is healthy as the business grows.

Make cost visible to engineers. Show each team its own spend, near its own dashboards. People who can see what their service costs make better choices without being told.

592. How do reservations, savings plans, and committed-use discounts work? ​

The short answer

You promise to use, and pay for, a certain amount of compute for one or three years, and in return get a large discount, often 30 to 70% off the on-demand price. Reservations commit to specific machine types or services in a region. Savings plans commit to an amount of spending per hour, and apply flexibly across machine types and services. Use them only for the steady base load you are confident will last, and cover the variable part with on-demand and spot.

On-demandReservationSavings planSpot
CommitmentNone1 or 3 years, specific type1 or 3 years, spend per hourNone
DiscountNoneHighMedium to highHighest
FlexibilityFullLow to mediumHighCan be taken away
Use forVariable and unknown loadSteady, predictable machines and databasesSteady compute that may change shapeInterruptible work

Names in each cloud: AWS has Reserved Instances and Savings Plans. Azure has Reservations and the Azure savings plan for compute. Google Cloud has committed use discounts, plus automatic sustained use discounts for machines that run most of the month.

How to size a commitment:

  1. Look at the last few months of usage, per hour.
  2. Find the base: the level you never dropped below.
  3. Commit to most of that base, not all of it, and not the peaks.
  4. Review every few months, and add more as the base grows.

An example. The shop's compute never drops below $40 an hour, and peaks at $90 on sale days. A savings plan for $35 an hour covers most of the steady part at a discount. The rest stays on-demand, autoscaled, and spot for batch jobs.

The risk. You pay for the commitment whether you use it or not. Committing to machines you then migrate away from, such as moving to serverless, can waste money for years. Commitment discounts also apply to databases and caches, which are often steadier than compute.

593. The cloud bill doubled this month; how would you investigate? ​

The short answer

Use the cost tools to find where the increase is, step by step: by service, then by account or subscription, then by region, then by resource and tag, comparing day by day to find when it started. Match that date with deployments, configuration changes and traffic. Typical causes are a forgotten large resource, autoscaling stuck at maximum, a log or data-transfer explosion, a new environment, or a security incident such as crypto mining. Fix the cause, then add budgets and anomaly alerts so the next one is caught in a day.

Step by step:

  1. Narrow the service. In AWS Cost Explorer or Azure Cost Management, group by service. Which line jumped: compute, storage, data transfer, logging?
  2. Find the day. Switch to a daily view. A step change on one day points to a change. A steady climb points to growth or a leak.
  3. Narrow the owner. Group by account or subscription, region, and tag. Untagged costs are a finding in themselves (question 528).
  4. Find the resource. Look at the top resources by cost in that service and region.
  5. Match with changes. What was deployed or changed that day? Check the audit log and the deployment history (question 555).
  6. Check for an incident. Large machines in regions you never use, especially GPU machines, are a classic sign of stolen credentials used for crypto mining (question 558).

Common causes, and what they look like:

CauseSign
Forgotten resourcesA test cluster or big database running since a demo
Autoscaling stuck highInstance count at maximum, low CPU each
Logging explosionLog ingestion cost jumped after a debug flag was left on
Data transferA new cross-region or internet-facing flow
NAT gateway processingLarge traffic to cloud services going through NAT instead of endpoints
Expired commitmentSame usage, but a reservation ended and prices went to on-demand
Crypto miningMany large machines in unusual regions

Prevent the next one:

  • Budgets with alerts at 50%, 80% and 100% of the expected monthly spend.
  • Anomaly detection, such as AWS Cost Anomaly Detection or Azure cost anomaly alerts, which spot unusual daily spending.
  • Guardrails on allowed regions and machine sizes (question 556).
  • Automatic clean-up of temporary environments after a set time.

594. How do you calculate availability, and what do the "nines" mean? ​

The short answer

Availability is the share of time a service works, written as a percentage, such as 99.9%, "three nines". Each extra nine cuts allowed downtime by ten times: 99.9% allows about 43 minutes a month, and 99.99% about 4 minutes. When services depend on each other in a chain, multiply their availabilities, so the whole is less available than any single part. When independent copies back each other up, the chance that all fail together is tiny, so availability rises sharply.

Allowed downtime:

AvailabilityPer yearPer month, 30 days
99%3.65 days7.2 hours
99.9%8.76 hours43.2 minutes
99.95%4.38 hours21.6 minutes
99.99%52.6 minutes4.3 minutes
99.999%5.26 minutes26 seconds

The arithmetic, with the shop as an example:

in series: every part must workload balancer 99.99%app 99.9%database 99.95%= about 99.84%in parallel: one copy is enoughcopy A 99%copy B 99%traffic= 99.99%
Chains lose nines, copies add them. In a chain, every part must work, so the total is lower than the weakest part. With independent copies, the system fails only when all of them fail at once.

In series, every part must work, so multiply:

text
0.9999 × 0.999 × 0.9995 ≈ 0.9984   →  about 99.84%

That is about 70 minutes of possible downtime a month, worse than any single part.

In parallel, with independent copies, the system fails only if every copy fails at the same moment. The chance one copy is down is 1%, or 0.01. The chance both are down is 0.01 × 0.01 = 0.0001:

text
1 − (0.01 × 0.01) = 0.9999   →  99.99%

The catch: "independent". Two copies in the same zone, or running the same bad release, fail together. Real independence means different zones, and rolling out changes to one copy at a time.

SLAs are not the same as real availability. A provider's SLA, service level agreement, is a promise with a refund, usually service credits, if it is broken. It is not a guarantee that it never happens. Your own targets, SLOs, should be based on what you measure and what users need (question 630).

Each extra nine is expensive. Going from 99.9% to 99.99% often means multi-zone everything, automatic failover, careful releases and on-call staff. Going to 99.999% often means multi-region active-active. Ask whether users need it.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet