Appearance
Reliability, Disaster Recovery and Cost
583. What is the difference between high availability, fault tolerance, and disaster recovery?
The short answer
High availability means the system keeps working through common failures, such as a machine or a zone going down, with at most a brief interruption, because there are spare copies ready to take over. Fault tolerance goes further: users notice nothing at all, because every part runs in duplicate at the same time. Disaster recovery is the plan for big events that high availability does not cover, such as losing a whole region, data corruption or ransomware: how you restore service and data, and how fast.
| High availability | Fault tolerance | Disaster recovery | |
|---|---|---|---|
| Handles | Machine and zone failures | Any single failure, invisibly | Region loss, corruption, attacks |
| User impact | Seconds to a minute, maybe some errors | None | Minutes to hours, by design |
| How | Several copies, health checks, automatic failover | Everything doubled and running in lock-step | Backups, a standby region, runbooks |
| Cost | Moderate | High | Depends on how fast you need to recover |
| Example | Three app copies across zones behind a load balancer | Aircraft control systems; some payment hardware | Restoring into another region from backups |
An everyday picture. A car with a spare tyre is highly available: you stop for ten minutes and carry on. A lorry with double rear tyres is fault tolerant: one bursts and it keeps driving. Breakdown cover that brings a replacement car after a crash is disaster recovery.
Why both HA and DR are needed. High availability copies everything instantly, including mistakes. If someone runs DELETE FROM orders by accident, the standby deletes the orders too, within milliseconds. Only backups and point-in-time restore, which are part of disaster recovery, bring them back (question 577).
584. What are RTO and RPO?
The short answer
The recovery time objective, RTO, is how long the service may be down after a disaster before it must be working again. The recovery point objective, RPO, is how much data, measured in time, you can afford to lose: an RPO of 15 minutes means losing at most the last 15 minutes of changes. The business sets both, based on the cost of downtime and lost data, and they decide which recovery strategy, and how much spending, is needed.
Two separate questions:
- RPO, looking back: "if the disaster happened now, how many minutes of orders could we lose?" It is decided by how often data is copied. Nightly backups mean an RPO of up to 24 hours. Continuous replication can bring it to seconds.
- RTO, looking forward: "how long until customers can order again?" It is decided by how much is ready in advance, and how practised the recovery is.
Examples for the shop:
| System | RTO | RPO | Why |
|---|---|---|---|
| Checkout and orders | 15 minutes | Near zero | Every minute down loses sales; lost orders are lost money |
| Product catalogue | 1 hour | 24 hours | Rarely changes; can be re-imported |
| Analytics | 1 day | 1 day | Nobody loses money if reports are late |
| CI system | 4 hours | 1 day | Developers wait, customers do not (question 307) |
Lower costs more. Moving from an RTO of four hours to four minutes usually means a second region running warm, permanently. Ask the business what an hour of downtime costs, and let that decide.
A measured number beats a promised one. Your real RTO is what a recovery drill achieves, not what the plan says (question 588).
585. What are the main disaster recovery strategies?
The short answer
There are four, from cheapest and slowest to most expensive and fastest. Backup and restore keeps only backups in another region and rebuilds everything after a disaster, taking hours. Pilot light keeps the data replicated and the core pieces ready but switched off, taking tens of minutes. Warm standby runs a smaller, working copy of everything that is scaled up on failover, taking minutes. Active-active runs full production in two or more regions at once, recovering almost instantly.
| Strategy | Running in region B | Typical RTO | Typical RPO | Cost |
|---|---|---|---|---|
| Backup and restore | Backups only | Hours | Hours, up to the backup interval | Lowest |
| Pilot light | Replicated data; infrastructure defined but off | Tens of minutes | Minutes or less | Low |
| Warm standby | A scaled-down full copy | Minutes | Seconds to minutes | Medium |
| Active-active | Full production, serving users | Near zero | Near zero | Highest |
What makes each work:
- Backup and restore needs infrastructure as code, so the whole environment can be recreated in a new region from scratch, plus backups copied there (question 595).
- Pilot light keeps the database replicated to region B. On failover, you start the app servers from code and images that are already there.
- Warm standby runs a small copy, say 10% of capacity, that already works. Failover means scaling it up and moving traffic.
- Active-active needs the application to handle users in both regions at once, which is the hard part: data written in two places must not conflict.
Choose from the RTO and RPO, per system, not one strategy for everything (question 584). The checkout may need warm standby while analytics only needs backups.
586. When is multi-zone enough, and when do you need multiple regions?
The short answer
Spreading across availability zones in one region handles the failures that actually happen most: a machine, a rack or a whole data centre going down. It is cheap, simple, and the network between zones is fast enough for synchronous database copies. Multiple regions are needed when the business cannot accept a full-region outage, when laws require copies far apart, or when users around the world need low latency. Multi-region costs much more and makes data consistency hard, so do it only for a clear reason.
| Multi-zone | Multi-region | |
|---|---|---|
| Protects against | Machine, rack, data-centre failures | A whole region failing, and wide disasters |
| Network delay between copies | About a millisecond or two | Tens to hundreds of milliseconds |
| Synchronous database copy | Practical | Usually too slow; asynchronous instead |
| Cost increase | Small | Large: duplicated capacity and data transfer |
| Complexity | Low; most managed services do it for you | High: data, failover, testing |
| Also gives | Lower latency for distant users, data residency |
Why the distance matters for data. A database that waits for a copy in another zone 2 milliseconds away barely notices. Waiting for a copy on another continent, 80 milliseconds away, on every write, makes everything slow. So multi-region designs usually copy asynchronously, which means a failover can lose the last few seconds of data. That shapes the RPO (question 584).
Signs you need multi-region:
- Regulators or contracts require recovery from a region loss.
- An hour of total outage would cost more than years of running a standby region.
- Customers on several continents need fast responses.
Signs you do not, yet: a young product, one market, and no zone redundancy yet. Get multi-zone right first: it covers most real failures for a fraction of the cost.
587. How would you design a highly available web application in the cloud?
The short answer
Remove every single point of failure, layer by layer. Put a CDN and a global or regional load balancer in front. Run at least two, better three, copies of each stateless service across zones, with health checks and autoscaling. Keep state out of the app servers, in a zone-redundant managed database with automatic failover, a managed cache and object storage. Use queues between services so one slow part does not stop the rest. Then add backups in another region and practise failing over.
Layer by layer, for the shop:
| Layer | Choice | Removes the risk of |
|---|---|---|
| DNS | Managed DNS with health checks | One name server failing |
| Edge | CDN with WAF | Overload, attacks, distant users |
| Load balancer | Zone-redundant managed load balancer | One balancer failing |
| App | 3 or more copies across 3 zones, autoscaled | One machine or zone failing |
| Sessions | In a shared cache or tokens, not in memory | Users logged out when a copy dies |
| Database | Managed, with a standby in another zone | Database server or zone failing |
| Files | Zone-redundant object storage | A disk failing |
| Background work | Queues and workers | A slow service blocking checkout |
| Recovery | Backups and copies in a second region | Region loss, corruption |
Rules that make it work:
- Stateless app servers. Any copy can serve any user, so losing one is harmless.
- Health checks that mean something, so the load balancer stops sending traffic to broken copies quickly (question 401).
- Timeouts and retries with backoff on every call between services, so a slow dependency does not freeze everything.
- Graceful degradation. If recommendations are down, show the product page without them, rather than an error page.
- Enough spare capacity to lose one zone. With three zones, each must handle half the load, not a third, during an outage of one.
- Rolling deployments with automatic rollback, because releases cause more outages than hardware (question 280).
The same thinking in Kubernetes: replicas spread across zones with topology spread constraints, PodDisruptionBudgets, and a zone-redundant control plane (questions 396 and 400).
588. What is chaos engineering, and how do you start safely?
The short answer
Chaos engineering means breaking things on purpose, in a controlled way, to find weaknesses before real failures find them. You state what should happen, for example "if one zone fails, checkout keeps working", then cause that failure for real, such as stopping instances or adding network delay, and watch. Start small and safe: in a non-production environment, with one clear question, a limited blast radius, a way to stop instantly, and people watching.
The idea came from Netflix. In 2011 they built Chaos Monkey, a tool that randomly shut down production servers during working hours. Engineers then had to build services that survived it, and real failures stopped being surprises.
How to run an experiment, step by step:
- Define normal with a measurable signal, such as orders per minute and error rate.
- Make a hypothesis: "if one of three app copies dies, error rate stays below 0.1%".
- Limit the blast radius: one service, one environment, a small share of traffic.
- Prepare the stop button: how to end the experiment in seconds.
- Inject the failure: stop an instance, block a dependency, add latency, fill a disk.
- Watch and compare against normal.
- Fix what you learn, then repeat, gradually closer to production.
Failures worth practising:
| Failure | What it reveals |
|---|---|
| One instance or pod killed | Whether health checks and replacement work |
| A whole zone unavailable | Whether spare capacity is really enough |
| Database failover | Whether apps reconnect, and how long errors last |
| A dependency slow, not down | Missing timeouts, which are worse than failures |
| A region failover drill | The real RTO, against the planned one |
Tools: AWS Fault Injection Service, Azure Chaos Studio, and in Kubernetes, Chaos Mesh or LitmusChaos.
Game days are planned sessions where a team runs such a scenario together, often a disaster recovery drill, and practises the response. The lessons, such as a missing alert or an outdated runbook, matter as much as the technical result.
589. How do you fail over traffic and data between regions?
The short answer
Traffic moves with a global entry point: a global load balancer or CDN, such as Azure Front Door or AWS Global Accelerator, or DNS failover with health checks, which sends users to the healthy region. Data is the hard part: the database in the standby region must be a replica that can be promoted to primary, accepting that asynchronous copies may lose the last seconds of writes. The whole procedure, from detection to promotion and traffic shift, should be written down, automated where safe, and practised.
Moving traffic, two ways:
| Global load balancer or anycast | DNS failover | |
|---|---|---|
| Examples | Azure Front Door, AWS Global Accelerator, Google global load balancer | Route 53 failover records, Azure Traffic Manager |
| Switch time | Seconds | Minutes: TTL plus client caching (question 540) |
| Works for | HTTP; some for any TCP or UDP | Anything that uses DNS |
Anycast means the same IP address is announced from many locations worldwide, so users connect to the nearest one, and traffic can be moved without any DNS change.
Moving the data, step by step:
- Confirm the primary region is really down, or unusable, not just a blip. Failing over by mistake is expensive.
- Stop writes to the old primary if it is still partly reachable, so two primaries do not accept writes at once. Two primaries accepting different writes is called split brain, and cleaning it up is painful.
- Promote the replica in region B to primary.
- Point the apps in region B at it, scale them up, and move traffic.
- Later, fail back in a calm, planned way: rebuild region A as a replica, catch up, then switch.
Automatic or manual? Traffic failover for stateless apps can be automatic. Database promotion is often a human decision, made in minutes, because a wrong automatic promotion can cause split brain or data loss. Managed features such as Azure SQL failover groups, Aurora Global Database and Cosmos DB multi-region support make each step faster and safer.
590. Why do data transfer and egress costs matter, and how do you reduce them?
The short answer
Clouds usually let data in for free, but charge for data going out to the internet, called egress, and often for data crossing between zones and regions. NAT gateways and some private links charge per gigabyte processed too. These charges grow quietly with traffic and are easy to miss in a design. Reduce them by serving users from a CDN, keeping chatty services in the same zone or region, using private endpoints or gateway endpoints for cloud services, compressing data, and watching transfer costs by tag.
Where the per-gigabyte charges usually are:
| Traffic | Usually charged? |
|---|---|
| From the internet into the cloud | No |
| Within one zone, private addresses | No |
| Between zones in a region | Often, a small charge each way, such as in AWS |
| Between regions | Yes |
| Out to the internet | Yes, the largest per-GB rate |
| Through a NAT gateway | Yes, a processing charge on top |
| Through private endpoints | Yes, a small charge per GB |
Prices change and differ by provider and region, so check the current price pages before relying on a number.
A worked example. A service in zone 1 calls a database in zone 2, sending 5 TB a month each way. At $0.01 per GB in each direction, that is 10,000 GB × $0.01 = $100 a month. Small. Now a log pipeline sends 200 TB a month across zones: $2,000 a month for traffic nobody planned.
Ways to reduce it:
- CDN for user downloads. CDN egress is often cheaper than egress from servers, and cached content does not touch your origin (question 541).
- Gateway endpoints for object storage traffic, which in AWS avoid NAT processing charges completely (question 537).
- Keep chatty services close. Prefer same-zone routing where your platform supports it, such as Kubernetes topology-aware routing.
- Compress API responses and logs.
- Filter logs before shipping them. Debug logs from every pod, sent to another region, add up quickly.
- Watch it. Break the bill down by "data transfer" and by tag, so a jump is noticed within days (question 593).
591. What is FinOps, and how do you optimise cloud costs?
The short answer
FinOps is the practice of managing cloud spending as a shared responsibility of engineering, finance and the business, with clear visibility, ownership and regular decisions about value. In practice, tag everything so each cost has an owner, show teams their own spending, and set budgets and anomaly alerts. Then work the main savings levers. Delete waste, right-size, and scale down when idle. Use commitment discounts for steady usage and spot for interruptible work. Choose cheaper storage tiers and architectures.
The three FinOps phases, a cycle that repeats:
- Inform: who spends what, and on what. Tags, cost reports per team, cost per unit of business, such as cost per order.
- Optimise: act on the biggest items first.
- Operate: make it routine: budgets, alerts, reviews, and cost as part of design decisions.
The savings levers, roughly in order of ease:
| Lever | Example | Typical effort |
|---|---|---|
| Delete waste | Unattached disks, old snapshots, idle load balancers, forgotten test environments | Low |
| Turn off when idle | Dev and test environments off at night and weekends | Low |
| Right-size | A machine at 10% CPU moved to half the size | Low to medium |
| Storage tiers and lifecycle | Old logs to archive (question 572) | Low |
| Commitment discounts | Reservations or savings plans for steady usage (question 592) | Low, but a long commitment |
| Spot capacity | CI runners and batch jobs (question 561) | Medium |
| Autoscaling | Scale with real demand, including to zero | Medium |
| Architecture changes | Fewer cross-region calls, caching, a cheaper service type | High |
Unit economics is the most useful number. "We spend $80,000 a month" means little. "Each order costs us 4 cents of cloud, down from 6 cents last quarter" tells you whether spending is healthy as the business grows.
Make cost visible to engineers. Show each team its own spend, near its own dashboards. People who can see what their service costs make better choices without being told.
592. How do reservations, savings plans, and committed-use discounts work?
The short answer
You promise to use, and pay for, a certain amount of compute for one or three years, and in return get a large discount, often 30 to 70% off the on-demand price. Reservations commit to specific machine types or services in a region. Savings plans commit to an amount of spending per hour, and apply flexibly across machine types and services. Use them only for the steady base load you are confident will last, and cover the variable part with on-demand and spot.
| On-demand | Reservation | Savings plan | Spot | |
|---|---|---|---|---|
| Commitment | None | 1 or 3 years, specific type | 1 or 3 years, spend per hour | None |
| Discount | None | High | Medium to high | Highest |
| Flexibility | Full | Low to medium | High | Can be taken away |
| Use for | Variable and unknown load | Steady, predictable machines and databases | Steady compute that may change shape | Interruptible work |
Names in each cloud: AWS has Reserved Instances and Savings Plans. Azure has Reservations and the Azure savings plan for compute. Google Cloud has committed use discounts, plus automatic sustained use discounts for machines that run most of the month.
How to size a commitment:
- Look at the last few months of usage, per hour.
- Find the base: the level you never dropped below.
- Commit to most of that base, not all of it, and not the peaks.
- Review every few months, and add more as the base grows.
An example. The shop's compute never drops below $40 an hour, and peaks at $90 on sale days. A savings plan for $35 an hour covers most of the steady part at a discount. The rest stays on-demand, autoscaled, and spot for batch jobs.
The risk. You pay for the commitment whether you use it or not. Committing to machines you then migrate away from, such as moving to serverless, can waste money for years. Commitment discounts also apply to databases and caches, which are often steadier than compute.
593. The cloud bill doubled this month; how would you investigate?
The short answer
Use the cost tools to find where the increase is, step by step: by service, then by account or subscription, then by region, then by resource and tag, comparing day by day to find when it started. Match that date with deployments, configuration changes and traffic. Typical causes are a forgotten large resource, autoscaling stuck at maximum, a log or data-transfer explosion, a new environment, or a security incident such as crypto mining. Fix the cause, then add budgets and anomaly alerts so the next one is caught in a day.
Step by step:
- Narrow the service. In AWS Cost Explorer or Azure Cost Management, group by service. Which line jumped: compute, storage, data transfer, logging?
- Find the day. Switch to a daily view. A step change on one day points to a change. A steady climb points to growth or a leak.
- Narrow the owner. Group by account or subscription, region, and tag. Untagged costs are a finding in themselves (question 528).
- Find the resource. Look at the top resources by cost in that service and region.
- Match with changes. What was deployed or changed that day? Check the audit log and the deployment history (question 555).
- Check for an incident. Large machines in regions you never use, especially GPU machines, are a classic sign of stolen credentials used for crypto mining (question 558).
Common causes, and what they look like:
| Cause | Sign |
|---|---|
| Forgotten resources | A test cluster or big database running since a demo |
| Autoscaling stuck high | Instance count at maximum, low CPU each |
| Logging explosion | Log ingestion cost jumped after a debug flag was left on |
| Data transfer | A new cross-region or internet-facing flow |
| NAT gateway processing | Large traffic to cloud services going through NAT instead of endpoints |
| Expired commitment | Same usage, but a reservation ended and prices went to on-demand |
| Crypto mining | Many large machines in unusual regions |
Prevent the next one:
- Budgets with alerts at 50%, 80% and 100% of the expected monthly spend.
- Anomaly detection, such as AWS Cost Anomaly Detection or Azure cost anomaly alerts, which spot unusual daily spending.
- Guardrails on allowed regions and machine sizes (question 556).
- Automatic clean-up of temporary environments after a set time.
594. How do you calculate availability, and what do the "nines" mean?
The short answer
Availability is the share of time a service works, written as a percentage, such as 99.9%, "three nines". Each extra nine cuts allowed downtime by ten times: 99.9% allows about 43 minutes a month, and 99.99% about 4 minutes. When services depend on each other in a chain, multiply their availabilities, so the whole is less available than any single part. When independent copies back each other up, the chance that all fail together is tiny, so availability rises sharply.
Allowed downtime:
| Availability | Per year | Per month, 30 days |
|---|---|---|
| 99% | 3.65 days | 7.2 hours |
| 99.9% | 8.76 hours | 43.2 minutes |
| 99.95% | 4.38 hours | 21.6 minutes |
| 99.99% | 52.6 minutes | 4.3 minutes |
| 99.999% | 5.26 minutes | 26 seconds |
The arithmetic, with the shop as an example:
In series, every part must work, so multiply:
text
0.9999 × 0.999 × 0.9995 ≈ 0.9984 → about 99.84%That is about 70 minutes of possible downtime a month, worse than any single part.
In parallel, with independent copies, the system fails only if every copy fails at the same moment. The chance one copy is down is 1%, or 0.01. The chance both are down is 0.01 × 0.01 = 0.0001:
text
1 − (0.01 × 0.01) = 0.9999 → 99.99%The catch: "independent". Two copies in the same zone, or running the same bad release, fail together. Real independence means different zones, and rolling out changes to one copy at a time.
SLAs are not the same as real availability. A provider's SLA, service level agreement, is a promise with a refund, usually service credits, if it is broken. It is not a guarantee that it never happens. Your own targets, SLOs, should be based on what you measure and what users need (question 630).
Each extra nine is expensive. Going from 99.9% to 99.99% often means multi-zone everything, automatic failover, careful releases and on-call staff. Going to 99.999% often means multi-region active-active. Ask whether users need it.