Appearance
Storage, Databases and Messaging
571. What is the difference between object, block, and file storage?
The short answer
Block storage is a raw virtual disk attached to one machine, which formats it with a file system: fast, low-latency, and used for operating systems and databases. File storage is a shared network folder that many machines mount at once, with normal folders and file locking. Object storage keeps whole files, called objects, with metadata, in a flat bucket reached over HTTP: it is cheap, almost unlimited and extremely durable, but you replace whole objects rather than editing parts of them in place.
| Block | File | Object | |
|---|---|---|---|
| Looks like | A disk | A network folder | A bucket of files with keys |
| Accessed by | One machine, mostly | Many machines | Anything, over HTTPS |
| Edit part of a file | Yes | Yes | No: replace the whole object |
| Speed | Fastest, lowest latency | Medium | Higher latency, huge throughput |
| Scale | Fixed size you choose | Grows, with limits | Practically unlimited |
| Cost per GB | Medium to high | Highest | Lowest |
| AWS | EBS | EFS, FSx | S3 |
| Azure | Managed Disks | Azure Files, NetApp Files | Blob Storage |
| Google Cloud | Persistent Disk | Filestore | Cloud Storage |
When to use which, in the shop example:
- Block: the database server's disk, and every virtual machine's system disk.
- File: an old app that expects a shared folder, or a few pods that really must share files (question 385).
- Object: product images, receipts as PDFs, uploads, backups, logs, static website files and data for analytics.
Object storage is the default for anything that is "a file". It is cheap, it cannot fill up, it is stored in several copies automatically, and any service can read it with the right identity. A key in object storage looks like a path, such as receipts/2026/10/8812.pdf, but there are no real folders: the name is just a string.
572. How do storage tiers and lifecycle policies reduce cost?
The short answer
Object storage offers tiers that trade storage price against access speed and cost. A hot tier is for data used often. Cooler tiers store data more cheaply, but charge more to read it and require a minimum stay. An archive tier is very cheap, but data can take hours to bring back. Lifecycle policies move objects between tiers, or delete them, automatically by age, so old logs and backups end up somewhere cheap without anyone remembering to do it.
| Use pattern | AWS S3 class | Azure Blob tier | Minimum stay | Read back |
|---|---|---|---|---|
| Used often | Standard | Hot | None | Immediately |
| Monthly or less | Standard-IA | Cool | 30 days | Immediately, read fee |
| Rarely, but fast when needed | Glacier Instant Retrieval | Cold | 90 days | Immediately, higher read fee |
| Archive | Glacier Flexible, Deep Archive | Archive | 90 to 180 days | Minutes to hours |
| Unknown pattern | Intelligent-Tiering | Not the same; lifecycle rules by last access | None | Immediately |
A minimum stay means you pay for that many days even if you delete the object sooner. Putting short-lived data in a cool tier can cost more than keeping it hot.
A lifecycle policy for the shop's logs:
json
{
"Rules": [{
"ID": "app-logs",
"Filter": { "Prefix": "logs/" },
"Status": "Enabled",
"Transitions": [
{ "Days": 30, "StorageClass": "STANDARD_IA" },
{ "Days": 90, "StorageClass": "GLACIER" }
],
"Expiration": { "Days": 400 }
}]
}It reads, in plain words:
- Applies to every object whose key starts with
logs/. - After 30 days, move to the cheaper infrequent-access class.
- After 90 days, move to the archive class.
- After 400 days, delete, which is just over a year, a common retention rule.
Do not forget old versions. With versioning on, overwritten and deleted objects keep their old versions, and they cost money too. Add a rule to expire non-current versions after some days.
Archive is not a backup you can use in an emergency quickly. Bringing data back from the deepest archive tiers can take many hours. So choose a tier by asking how fast you might need the data back (question 584).
573. What is the difference between durability and availability, and what redundancy options do clouds offer?
The short answer
Durability is the chance that your data is not lost. Availability is the chance that you can reach it right now. Object storage is designed for extreme durability, such as eleven nines, 99.999999999% of objects kept per year, by storing several copies. Availability is lower, such as 99.9 to 99.99%, because brief outages happen even when nothing is lost. You choose how far apart the copies live: in one data centre, across zones, or in a second region.
An example of the difference. During a zone power cut, your files may be unreachable for an hour. That is a loss of availability. When the power returns, every file is still there. Durability was never at risk. Durability fails only if every copy is destroyed.
| Option, Azure name | Copies | Protects against | Typical use |
|---|---|---|---|
| LRS, locally redundant | 3 in one data centre | Disk and rack failures | Data you can recreate |
| ZRS, zone redundant | 3 across zones | A zone outage | Most production data |
| GRS, geo redundant | 3 local + 3 in a paired region | A region disaster | Important data, compliance |
| GZRS | 3 across zones + 3 in another region | Zone and region failures | The most critical data |
AWS S3 Standard always stores data across at least three zones. Copying to another region is a separate feature, replication, which you turn on per bucket.
Two things redundancy does not protect against:
- Deleting or overwriting by mistake. The deletion is copied to every replica faithfully. Versioning, soft delete and backups protect against that.
- Ransomware, malicious software that encrypts or deletes your data. Use immutable, write-once backups that even an admin cannot delete for a set time (question 577).
574. How do disk snapshots work, and are they a backup?
The short answer
A snapshot is a point-in-time copy of a disk, stored in object storage. The first one copies all used blocks, and later ones store only the blocks that changed, so they are fast and cheap. A snapshot becomes a real backup only when it is kept independent of the original, ideally in another account or region, protected from deletion, and you know it restores. Snapshots of a running database also need care, because a disk captured mid-write may not restore cleanly.
How incremental snapshots work:
- Monday's snapshot copies every used block of a 100 GB disk.
- On Tuesday, 2 GB changed. Tuesday's snapshot stores only those 2 GB, and points to Monday's copy for the rest.
- Deleting Monday's snapshot does not break Tuesday's: the cloud keeps any blocks that a remaining snapshot still needs.
Consistency, the hidden trap:
| Kind | What it captures | Safe for databases? |
|---|---|---|
| Crash-consistent | The disk as if the power was pulled | Usually recoverable, the database replays its log, but not guaranteed |
| Application-consistent | The disk after the app flushed and paused writes | Yes |
For databases, prefer the database's own backups, or a managed database's built-in backups (question 577). For virtual machines with several disks, take snapshots of all disks together, as one group, so they match each other.
When a snapshot is not a backup:
- It sits in the same account as the disk, so an attacker or a mistake that deletes the disk can delete its snapshots too.
- It has never been restored, so nobody knows it works.
- It is only in one region.
Make it a backup: copy snapshots to another region and a separate backup account, lock them against deletion for a period, and restore one regularly as a drill. AWS Backup and Azure Backup automate the schedule, the copying and the locks.
575. What does a managed database service do for you, and what is still your job?
The short answer
A managed database, such as Amazon RDS, Azure SQL Database or Cloud SQL, gives you a running database without managing its servers. The provider installs it, patches it, takes automatic backups, offers point-in-time restore, handles failover to a standby, and monitors the hardware. You still own the schema, the queries and indexes, choosing the size, network access, users and permissions, the backup retention, testing restores, and planning major version upgrades.
| Task | Managed service | You |
|---|---|---|
| Hardware, OS, database install | Yes | |
| Minor version patches | Yes, in a maintenance window you choose | Pick the window |
| Automatic backups and point-in-time restore | Yes | Set retention, practise restores |
| Standby and failover | Yes, if you turn it on | Pay for it, and make apps reconnect |
| Monitoring of the hardware | Yes | Watch your own queries and slow logs |
| Schema, indexes, query performance | Yes | |
| Size and storage growth | Offers autoscaling of storage | Choose size, watch limits |
| Network access and encryption | Provides the options | Configure private access and TLS |
| Users and permissions | Yes | |
| Major version upgrades | Provides the tools | Plan, rehearse and run |
The types of managed databases:
- Relational, such as PostgreSQL, MySQL and SQL Server: RDS, Aurora, Azure SQL, Azure Database for PostgreSQL, Cloud SQL.
- Key-value and document: DynamoDB, Cosmos DB, Firestore (question 578).
- Caches: ElastiCache, Azure Cache for Redis, Memorystore (question 579).
The honest trade-off. You lose some control: you cannot log in to the server, some extensions and settings are unavailable, and you depend on the provider's maintenance. For almost every team, that is a very good trade, because running a highly available database well is a full-time job.
Serverless database options go one step further: capacity scales up and down automatically, sometimes to zero, as with Aurora Serverless or Azure SQL serverless. They suit spiky or idle workloads. A paused database takes a moment to wake on its first request.
576. What is the difference between a standby for high availability and a read replica, and how do databases scale?
The short answer
A high-availability standby is a copy in another zone kept exactly in step, synchronously, which takes over automatically if the primary fails, usually within a minute or two. It is about staying up, and often does not serve traffic. A read replica is a copy updated asynchronously, slightly behind, that serves read queries to take load off the primary. Databases scale first vertically, with a bigger machine, then with read replicas for reads, caching, and finally sharding, splitting data across several databases, for writes.
| High-availability standby | Read replica | |
|---|---|---|
| Purpose | Survive a failure | Spread read load |
| Updated | Synchronously, before the write is confirmed | Asynchronously, slightly behind |
| Serves traffic | Usually not | Yes, reads only |
| Data lost on failover | None | Recent writes, if promoted |
| Location | Another zone | Same or another region |
Synchronous means the primary waits until the standby has the change before saying "done". That costs a little write latency, but no confirmed write can be lost. Asynchronous means the primary does not wait, so the replica may be some milliseconds or seconds behind. That delay is called replication lag.
What failover means for the app. The database address, a DNS name, moves to the standby. Existing connections break. Apps must reconnect and retry, which good database drivers and connection pools do. Never connect to a database by its IP address.
Read-your-writes trap. A user updates their address, then the next page reads from a replica that is two seconds behind and shows the old address. Read from the primary right after a user's own write, or for anything where staleness matters.
Scaling, in order:
- Fix queries and indexes. Often the cheapest, biggest win.
- Vertical scaling: a bigger machine.
- Caching frequent reads (question 579).
- Read replicas for read-heavy traffic.
- Sharding: split data by a key, such as customer ID, across several databases. Each holds part of the data and takes part of the writes. It is powerful and complicated: queries across shards and moving data between shards are hard. Some databases, such as Cosmos DB, DynamoDB, or distributed SQL databases, do it for you.
577. How do backups and point-in-time restore work for cloud databases?
The short answer
Managed databases take a full backup daily and keep every change log between backups. Together, these let you restore the database to any second within the retention period, usually up to 35 days, which is point-in-time restore. A restore always creates a new database, which you then check and switch to. Complete the picture with long-term backups for compliance, copies in another region and account, deletion locks against ransomware, and regular restore drills so you know the time it takes.
How point-in-time restore works:
- A full backup is taken every night.
- Every change is written to the database's transaction log, and the log is saved to storage every few minutes.
- To restore to 14:31:05 on Tuesday, the service takes Tuesday's full backup and replays the log up to exactly that second.
A real use. At 14:32 a bad migration deletes the discount column from every product. You restore a copy of the database to 14:31, then either switch the app to it, or copy the lost column back into the live database.
What to decide:
| Question | Example answer |
|---|---|
| How far back can we restore? | 14 days of point-in-time restore |
| How long do we keep monthly copies? | 7 years, for financial records |
| Where are copies kept? | Another region and a separate backup account |
| Who can delete backups? | Nobody, during a locked retention period |
| How long does a restore take? | Measured in a drill: 40 minutes for 200 GB |
A restore is a new database. It gets a new name and address. Your runbook must include pointing the app at it, or copying data from it, and that step takes time too.
Deleting the database often deletes its automatic backups. Many services remove automatic backups when the database itself is deleted, unless you keep a final snapshot or use a separate backup service. Protect production databases with deletion protection.
Practise. A backup that has never been restored is a hope, not a plan. Restoring regularly also tells you your real recovery time, which you need for your recovery time objective (question 584).
578. When would you choose a NoSQL database such as DynamoDB or Cosmos DB over a relational database?
The short answer
Choose a relational database by default: it handles flexible queries, joins and transactions across tables, and most apps fit it well. Choose a NoSQL key-value or document database when you know your access patterns in advance and need huge scale with steady low latency, a flexible document shape, or easy multi-region writes. The price is that you design data around your queries: queries you did not plan for are slow or impossible, and joins mostly happen in your code.
| Relational, such as PostgreSQL | NoSQL, such as DynamoDB, Cosmos DB | |
|---|---|---|
| Data shape | Tables with a fixed schema | Items or documents, flexible |
| Queries | Any query, with joins | Fast by key; others must be planned |
| Transactions | Across any tables | Limited, often within one partition |
| Scaling | Vertical, then replicas, then sharding | Built-in horizontal partitioning |
| Latency at huge scale | Needs work | Steady single-digit milliseconds by key |
| Multi-region writes | Hard | Built into some, such as Cosmos DB |
Partition keys are everything. NoSQL databases spread data across many machines by a partition key, such as customerId. Reading by that key is fast at any scale. A poor key, such as a status field with only three values, puts most data in a few partitions, called hot partitions, which slow down and throttle.
Example fit: a shopping cart. It is always read and written by cartId, needs low latency at sale-day peaks, and has a shape that changes with new features. That is a good NoSQL fit. The order and payment history, which finance queries in many ways, fits a relational database better.
Consistency choices. Some NoSQL databases let you pick consistency per request. DynamoDB offers eventually consistent reads, cheaper and possibly slightly stale, or strongly consistent ones. Cosmos DB offers five levels, from strong to eventual.
579. How does caching work with a managed cache such as Redis, and what are its pitfalls?
The short answer
A cache keeps copies of frequently read data in memory, so most reads skip the slower database. The most common pattern is cache-aside: the app checks the cache first, and on a miss reads the database and stores the result with an expiry time. The hard parts are keeping cached data fresh after changes, protecting the database when many entries expire at once or a hot key expires, and remembering the cache can be empty at any time.
In code, for the shop's product page, in TypeScript with the redis client:
ts
import { createClient } from 'redis'
import { db, type Product } from './db'
const cache = createClient({ url: process.env.REDIS_URL })
await cache.connect()
export async function getProduct(id: string): Promise<Product> {
const key = `product:${id}`
const hit = await cache.get(key) // (1)
if (hit) return JSON.parse(hit)
const product = await db.products.findById(id) // (2)
await cache.set(key, JSON.stringify(product), { EX: 300 }) // (3)
return product
}
export async function updatePrice(id: string, price: number) {
await db.products.update(id, { price })
await cache.del(`product:${id}`) // (4)
}- Look in the cache. A hit returns in well under a millisecond inside the same region.
- On a miss, read the database, which remains the source of truth.
- Store with an expiry, here 300 seconds. Even if something goes wrong, stale data disappears within five minutes.
- On a change, delete the cached copy. The next read fetches the new price. Deleting is safer than updating the cache, because two updates racing each other could leave the older value in the cache.
The pitfalls:
| Pitfall | What happens | Fix |
|---|---|---|
| Stale data | Users see old prices | Delete on write, short expiry |
| Cache stampede | A popular key expires; 1,000 requests hit the database at once | Let one request refresh it, add random jitter to expiry times |
| Cold cache | After a restart, everything misses at once | Warm up popular keys, scale gradually |
| Treating the cache as storage | Data lost when the cache restarts | Keep the database as the truth |
| Huge values or keys without expiry | Memory fills; evictions | Set expiries, keep values small |
Managed Redis services are Amazon ElastiCache, Azure Managed Redis or Azure Cache for Redis, and Google Memorystore. Treat them like a database for security: private network access, TLS, and authentication.
580. What is the difference between queues, publish-subscribe topics, and event streams?
The short answer
A queue holds messages until one consumer takes and processes each one, which spreads work across workers and absorbs bursts. A publish-subscribe topic delivers a copy of each message to every subscriber, so several services can react to the same event. An event stream is an ordered log that keeps events for a period, and each consumer reads at its own position and can replay from the past. Use queues for work, topics for notifications, and streams for high-volume ordered data and replay.
| Queue | Topic, publish-subscribe | Stream | |
|---|---|---|---|
| Each message goes to | One consumer | Every subscriber | Every reader, at its own pace |
| After reading | Deleted | Gone, once delivered | Kept until retention ends |
| Replay | No | No | Yes |
| Ordering | Best effort, or FIFO options | Usually none | Within a partition |
| Use for | Background jobs, load levelling | "Something happened" fan-out | Activity logs, analytics, high volume |
| AWS | SQS | SNS, EventBridge | Kinesis, MSK |
| Azure | Service Bus queues, Storage queues | Service Bus topics, Event Grid | Event Hubs |
| Google Cloud | Cloud Tasks | Pub/Sub | Pub/Sub, Managed Kafka |
Load levelling is what a queue does during a burst. On sale day, 50,000 orders arrive in ten minutes. The queue holds them, and workers process them at the rate the database can handle, instead of the database falling over (question 565).
A common combination, called fan-out: the order service publishes order-placed to a topic, and each subscriber, such as email, warehouse and analytics, gets its own queue fed from the topic. Each can process at its own pace and retry on its own, without affecting the others.
Delivery and ordering, in brief:
- Most services deliver at least once, so consumers must be idempotent (question 570).
- Ordering costs throughput. SQS FIFO queues and Service Bus sessions keep order per group, such as per customer, rather than across everything.
- Dead-letter queues collect messages that keep failing.
581. How would you migrate a production database to the cloud with minimal downtime?
The short answer
Copy the data while the old database keeps running, then keep the copy in sync by streaming every new change, a technique called change data capture. Check that the two match. Then, in a short planned window, stop writes to the old database, let the last changes arrive, point the apps at the new database, and start writes again. Keep the way back open for a while, by syncing changes in reverse or keeping the old database ready, until you trust the new one.
Step by step:
- Prepare the target: the managed database, its network, users, and the same version or a compatible one.
- Full load: copy all existing data while production keeps running.
- Change data capture: read the old database's transaction log and apply every new insert, update and delete to the new one, continuously. Tools: AWS Database Migration Service, Azure Database Migration Service, Google Database Migration Service, or the database's own replication.
- Validate: compare row counts and checksums, and run the app's read traffic against the new database in a test environment.
- Rehearse the cut-over in a staging copy, and time it.
- Cut over, in a quiet hour:
- Put the app into read-only or maintenance mode.
- Wait until replication lag reaches zero.
- Switch the connection string, ideally a DNS name, to the new database.
- Turn writes back on, and watch errors and latency closely.
- Keep a way back: for days, keep the old database, and if possible stream changes back to it, so a rollback does not lose new orders.
What goes wrong:
| Problem | Prevention |
|---|---|
| Sequences, triggers or jobs not migrated | Check the full list of database objects, not just tables |
| Different versions behave differently | Rehearse with the app, not only the data |
| Network too slow for the full load | Measure first; use a dedicated connection or offline transfer (question 536) |
| Apps with the old address hard-coded | Find every connection string before the day |
| Time zones, character sets, collation | Compare settings explicitly |
Collation is the set of rules a database uses to sort and compare text, such as whether "a" equals "A".
582. How should applications connect to a managed database securely and reliably?
The short answer
Reach the database privately, through a private endpoint or private network address, with public access turned off. Encrypt the connection with TLS and verify the server's certificate. Use identity-based authentication where the database supports it, or a password from the secret store, with a separate least-privilege user per app. And use a connection pool sized for the database's limits, with retries and reconnection, so failovers and spikes do not turn into outages.
Securely:
- Private network only. Private endpoint, or VNet integration, and public access disabled (question 538).
- TLS, verified. Require TLS on the server, and make the client check the certificate, so nobody can pretend to be the database.
- Identity instead of passwords. Azure SQL and Azure Database for PostgreSQL accept Entra ID tokens, and RDS accepts IAM authentication tokens. The app's managed identity gets a short-lived token instead of storing a password (question 550).
- One user per app, with only the rights it needs: the reporting app gets read-only access, and no app connects as the admin user.
- Audit logins and, for sensitive data, queries.
Reliably:
| Concern | Practice |
|---|---|
| Connection limits | A pool per app instance; total connections across all instances within the database's limit |
| Many short-lived instances, such as functions | A connection proxy or pooler, such as RDS Proxy or PgBouncer |
| Failover | Connect by DNS name; drivers reconnect and retry |
| Slow queries | Timeouts on every query, so one slow query cannot hold every connection |
| Retrying | Retry only safe operations, with backoff |
The connection arithmetic. Twenty app pods, each with a pool of 20 connections, can open 400 connections. If the database allows 300, the twenty-first busy minute brings errors. Set pool sizes from the database's limit, divided by the maximum number of instances, and remember that autoscaling raises the instance count (question 560).
PgBouncer is a small, widely used connection pooler for PostgreSQL that lets many app connections share a few real database connections.