Appearance
Compute, Containers and Serverless
559. How do you choose between virtual machines, a container service, Kubernetes, and serverless functions?
The short answer
Each step up the ladder hands more work to the cloud and gives you less control. Virtual machines give full control of the operating system, and you patch and scale everything. Serverless container services run your image with almost no platform work. Kubernetes gives the most flexible platform for many services, but someone must run it. Functions run small pieces of code on events and scale to zero. Pick the most managed option that still meets your needs, and move down only for a concrete reason.
A quick guide:
| Option | Choose it when | Watch out for |
|---|---|---|
| Virtual machines | Special OS needs, licensed software, legacy apps | Patching, scaling and recovery are yours |
| Kubernetes | Many services, a platform team, need for its extensions | Running and upgrading the cluster (question 309) |
| Serverless containers | Web apps and APIs, small teams, spiky load | Fewer controls, platform limits |
| Functions | Short event-driven jobs, glue between services | Time limits, cold starts, cost at steady high load |
| Platform services, PaaS | A standard web app with no container need | Less control of the runtime (question 566) |
Questions that decide it:
- How many services, and who runs the platform? Two services and three developers rarely justify a Kubernetes cluster.
- How steady is the load? Spiky and often idle suits scale-to-zero options. Steady high load is cheaper on reserved machines or nodes.
- How long does work run? Functions have time limits, such as 15 minutes on AWS Lambda.
- What does it need from the OS? Special kernels, drivers, GPUs or licensed software point to virtual machines or Kubernetes.
- Portability. A container image runs on any of the container options, and can move between them later.
A sensible default for a new web service in a small team: a serverless container service. Move to Kubernetes when the number of services, or a specific need, makes it worth running.
560. What are autoscaling groups and virtual machine scale sets?
The short answer
They are managed groups of identical virtual machines built from one template. You set the minimum, maximum and desired number of machines, and the group keeps that many healthy ones running, replacing any that fail. Scaling rules add machines when load rises, for example when average CPU passes 60%, and remove them when it falls. A load balancer in front sends traffic to whichever machines are currently in the group. AWS calls them Auto Scaling groups, Azure calls them virtual machine scale sets.
The parts:
- A template, called a launch template in AWS and a model in Azure: the machine image, size, network, identity and a start-up script.
- Size limits: minimum, maximum and desired count.
- Health checks: from the load balancer or the machine itself. An unhealthy machine is replaced.
- Scaling rules: when to add or remove machines.
- Zones: spread across availability zones, so losing a zone leaves machines in the others.
Kinds of scaling rule:
| Kind | How it works | Example |
|---|---|---|
| Target tracking | Keep a metric near a value | Keep average CPU around 50% |
| Step or threshold | Add or remove when a metric crosses a line | Above 70% CPU for 5 minutes: add 2 |
| Scheduled | Change counts at set times | 10 machines from 8 a.m. on weekdays |
| Predictive | Learn the daily pattern and scale ahead | Add machines before the morning rush |
What makes this work: stateless machines. Any machine may be removed at any time, so nothing important may live only on it. Sessions go to a shared store, files go to object storage, and logs are shipped out (question 571).
Updating the template does not change running machines on its own. A rolling update, called an instance refresh in AWS, replaces machines in batches, much like a Kubernetes rolling update (question 350).
561. What are spot or preemptible instances, and how do you use them safely?
The short answer
Spot instances are the cloud's spare capacity sold at a deep discount, often 60 to 90% off, with one condition: the provider can take them back when it needs the capacity, with only a short warning. AWS gives two minutes, while Azure and Google Cloud give about 30 seconds. Use them for work that can be interrupted and restarted, such as batch jobs, CI runners and stateless services spread over many machines, and never as the only home of something that must stay up.
| Provider | Name | Warning before removal |
|---|---|---|
| AWS | Spot Instances | About 2 minutes |
| Azure | Spot Virtual Machines | As little as 30 seconds |
| Google Cloud | Spot VMs, formerly preemptible | About 30 seconds |
Good uses:
- Batch and data processing, where a lost job is simply retried.
- CI runners: a cancelled build reruns elsewhere.
- Stateless web services, as part of a mix with normal machines.
- Kubernetes node pools for workloads that tolerate interruption (question 612).
Bad uses: a single database, a single server, long jobs with no checkpoints, anything holding state only in memory.
Using them safely:
- Mix them. Keep a base of normal on-demand or reserved machines, and add spot capacity on top.
- Diversify. Ask for several machine sizes and several zones, so one size running out does not take all your capacity.
- Watch for the warning. The metadata service announces the coming removal (question 550). Use the time to stop taking new work, finish or save current work, and leave the load balancer.
- Checkpoint long work, so a restarted job continues instead of starting over.
- In Kubernetes, drain the node on warning, and use PodDisruptionBudgets so not every copy of a service goes at once (question 400).
562. What are serverless container services such as Azure Container Apps, AWS Fargate, and Google Cloud Run?
The short answer
They run your container image without you running any servers or cluster. You give the image, the CPU and memory it needs, and scaling rules, and the service runs as many copies as needed, gives them an HTTPS address, and handles patching of everything underneath. Cloud Run and Container Apps can scale to zero when idle and scale on requests or queue length. Fargate is the serverless way to run containers for AWS ECS, and EKS on AWS, where you still define tasks and services but no machines.
What you provide, and what the service does:
| You provide | The service does |
|---|---|
| A container image | Pulls and runs it |
| CPU and memory per copy | Finds capacity for it |
| Scaling rules | Adds and removes copies |
| Environment, secrets, identity | Injects them at start-up |
| Port and health checks | Gives an HTTPS address, routes traffic, restarts failures |
How the three differ:
| Azure Container Apps | Google Cloud Run | AWS Fargate with ECS | |
|---|---|---|---|
| Scale to zero | Yes | Yes | No; a service keeps its set count |
| Scales on | HTTP, queues, events, through KEDA | Requests, CPU | CPU, memory, custom metrics |
| Revisions and traffic split | Yes | Yes | Through deployments and load balancer rules |
| Built on | Kubernetes, hidden from you | Google's own platform | AWS's own platform |
KEDA, Kubernetes Event-driven Autoscaling, scales on outside events such as queue length (question 496). Container Apps uses it underneath.
Why it fits many teams: it gives most of what people use Kubernetes for, which is rolling out images, scaling, HTTPS and health checks, without a cluster to upgrade (question 621).
Where it falls short: very specific networking, custom operators, privileged or GPU-heavy workloads in some services, and dozens of services that need a shared platform with its own policies.
563. What is serverless computing, and how do functions like AWS Lambda and Azure Functions work?
The short answer
Serverless means you run code without managing any servers, and you pay only while the code runs. With functions, you upload a small piece of code and connect it to a trigger, such as an HTTP request, a new file in storage, a message on a queue, or a timer. When the event happens, the platform starts an instance of your function, passes it the event, and runs it. Many events at once mean many instances in parallel, and no events means no instances and no cost.
An example: an order pipeline. When a customer pays, the payment service puts a message on a queue. A function picks it up and sends the confirmation email. Nobody runs a server for this, and at 3 a.m., when nobody orders, it costs nothing.
How billing works. You pay per request and per unit of compute time, measured as memory size multiplied by run time, rounded to the millisecond on Lambda. A function with 512 MB that runs for 200 ms is billed for 0.1 gigabyte-seconds.
Key limits, worth knowing by heart:
| AWS Lambda | Azure Functions | |
|---|---|---|
| Longest single run | 15 minutes | Depends on the plan: minutes on consumption plans, longer on premium or dedicated plans |
| Memory | 128 MB to 10 GB; CPU grows with memory | Depends on the plan |
| Scale to zero | Yes | Yes, on consumption-style plans |
Functions should be stateless. An instance may be reused for the next event, or thrown away at any moment. Anything that must last goes to a database, storage or a queue.
Instances are reused. After a run, the platform keeps the instance warm for a while. Code outside the handler, such as creating a database client, runs once per instance and is then reused, which is why you put set-up there (question 564).
564. What is a cold start, and how do you reduce it?
The short answer
A cold start is the extra delay when a function, or a scaled-to-zero container, has no warm instance ready. Before it can handle the event, the platform must find capacity, load your code or image, start the runtime, and run your set-up code. That can add from about a hundred milliseconds to several seconds. Reduce it by keeping packages small, doing less at start-up, choosing faster-starting runtimes, and paying to keep some instances always warm, with provisioned concurrency, minimum instances or premium plans.
When it happens:
- The first request after a quiet period.
- A sudden burst, when many new instances start at once.
- After a new deployment, since old warm instances are replaced.
Ways to reduce it, cheapest first:
- Smaller packages and images. Less to download and load (question 72).
- Less work at start-up. Create clients lazily, load only what is needed, avoid big frameworks for tiny functions.
- Faster runtimes. Node.js, Python and Go usually start faster than the JVM or .NET without tuning. Lambda SnapStart saves a snapshot of an initialised instance for Java, Python and .NET, and resumes from it.
- More memory on Lambda, because CPU grows with memory, so start-up code runs faster.
- Keep instances warm, for a fee: Lambda provisioned concurrency, Azure Functions always-ready or pre-warmed instances, Cloud Run and Container Apps minimum instances.
Is it worth fixing? For a nightly batch job, a two-second cold start does not matter. For an API that users wait on, it does. Measure the slowest requests, the 99th percentile, not the average, because cold starts hide in the tail (question 629).
565. When should you not use serverless functions?
The short answer
Avoid functions for work that runs longer than their time limits. Avoid them for steady high load, where always-on compute is cheaper, and for workloads that need very low latency on every single request. They also fit badly with long connections and in-memory state. And hundreds of tiny functions can be harder to understand than one service. Also be careful that unlimited scaling does not overwhelm a database that cannot scale with it.
| Situation | Why functions struggle | Better choice |
|---|---|---|
| Jobs longer than 15 minutes | Hard time limits | Containers, batch services |
| Steady high traffic all day | Per-run pricing costs more than always-on | Containers, reserved machines |
| Strict latency on every request | Cold starts in the tail | Always-warm containers |
| WebSockets, long connections | Short-lived instances | Containers, managed WebSocket services |
| In-memory caches or state | Instances come and go | A service with a shared cache |
| Very many tiny functions | Hard to trace, deploy and reason about | A few well-shaped services |
The database trap. A traffic spike makes the platform start 1,000 instances. Each opens a database connection. The database, sized for 200 connections, falls over. Fix it with a connection pooler or proxy, such as RDS Proxy, by limiting the function's concurrency, or by putting a queue in between so work is processed at a steady rate.
The cost crossover. Functions are very cheap for spiky or low traffic. As traffic becomes steady and high, a few always-on containers become cheaper. Do the arithmetic with real numbers before deciding, and revisit it as traffic grows.
Where functions shine: glue between services, file processing on upload, scheduled jobs, webhooks, and spiky low-volume APIs.
566. What are platform services such as Azure App Service and AWS Elastic Beanstalk, and when do they fit?
The short answer
They are platform as a service offerings for web apps: you hand over your code, or a container, and the platform provides the servers, runtime, load balancing, scaling, TLS certificates, deployment slots and logs. They fit standard web apps and APIs where the team wants to focus on code, not infrastructure. You give up some control over the operating system and networking, and very custom or multi-service systems may fit containers or Kubernetes better.
What you get, for example with Azure App Service:
- Runtimes ready to use: .NET, Node.js, Python, Java, PHP, or your own container.
- Scaling: more instances by rule or schedule.
- Custom domains and managed TLS certificates.
- Deployment slots: deploy to a "staging" slot, warm it up, then swap it with production instantly. Swapping back is the rollback, a form of blue-green deployment (question 281).
- Built-in authentication, logs and health checks.
AWS Elastic Beanstalk gives a similar experience on top of normal AWS resources: it creates the machines, load balancer and scaling group for you, from your application bundle.
| Platform service | Serverless containers | Kubernetes | |
|---|---|---|---|
| You provide | Code or a container | A container | Containers and manifests |
| Platform work | Very little | Very little | A lot |
| Flexibility | Lowest | Medium | Highest |
| Good for | Classic web apps | Modern web apps and APIs | Many services, special needs |
A practical note. The lines are blurring. App Service runs containers, and serverless container services run web apps. Choose by how much control you need and what your team already knows.
567. What is immutable infrastructure, and what are golden images?
The short answer
Immutable infrastructure means you never change a running server. To update anything, you build a new image, start new servers from it, move traffic to them, and remove the old ones. A golden image is a pre-built machine image with the operating system, patches, agents and settings already in place, built automatically from code with tools such as Packer, and used as the starting point for every server. Every server is then identical and reproducible, and a rollback is simply returning to the previous image.
The old way, and its problem. Servers were updated in place: log in, install a patch, change a setting. Over months, each server became slightly different. This is called configuration drift. Nobody could rebuild a server exactly, and "it works on server 3 but not on server 4" became normal.
The immutable way:
- Build an image from code: a base OS, plus patches, security agents and your settings.
- Scan and test the image, and give it a version.
- Roll out new machines from it, through the scale set's rolling update (question 560).
- Remove old machines.
- To roll back, deploy the previous image version.
This is exactly how containers work. An image is built once and never changed while running. Golden images apply the same idea to whole virtual machines (question 28).
Tools:
| Tool | Does |
|---|---|
| HashiCorp Packer | Builds images for any cloud from a template |
| EC2 Image Builder | AWS's managed image pipeline |
| Azure VM Image Builder, Compute Gallery | Azure's image building and sharing |
Baking versus frying. Baking puts everything into the image in advance, so machines start fast and identical. Frying installs things at start-up with a script, which is more flexible but slower and can fail on start-up. Most teams bake the slow, stable parts, and fry only the small environment-specific parts (question 569).
568. How do you choose a machine size or instance type?
The short answer
Machine types are grouped into families by their balance of CPU and memory: general purpose, compute-optimised, memory-optimised, storage-optimised, and accelerated ones with GPUs. Start from what the app actually uses, measured under realistic load, pick the family that matches its shape, and pick the smallest size that leaves sensible headroom. Then keep resizing as real usage data comes in, because most cloud machines are bigger than they need to be.
| Family | Balance | Good for | AWS example | Azure example |
|---|---|---|---|---|
| General purpose | Even CPU and memory | Web servers, most apps | M | D |
| Compute optimised | More CPU per GB | Batch, encoding, busy APIs | C | F |
| Memory optimised | More memory per CPU | Databases, caches, analytics | R | E |
| Storage optimised | Fast local disks | Search, some databases | I | L |
| Accelerated | GPUs or other chips | Machine learning, graphics | P, G | N |
| Burstable | Low baseline, short bursts | Small, mostly idle machines | T | B |
Burstable machines run at a low baseline and save up "CPU credits" while idle, spending them on short bursts. They are cheap for small, quiet workloads, and a nasty surprise if the load is steady, because the credits run out and the machine slows down.
How to choose, step by step:
- Measure CPU, memory, disk and network under realistic load.
- Match the shape. An app using 2 CPUs and 14 GB of memory is memory-heavy: a memory-optimised type fits better than a bigger general one.
- Pick the current generation. Newer generations usually give more performance for the same price.
- Consider ARM processors, such as AWS Graviton or Azure Cobalt. They are often cheaper for the same work, if your software and images support ARM (question 82).
- Leave headroom, then resize with data. Cloud tools such as AWS Compute Optimizer and Azure Advisor suggest right-sizing.
For Kubernetes nodes, the same thinking applies to node pools, plus one more factor: larger nodes waste less on system overhead per pod, but losing one hurts more (question 612).
569. How do start-up scripts like cloud-init and user data configure a new machine?
The short answer
When a cloud machine boots for the first time, it can run a script or configuration you provide, called user data in AWS and custom data in Azure. On Linux, a tool called cloud-init reads it and does the work: installing packages, writing files, creating users, and starting services. It is how a generic image becomes a specific server, for example joining a cluster or fetching configuration. Keep it small and quick, and put the slow, stable parts in the image instead.
An example, cloud-init configuration for a worker that processes orders:
yaml
#cloud-config
package_update: true
packages:
- docker.io
write_files:
- path: /etc/order-worker/env
content: |
QUEUE_NAME=orders
REGION=westeurope
runcmd:
- systemctl enable --now docker
- docker run -d --restart=always --env-file /etc/order-worker/env registry.shop.example/order-worker:1.8.2The first line, #cloud-config, tells cloud-init what kind of content this is. Then, in order:
package_updateandpackagesrefresh the package list and install Docker.write_fileswrites the worker's settings to a file.runcmdruns commands at the end: start Docker, then start the worker container, restarted automatically if it stops.
Good habits:
- No secrets in user data. It can be read from the machine's metadata by anything on the machine, and often by anyone who can view the machine's settings in the cloud console. Fetch secrets at runtime with the machine's identity (question 553).
- Keep it short. Every minute here is a minute before the new machine can take traffic, which matters a lot when scaling up during a spike.
- Bake the stable parts into a golden image, such as Docker and agents (question 567).
- Logs are in
/var/log/cloud-init-output.logwhen something goes wrong.
570. How do you handle retries and failures in event-driven serverless code?
The short answer
Event platforms deliver each event at least once, so your function may see the same event twice, and they retry automatically when it fails. So make every function idempotent: processing the same event twice must have the same effect as processing it once. Report failures per message rather than failing a whole batch, send events that keep failing to a dead-letter queue after a few attempts, and alert on that queue. Never let a broken message block the ones behind it for ever.
Idempotent means doing it twice gives the same result as doing it once. Setting an order's status to "paid" is idempotent. Adding £10 to a balance is not.
An example: an AWS Lambda function processing order messages from an SQS queue, in TypeScript:
ts
import type { SQSEvent, SQSBatchResponse } from 'aws-lambda'
import { markOrderPaid, alreadyProcessed, rememberProcessed } from './orders'
export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
const failures: { itemIdentifier: string }[] = []
for (const record of event.Records) { // (1)
const { orderId, paymentId } = JSON.parse(record.body)
try {
if (await alreadyProcessed(paymentId)) continue // (2)
await markOrderPaid(orderId, paymentId)
await rememberProcessed(paymentId)
} catch (err) {
console.error('failed', { orderId, paymentId, err })
failures.push({ itemIdentifier: record.messageId }) // (3)
}
}
return { batchItemFailures: failures } // (4)
}- A batch. The queue delivers several messages in one call, here up to ten.
- Skip duplicates. If this payment was already handled, do nothing. That makes a repeated delivery harmless. In real code, the check and the update should happen in one database transaction, or use a unique constraint on
paymentId, so two copies running at once cannot both pass the check. - Record only the failed message. One bad message should not force the good ones to be retried.
- Report partial failure. With "report batch item failures" turned on, only the listed messages go back to the queue to be retried. It prints nothing for the successful ones.
Why "at least once"? Guaranteeing exactly one delivery across machines that can fail is very hard. Platforms choose to deliver again when unsure, and leave duplicate handling to you. That is why idempotency matters.
Good habits:
- A dead-letter queue on every queue and asynchronous trigger, with an alarm when it is not empty.
- Retries with backoff: wait longer between each attempt, so a struggling dependency can recover.
- A visibility timeout longer than the function's run time. Otherwise the queue thinks the message was lost and hands it to another instance while the first is still working.
- Replay tooling: once the bug is fixed, move messages from the dead-letter queue back to the main queue.
- Logs with IDs, such as
orderIdandpaymentId, so one message can be followed through every step (question 634).