Appearance
13.3 — Compute: Machines, Containers and Functions
Compute is the cloud's word for "something that runs your code". A provider sells it in several forms. One of them is the function: you hand over a small piece of code, and the platform runs it each time a request arrives. Here is a function that builds a sales report and returns it to the caller. It works in development against test data. In production it fails:
Response payload size exceeded maximum allowed payload size (6291456 bytes)The number in the message is six megabytes. That is the largest reply this kind of function may return. The report is 9 MB, and there is no setting that raises the limit. The only fix is to change the design. The function writes the report into object storage, the file service from Chapter 13.1. It then returns a short link to the file. The link is a pre-signed URL, which works for a few minutes and then expires (Chapter 13.4). This is a better design anyway. But the team found it only by running into the limit.
The limits of a serverless platform are fixed. You cannot tune them, so you have to design around them. The same is true of the other forms of compute. Each one fixes some things about what your application can be. This page goes through the forms one at a time.
1. The ladder
| Level | You manage | Scales in | Pay for |
|---|---|---|---|
| Virtual machine | OS, runtime, app | Minutes | Uptime |
| Container on your cluster | Cluster, image, app | Seconds–minutes | Node uptime |
| Managed container (Cloud Run, Fargate, Container Apps) | Image, app | Seconds | Container time, can be zero |
| Function (Lambda, Functions) | Code | Milliseconds | Execution time |
Three words in the table need a line each. A virtual machine is a share of a real server that behaves like a whole computer of your own. A container is a package holding your application together with everything it needs to run, so it runs the same way on any machine. The package file is called an image, and Part 14 builds containers from the ground up. A cluster is a group of machines that run your containers. "Scales in" is how long it takes to add capacity when traffic grows.
Each step down the table takes work away from you and adds limits. Do not ask which row is the most modern. Ask two things. Which limits can your application live with? And is the work being taken away work that your team was doing badly anyway?
2. Virtual machines
A rented virtual machine is called an instance. Providers sell instances in families. Each family has a different balance of processor power (CPU), memory and disk. Choosing the family is your first decision, and choosing the wrong one is the most common way to waste money.
- General purpose has a balanced amount of CPU and memory. Pick it when you do not yet know what your application needs.
- Compute optimised has more CPU for each gigabyte of memory. It suits work that calculates a lot, such as converting video or serving web pages.
- Memory optimised has more memory for each CPU. It suits databases and caches, which keep a lot of data in memory.
- Storage optimised has very fast disks attached directly to the machine. It suits systems that store data themselves.
- Accelerated has graphics processors (GPUs), which AI workloads need (Chapter 12.12.3).
Choose the size from measurements. Do not copy the size of the machine you had before. Choosing a size that fits the real load is called right-sizing. Here is the usual mistake. A team moves an application from its own data centre to the cloud and orders a machine with the same specification. But the old machine was bought to cope with the busiest day of the next three years, with spare room on top. Most days it sits nearly idle. So measure the CPU and memory the application really uses over two weeks, and then choose.
Burstable instances need a warning of their own. They are the cheapest kind, and on AWS their names start with t. A burstable instance earns CPU credits while it is idle and spends them while it is busy. If it stays busy all the time, the credits run out, and the provider slows the instance to a fraction of one CPU core. The slowdown is sudden, it lasts until the load drops, and no code changed. It is the same thing that happened to the disk's burst credits in Chapter 13.1. Burstable instances are excellent for work that is quiet most of the time with short busy spells. They are wrong for anything with a steady load.
Next you decide how the software gets onto the machine. There are two approaches.
- A golden image. A machine image is a saved copy of a whole disk, with the operating system and everything installed on it. You install your application once, save the result as an image, and start every machine from that image. Machines start fast and are identical. To change something, you build a new image.
- Configuration at boot. You start from a plain image. A tool then installs and configures everything each time a machine starts. This is flexible, but machines take longer to start. There is a second problem. Machines set up on different days pick up different versions of things, so over time they stop being identical. This slow divergence is called drift.
Prefer golden images, together with a rule called immutable infrastructure. The rule is that you never change a running machine. When something must change, you build a new image and replace the machine. Then every machine is known to be the same, and any of them can be rebuilt exactly. Containers work on the same idea.
An auto-scaling group keeps the right number of machines running for you. You tell it how many healthy instances you want. It replaces any that fail, and it spreads them across zones. It also adds or removes instances as a measurement, such as average CPU use, goes up or down. People commonly get three things wrong.
- Add machines quickly and remove them slowly. If you remove machines the moment traffic dips, you will be short when it rises again a minute later. The waiting time before removing is called a cooldown.
- Make the health check test the application. A machine can be switched on and running while the application on it has crashed. The check should ask the application itself.
- Give machines time to finish before they are removed. A lifecycle hook makes the group wait while a machine completes the requests it has already accepted. Chapter 9.9.7 covers this graceful shutdown.
3. Serverless functions, precisely
A function runs like this. Something happens that the function should react to, such as a web request or a file being uploaded. That is called an event. The platform needs a place to run your code, called an execution environment. It is a small isolated box with your code loaded into it. If an environment is already loaded and waiting, it is called warm, and the platform uses it. If none is waiting, the platform creates one. It then calls your handler, which is the one function in your code that receives the event. Afterwards it keeps the environment warm for a while, in case another event arrives.
One environment handles one request at a time. This is the most important fact about functions, and people often get it wrong. The platform handles more requests at once by creating more environments, not by running threads inside one. So 100 simultaneous requests means 100 environments. It also means 100 database connections, which is section 5's problem.
There is now one exception to know about. Lambda is the name of AWS's function service, and EC2 is the name of its virtual machine service. In December 2025 AWS launched Lambda Managed Instances. Your functions run on EC2 machines in your own account, and one environment can serve many requests at the same time, like a normal server. That lowers the cost at steady volume. It also brings back the problems of shared state: two requests in the same environment can now race on the same global variable or the same file in /tmp. Standard Lambda, which is what most people mean by "Lambda", still serves one request per environment.
A cold start is the delay when no warm environment exists. The platform has to do three things before it can answer. It downloads your code package. It starts the runtime, which is the program that runs your language, such as Node.js for JavaScript or the Java virtual machine (JVM) for Java. Then it runs your start-up code. For a small package on a runtime that starts quickly, all of this takes tens of milliseconds. For a large package on the JVM or on .NET, it takes from hundreds of milliseconds to a few seconds.
Five things really do make cold starts shorter.
- Make the package smaller. Use fewer libraries, and leave out code and files the function never uses.
- Pick a runtime that starts fast wherever users are waiting for the reply. Node, Python, Go and Rust start much faster than the JVM.
- Do the expensive work once, outside the handler. Code written outside the handler runs only when the environment is created. Every later call on that environment reuses the result. Opening a database connection belongs there.
- Buy provisioned concurrency. You pay the platform to keep a chosen number of environments warm all the time. It works. But now you are paying for idle capacity, and not paying for idle capacity was the main reason to use functions. So use it only where users are waiting.
- Start from a snapshot. AWS calls this SnapStart. The platform runs your start-up code once and saves a copy of the environment's memory. Every new environment is then restored from that copy, which removes most of the JVM's delay. There is a catch. Anything that should be different in each environment is copied too. A random number seed, a saved password and an open network connection are examples. Your code must create those again after the restore.
"Keep-warm" pings do not help, so drop them. The idea is to call your own function every few minutes so that it never goes cold. That keeps one environment warm. But when eleven requests arrive at once, ten of them still need new environments, and those ten are the cold starts your users feel.
4. The limits that decide the design
Check these limits before you choose functions. Finding them afterwards means a redesign. The word payload in this list means the data sent to the function or returned by it.
These are AWS Lambda's numbers as of 2026. Azure Functions and Google Cloud Run functions have limits of the same kind.
- Execution time — 15 minutes at most. Longer work needs a container or a workflow engine (section 8).
- Payload for a normal call — 6 MB in each direction. Return a reference, not a blob, which is the opening story.
- Payload for a streamed response — up to 200 MB since July 2025, if the function streams its answer in pieces instead of returning it all at once. The first 6 MB arrive at full speed, and the rest is limited to about 2 MB per second.
- Payload for an asynchronous call — 1 MB since October 2025, up from 256 KB. Asynchronous means the caller hands over the event and does not wait for the answer.
- Package size — 50 MB zipped and 250 MB unzipped. A container image can be up to 10 GB, and a larger image starts more slowly.
- Temporary disk —
/tmpis 512 MB by default and can be raised to 10 GB. It may persist between calls on the same environment. So user data written there can be read by the next request that lands on that environment, unless you delete it. - Memory — from 128 MB to 10,240 MB. CPU is handed out in proportion to memory, and at 1,769 MB you get one full CPU core. So raising memory for a CPU-bound function can make it cheaper: it finishes in proportionally less time, and you pay for memory × time. This is genuinely counter-intuitive, so measure it rather than assume.
- Concurrency — by default 1,000 environments running at once per region, which you can ask to raise. Each function can add up to 1,000 new environments every 10 seconds. A spike faster than that is throttled, which is a real availability limit.
- No memory of earlier calls. Each run of the function is called an invocation. Nothing the function keeps in memory or on its disk is promised to be there on the next invocation. The next request may land on a different environment altogether.
5. Functions and databases
Functions and ordinary databases do not fit together well. Here is why. Each environment opens its own connection to the database. Suppose a burst of traffic causes 500 environments to run at the same time. That opens 500 connections. But a database handles only a small number of connections well. Chapter 7.2.4 worked through the arithmetic, and for a typical database the right number is closer to twenty. With 500, the database runs out of memory or starts refusing new connections.
There are three ways to solve this.
Put a connection proxy in front of the database. A proxy is a small service that sits between the functions and the database. Hundreds of functions connect to the proxy. The proxy passes their queries to the database over a few real connections, which it shares between them. AWS sells one called RDS Proxy, and PgBouncer is a free one for PostgreSQL (Chapter 7.2.4). This is the standard fix.
Use an HTTP data API. Some managed databases accept each query as a separate web request. No connection stays open, so there is nothing to run out of. Each query is a little slower.
Use a different kind of database. Some databases have no connections at all. Every read and write is its own separate request. AWS's DynamoDB is one. This is why teams that use functions tend to end up on databases of this kind. Choosing functions pushes you towards a certain database. It is better to know that before you start.
Whichever you choose, open the connection outside the handler. Then one warm environment reuses the same connection for every call. Your code must also cope with finding the connection dead, because the database closes connections that have sat idle for a while.
6. Serverless containers: the middle that usually wins
Google Cloud Run, AWS Fargate and Azure Container Apps sit between the two extremes. You give them a container image, and they run it without you managing any machines. They charge for each second the container is in use. Most of them can also scale to zero, which means that when no requests arrive, nothing runs and you pay nothing.
That combination gives you four things.
- You package the application your own way. It is an ordinary container with the runtime you chose. You do not have to fit the provider's special function format.
- The function limits are gone. There is no 15-minute limit and no 6 MB payload limit.
- One instance can serve many requests at the same time on some of these platforms. Fifty requests can share one container and one database connection, so the connection problem from section 5 goes away.
- The image is portable. The same image runs on your laptop, on Kubernetes and on any of these services.
You pay for this in two ways. Starting the first container from zero is slower than starting a function. And these services ask for a little more setting up and looking after than a function does.
For most teams building a web service, this is the right place to start. Kubernetes makes you run a cluster yourself (Chapter 14.3.1), and a managed container service does that work for you. You still get a portable container, and you avoid the limits of functions. Use functions when the work really is a reaction to events that arrive in bursts. Use Kubernetes when you need the things only Kubernetes offers.
7. When serverless is cheaper, and when it is not
A function is charged in two parts. You pay a small fee for each request. You also pay for the memory the function is given, multiplied by the time it runs. Whether this is cheap or expensive depends on how busy the function is.
Serverless is far cheaper when traffic is low on average and comes in spikes. Take an address on your site that is called 10,000 times a day. As a function it costs pennies. Without functions you would need a machine switched on all day for it. This is the real advantage of serverless, and it is a large one.
Serverless is more expensive when traffic is high and steady. Take a service that handles thousands of requests every second, all day. A machine running it is always busy, so none of what you pay for is wasted. A reserved instance does that work for several times less than functions would. Work out the crossing point with your own numbers. Between one application and another, the answer can differ by a factor of ten.
The calculation needs only three numbers about your traffic: requests per second, average duration, and memory. Here it is for an order-status API, using AWS's 2026 list prices in its largest region:
ts
// Should the order-status API run on Lambda or on always-on Fargate tasks?
const HOURS_PER_MONTH = 730;
const lambda = { perMillionRequests: 0.2, perGbSecond: 0.0000166667 }; // (1)
const fargate = { perVcpuHour: 0.04048, perGbHour: 0.004445 }; // (2)
type Workload = { requestsPerSecond: number; avgSeconds: number; memoryGb: number };
function lambdaMonthly(w: Workload): number {
const requests = w.requestsPerSecond * 3600 * HOURS_PER_MONTH;
const requestCost = (requests / 1_000_000) * lambda.perMillionRequests;
const computeCost = requests * w.avgSeconds * w.memoryGb * lambda.perGbSecond; // (3)
return requestCost + computeCost;
}
function fargateMonthly(w: Workload): number {
const perTask = (1 * fargate.perVcpuHour + 2 * fargate.perGbHour) * HOURS_PER_MONTH; // (4)
const rpsPerTask = 50; // (5)
const tasks = Math.max(2, Math.ceil(w.requestsPerSecond / rpsPerTask)); // (6)
return tasks * perTask;
}
for (const rps of [0.1, 5, 20, 100, 1_000]) {
const w: Workload = { requestsPerSecond: rps, avgSeconds: 0.12, memoryGb: 0.5 };
const l = lambdaMonthly(w).toFixed(2).padStart(9);
const f = fargateMonthly(w).toFixed(2).padStart(9);
console.log(`${String(rps).padStart(5)} req/s Lambda $${l} Fargate $${f}`);
}(1) Lambda charges twice: once per request, and once for memory multiplied by time, measured in gigabyte-seconds. (2) Fargate charges per hour for each CPU and each gigabyte of memory, whether or not requests arrive. (3) A 120 ms call at 512 MB uses 0.12 × 0.5 = 0.06 gigabyte-seconds. (4) One task with 1 CPU and 2 GB costs about $36 a month. (5) This is an assumption you must measure for your own service: how many requests one task can handle per second. (6) Never fewer than two tasks, one per zone, so a zone failure does not take the API down.
It prints:
0.1 req/s Lambda $ 0.32 Fargate $ 72.08
5 req/s Lambda $ 15.77 Fargate $ 72.08
20 req/s Lambda $ 63.07 Fargate $ 72.08
100 req/s Lambda $ 315.36 Fargate $ 72.08
1000 req/s Lambda $ 3153.61 Fargate $ 720.80Read the table from the top. At a request every ten seconds, Lambda costs 32 cents and two always-on tasks cost $72. Lambda wins by more than two hundred times. The lines cross a little above 20 requests per second. At 1,000 requests per second, Lambda costs more than four times as much. The answer flips because Lambda's price grows with every request, while a task's price stays flat until it is full. Change the duration or memory and the crossing point moves, which is why you run the numbers rather than trust a rule of thumb.
Functions also have costs that never appear on the invoice.
- Running the code on your own laptop is harder. The tools that imitate the cloud platform locally do not behave exactly like the real one.
- Seeing what the system is doing is harder. There is no long-running program to look inside. There are thousands of short runs. To follow one user's request through them you need distributed tracing (Chapter 10.10).
- You are tied closely to one provider. The format of the events, the permission rules and the packaging are all different on each provider.
- Finding a bug across many small functions is harder than finding it in one service.
A practical rule: use functions for small connecting jobs, for reacting to events and for spiky work. Use a container service for anything with a steady flow of requests.
8. Event-driven glue
A serverless system is mostly made of the services that sit between the functions and pass work from one to the next. There are four kinds.
Queues hold work that must not be lost. One function puts a message on the queue, and another takes it off and does the work. A message that keeps failing is moved aside to a second queue, called a dead-letter queue, so that a person can look at it (Chapter 10.8.1).
Event buses deliver each event to every function that has asked for that kind of event. This arrangement is called publish-subscribe. The sender publishes the event and does not need to know who receives it.
Storage and database events start a function when a file is uploaded or a database row changes. Nothing has to keep checking whether new work has arrived.
Managed workflow engines run a job that has many steps. AWS Step Functions, Azure Durable Functions and Azure Logic Apps are examples. You describe the steps, and the engine carries them out. It can run steps one after another or side by side, choose between branches, retry a failed step after a pause, and give up on a step that takes too long. It can also wait for hours or days with no program sitting there running. While it waits, the engine simply stores a note of where the job has got to. That is how a workflow gets past the 15-minute limit. A workflow engine is the right place for the saga pattern (Chapter 10.8.4) and for any long-running job.
Since December 2025, Lambda can do some of this itself. Lambda durable functions let you write the steps of a workflow as ordinary code. After each step, Lambda saves a checkpoint. A function can then pause, for minutes or for days, and you pay nothing while it waits. When it resumes, or when a step fails and is retried, Lambda runs the handler again from the start but skips every step that already has a saved result. That replay is why each step must give the same answer when it is skipped and re-read, and why anything random or time-dependent belongs inside a step, so its result is saved.
Two rules keep a system built from events under control.
Make every consumer idempotent. A consumer is the code that receives a message. Idempotent means that doing the same thing twice has the same result as doing it once. You need this because queues promise to deliver each message at least once. So now and then the same message arrives twice (Chapter 10.4). A consumer that charges a card must not charge it twice.
Give every flow a dead-letter queue, and put an alert on it. The typical failure in these systems makes no noise. A message fails, is retried a few times, and then is dropped. Nobody is watching, so nobody knows that an order was never processed.
9. Choosing
| Situation | Choose |
|---|---|
| Steady HTTP service | Managed container |
| Spiky or event-driven work | Functions |
| Long-running (> 15 min) | Container or workflow engine |
| Needs specific OS, kernel or hardware | Virtual machine |
| Many services, complex networking, a platform team | Kubernetes (Chapter 14.3.1) |
| Legacy application, unchanged | Virtual machine — lift and shift |
| Batch, interruptible | Spot instances or spot containers |
Start with the option that does the most work for you, as long as your application fits within its limits. Move to one that gives you more control only when a limit forces you to. Some teams start on Kubernetes because they expect to need it one day. They then spend their first year running a cluster, when they could have been building their product.
What the interviewer will push on
"When would you not use serverless functions?" Steady high volume (cost), work over 15 minutes, large payloads, latency-sensitive paths where cold starts matter, and anything needing a persistent connection pool. Then the strong version: the constraints are shape constraints, not knobs — a 9 MB response cannot be configured away, it must be redesigned as a reference.
"Why do functions cause database problems?" One environment serves one request, so concurrency means more environments and therefore more connections — 500 concurrent executions against a database sized for 20. Fix with a connection proxy, an HTTP data API, or a request-based store. Naming the one-request-per-environment model is what shows you understand it rather than repeating a warning.
"How do you reduce cold starts?" Smaller packages, faster runtimes, expensive work in initialisation, provisioned concurrency for the latency-sensitive path, and snapshot starts — with the caveat that a snapshot captures anything unique per instance, so seeds and credentials must be re-initialised. Then dismiss keep-warm pings: they warm one environment and do nothing for concurrent load.
"Increasing a function's memory made it cheaper. Why?" CPU is allocated proportionally to memory, so a CPU-bound function finishes proportionally faster and the gigabyte-seconds fall. It is counter-intuitive, it is real, and it is why you measure the cost curve rather than assuming more memory costs more.
"Kubernetes, managed containers or functions?" Start at the highest level that fits and move down when a constraint forces it. Managed containers are the right default for an HTTP service; functions for spiky event-driven glue; Kubernetes when you genuinely need it and have someone to operate it. Teams that start on Kubernetes spend their first year operating a cluster.
"Your instances got slow after a month with no deploy." Burstable instance credits exhausted, or storage burst credits, or the working set outgrowing memory. All three are gradual, none has a code change, and all three look like an application regression.
One thing to volunteer: mention that a function's /tmp can persist between invocations on the same environment, so user data written there is visible to the next request that lands on it. It is a real cross-request data-leak path that almost nobody accounts for, and the fix is simply to clean up.
Recall
- The ladder — virtual machine → container → managed container → function — removes work and adds constraint at each step. Start at the highest level that fits and move down only when a constraint forces it.
- Burstable instances throttle to a fraction of a core when credits run out, which is the "slow after a month with no deploy" failure. Right-size from two weeks of measurement, and prefer baked images and immutable infrastructure over configuring live machines.
- One function environment serves one request at a time — concurrency means more environments, hence more database connections. Fix with a connection proxy, an HTTP data API, or a request-based store.
- Cold starts: smaller packages, faster runtimes, work in initialisation, provisioned concurrency, and snapshot starts — which capture per-instance state such as seeds and credentials. Keep-warm pings warm one environment and do nothing for concurrency.
- Shape constraints, not knobs: 15 minutes; 6 MB per normal call, 200 MB streamed, 1 MB asynchronous; 250 MB unzipped package.
/tmpis 512 MB to 10 GB and may persist across invocations. Concurrency is 1,000 environments per region by default, and each function adds at most 1,000 every 10 seconds. - Lambda Managed Instances (2025) break the one-request-per-environment rule on purpose, for cost at steady volume, and bring back shared-state races. Lambda durable functions (2025) checkpoint each step and replay the handler, so steps must be safe to skip and re-read.
- CPU scales with configured memory, so raising memory on a CPU-bound function can reduce cost. Measure the curve.
- Managed containers are the right default for a steady HTTP service: scale to zero, your image, no 15-minute or payload limits, and multiple concurrent requests per instance.
- Serverless is much cheaper for spiky low-average load and more expensive at sustained volume. Managed workflow engines are the correct home for long-running orchestration and sagas, and every consumer needs idempotency plus a dead-letter path with an alert.
Self-test: Why can't a 9 MB response be fixed with configuration? · What exactly makes functions open too many database connections? · What does a snapshot-based cold start silently capture? · Why might more memory be cheaper? · What does a managed container give you that a function does not? · Which failure looks like an application regression but is a credit balance?
Next: this page kept sending large files to object storage. 13.4 covers that service: how it stores files, where it surprises people, and how a CDN makes a file kept in Sydney load quickly in Los Angeles.