Appearance
13.2 — Cloud Networking in Practice
A company starts with one private network in the cloud, and it works. Two years later it has six of them, an office network and a partner's network. Now it needs them to talk to each other, and it finds that two of them cannot be joined at all. This page is about the problems that appear only when you have more than one network.
Chapter 5.10 built the basic pieces. Four of them matter here, so here they are again in brief.
- A virtual network is a block of private addresses, such as
10.0.0.0/16, that the provider keeps separate from every other customer's. AWS calls it a VPC, short for virtual private cloud. Azure calls it a VNet. - A subnet is a slice of that block. Every subnet has a route table, which is a list of rules saying where traffic for each destination should be sent. The destination
0.0.0.0/0means "every address not listed elsewhere". A subnet is public only if its route table sends0.0.0.0/0to an internet gateway, the device that joins the virtual network to the internet. Otherwise the subnet is private. - Two kinds of firewall rule protect the machines. A security group is attached to a machine, and it is stateful. That means it remembers each request it let through, so the reply is let back in automatically. A network ACL (access control list) is attached to a subnet, and it is stateless. It remembers nothing, so the reply needs a rule of its own.
- Two networks whose address ranges overlap can never be connected to each other. The only fix is to give one of them new addresses while it is in use.
1. Address planning, before it is too late
Almost every company makes the same mistake once. It uses 10.0.0.0/16 for everything.
That range is the one every tutorial uses, so the first network gets it. Then the production network gets it too. So do the staging and development networks, and so does the network of a company you bought last year. Now imagine a machine that wants to send a message to 10.0.3.7. That address exists in all of those networks. Nothing can tell which one is meant. So these networks can never be connected. The only cure is to change every address in one of them, while the systems inside it are running.
IPAM stands for IP address management. It means one central place hands out address ranges, and no team ever picks its own. The cloud providers sell a service that does this. A shared spreadsheet that everybody really uses does the job too, and it is better than a service nobody uses.
A workable plan looks like this:
10.0.0.0/8 organisation
10.0.0.0/12 production (10.0.0.0 – 10.15.255.255)
10.0.0.0/16 prod-eu-west
10.1.0.0/16 prod-us-east
10.16.0.0/12 non-production
10.32.0.0/12 shared services (DNS, directories, tooling)
10.240.0.0/12 reserved for acquisitions and future regionsThe number after the slash tells you how big each block is. This way of writing a range is called CIDR notation (Chapter 5.3.1), and a range written this way is often just called "a CIDR". An IPv4 address has 32 bits. The number after the slash says how many of those bits are fixed. The rest are free, and each free bit doubles the count:
| Block | Free bits | Addresses |
|---|---|---|
/8 | 24 | 16,777,216 |
/12 | 20 | 1,048,576 |
/16 | 16 | 65,536 |
/20 | 12 | 4,096 |
/24 | 8 | 256 |
So in the plan above, the production /12 holds sixteen /16 networks, and each /16 holds sixty-four /22 subnets of about a thousand addresses each. Every subnet also loses a few addresses to the provider. AWS keeps the first four and the last one in every subnet. The second one, base + 2, is the DNS resolver. So a /24 subnet gives you 251 usable addresses, not 256.
Three rules that save years of pain:
Leave large gaps. Address space is free and renumbering is not. A /16 per virtual network is generous and it is the right default.
Never reuse a range, even after decommissioning, until you are certain nothing routes to it.
Record allocations somewhere a person must go through to get one. The failure is always someone in a hurry picking a range that "looked free".
"Looked free" fails because a larger block hides smaller ones inside it. Checking for an overlap is simple once you turn each block into a start address and an end address. Here is the check an IPAM tool makes before it hands out a range:
ts
// IPAM check: does a requested range collide with anything already allocated?
type Cidr = { text: string; start: number; end: number };
function parseCidr(text: string): Cidr {
const [ip, bits] = text.split("/");
const octets = ip.split(".").map(Number);
const address = octets.reduce((acc, o) => acc * 256 + o, 0); // (1)
const size = 2 ** (32 - Number(bits)); // (2)
const start = address - (address % size); // (3)
return { text, start, end: start + size - 1 };
}
function overlaps(a: Cidr, b: Cidr): boolean {
return a.start <= b.end && b.start <= a.end; // (4)
}
const allocated = ["10.0.0.0/16", "10.1.0.0/16", "10.16.0.0/12", "172.16.0.0/12"].map(parseCidr);
for (const request of ["10.1.128.0/20", "10.2.0.0/16", "10.31.0.0/16"]) {
const wanted = parseCidr(request);
const clash = allocated.find((existing) => overlaps(existing, wanted));
const addresses = wanted.end - wanted.start + 1;
console.log(request, addresses, "addresses", clash ? `CLASHES with ${clash.text}` : "free");
}(1) An address like 10.1.128.0 is really one 32-bit number. Each of the four parts is a base-256 digit, so we build the number the way you would build 1,234 from its digits. (2) A /20 has 12 free bits, so it holds 2¹² = 4,096 addresses. (3) A block always starts on a multiple of its own size, so we round the address down to the nearest multiple. (4) Two ranges overlap when each one starts before the other ends. This one comparison covers every case: one inside the other, partial overlap, and exact match.
It prints:
10.1.128.0/20 4096 addresses CLASHES with 10.1.0.0/16
10.2.0.0/16 65536 addresses free
10.31.0.0/16 65536 addresses CLASHES with 10.16.0.0/12The third line is the one people get wrong by eye. 10.31 looks nowhere near 10.16. But a /12 starting at 10.16.0.0 runs all the way to 10.31.255.255, so 10.31.0.0/16 sits inside it.
Some address ranges are easy to forget, and they belong in the plan too.
- The network in your own office or data centre.
- The networks of partner companies you connect to.
- The addresses handed to staff laptops when they connect from home through a VPN.
- The internal ranges of a Kubernetes cluster. Kubernetes is a system that runs many copies of your applications across a group of machines, and Part 14 covers it. It gives every running copy an address of its own, taken from a range chosen by whoever created the cluster (Chapter 14.3.6).
None of these may collide with each other. Suppose a cluster's internal range overlaps the office network. Then a program inside the cluster that tries to reach an office machine may reach a neighbour inside the cluster, or nothing at all. Some office machines can be reached and others cannot. That pattern looks random, so it is very hard to diagnose.
2. Connecting to what you already have
Almost every organisation has something outside the cloud. It may be an office, a factory, or an old data centre that still runs the payroll system. A setup with one part in the cloud and one part on your own premises is called hybrid. There are two main ways to join the two parts.
A site-to-site VPN. VPN stands for virtual private network. It is an encrypted tunnel that runs over the public internet. Everything entering one end is scrambled, and only the other end can unscramble it. A VPN is cheap, and you can have one working within an hour. But it travels over the internet. So its speed goes up and down with the internet's, and when the internet path breaks, the VPN breaks. A VPN is the right choice for small amounts of traffic that are not critical, and as a backup path.
A dedicated connection. This is a private cable from your network to the provider's network, rented through a telecoms company. AWS calls its version Direct Connect, Azure calls it ExpressRoute, and Google calls it Cloud Interconnect. It takes weeks to months to set up, because a real cable has to be arranged. In return the delay is steady and the line carries a lot of traffic. There is one more benefit, and it is often the one that pays for the line. Egress over a dedicated connection is charged at a much lower rate. A company that moves a great deal of data out of the cloud can spend less in total, even after paying for the line.
A dedicated connection is not encrypted by default. It is private, in the sense that nobody else shares the cable. But the data on it is not scrambled. If the data is sensitive, run a VPN over the dedicated connection, or make sure each application encrypts its own traffic with TLS (Chapter 5.7).
Most production systems use both. The dedicated connection carries the traffic. A VPN stands by and takes over automatically if the cable fails. That gives steady performance on a normal day, and the system survives the loss of the cable.
Large companies with many branch offices often add a third tool, called a software-defined WAN. A WAN, or wide area network, is the network that joins a company's distant sites together. "Software-defined" means that all of those links, and the links to the cloud, are set up from one central program, so nobody has to configure each office's equipment by hand.
The two sides tell each other which addresses they hold using BGP, the Border Gateway Protocol (Chapter 5.3.2). Each side advertises the address ranges it can reach. The other side then sends traffic for those ranges down the link. Two things go wrong:
- Advertising more than you mean to. An office router that advertises
10.0.0.0/8as "mine" also claims every cloud network inside that block. Traffic meant for the cloud now heads to the office. Nothing errors, and some connections quietly stop working. Advertise only the exact ranges you own. - Asymmetric routing. With two links, a request can leave by the VPN and the reply can come back over the dedicated connection. A stateful firewall (Chapter 5.10) on the return path never saw the request go out. So it treats the reply as an unknown stranger and drops it. Make sure each flow goes out and comes back on the same path.
3. DNS across the boundary
DNS is the system that turns a name such as example.com into an address (Chapter 5.5). Turning a name into an address is called resolving it, and the server that does the work is called a resolver. Once the office and the cloud are joined, machines on each side need to resolve the names of machines on the other side. A large share of the faults in hybrid setups turn out to be DNS faults.
A private hosted zone is a list of names that resolve only inside your virtual network. For example, db.internal.example.com can resolve to the private address of your database. From the internet, that name does not exist. This lets your applications use names and never write a private address into their settings. Private addresses change whenever a machine is replaced, so you need that.
Split-horizon DNS gives different answers for the same name, depending on who asks. Inside the network, api.example.com resolves to a private address. Outside, the same name resolves to a public one. This is useful, and it regularly confuses people. When someone says "it resolves differently on my laptop", this is usually the cause, and it is working exactly as designed.
Resolving office names from the cloud needs a forwarding rule. The rule tells the cloud's resolver: "send any question about corp.example.com to these DNS servers in the office."
Resolving cloud names from the office needs the same thing in reverse, and it has two parts. First you create an inbound resolver endpoint. That is an address inside the virtual network that accepts DNS questions from outside it. Then you tell the office DNS servers to send questions about your cloud names to that address. A rule that forwards only the questions about certain names is called conditional forwarding.
The two directions are set up separately, and people often set up only one. The result is confusing. Applications in the cloud reach the office with no trouble. Nothing in the office can find anything in the cloud.
Remember where the cloud's resolver lives. Inside a virtual network, the resolver sits at the network's first address plus two (Chapter 5.10). In 10.0.0.0/16 that is 10.0.0.2. If a security group or firewall rule blocks that address, no name can be resolved, and everything fails. It looks like a broken network, when the real fault is DNS.
4. How traffic gets in and out
A request from a user passes through five steps on its way in.
- DNS turns your site's name into an address (Chapter 5.5).
- A CDN may receive the request first. A CDN, or content delivery network, is a set of servers placed close to users around the world. The user's connection ends at one of those servers, and that server talks to your system on the user's behalf (Chapter 13.4).
- A load balancer receives the request next. It sits in the public subnets, with a presence in each zone, and shares the requests across your servers. Chapter 10.15 explains the two kinds, called layer 4 and layer 7.
- A web application firewall, or WAF, inspects each request and blocks those that look like known attacks. It is usually attached to the load balancer or to the CDN.
- Your servers sit in private subnets. Their security group accepts traffic only from the load balancer, so nothing else can reach them.
An API gateway also sits at this entrance, and it does jobs that a load balancer does not do. It checks who each caller is and what they are allowed to do. It limits how many requests each customer may send. It can reshape a request before passing it on, and reshape the reply on the way back. It hands out API keys and counts each customer's usage. And it sends each kind of request to the right service behind it. A short way to remember the difference: a load balancer spreads traffic, and an API gateway manages an API. Chapter 10.8.3 covers the idea in general.
Traffic going out is where most of the money goes. There are three ways out.
- A NAT gateway lets machines in private subnets reach the internet, while nothing on the internet can start a connection to them (Chapter 5.10). You pay for every hour it runs, and again for every gigabyte it handles. You also need one in each zone, so that losing a zone does not cut off the others.
- A VPC endpoint is a private doorway from your network to the provider's own services, such as object storage. Traffic through it skips the NAT gateway completely. Chapter 5.10 showed that this is safer. It is also much cheaper. A job that pulls terabytes from object storage through a NAT gateway runs up a large bill, and the endpoint makes that bill disappear.
- An egress firewall inspects outgoing traffic and allows only the destinations you have listed, by domain name. This is where the egress filtering of Chapter 8.6.2 is carried out. It is also the place to block the metadata address. That is a special address,
169.254.169.254, at which a cloud machine can read details about itself, including temporary passwords. An attacker who tricks your server into fetching that address can steal them.
Know every path by which a byte can leave your network, and know what each path costs.
5. Why IPv4 costs money now
Chapter 5.3.3 explained the exhaustion: there are only about 4.3 billion IPv4 addresses, and they ran out years ago. The commercial consequence arrived in February 2024. AWS now charges for every public IPv4 address, whether it is attached to something or sitting idle, and other providers charge similarly. The AWS price is $0.005 an hour. That sounds like nothing, but it is $43.80 a year per address. An estate with 2,000 public addresses pays about $88,000 a year just to hold them. The charge is also a deliberate push towards IPv6, which has so many addresses that they are free.
What to do about it:
- Do not give your machines public addresses. Put them in private subnets. Let traffic come in through a load balancer and go out through a NAT gateway.
- Release addresses that are attached to nothing. They are charged whether you use them or not, and most companies have some that were reserved once and forgotten.
- Put many services behind one load balancer. One load balancer per service means one public address per service.
- Use IPv6 where every step of the path supports it. IPv6 addresses carry no charge. In practice you run dual-stack, which means each machine has both an IPv4 and an IPv6 address. You need both because parts of the internet, and many partner systems, still speak only IPv4.
IPv6 in the cloud works well, but it does take some effort. Some managed services do not support it yet. Outbound traffic uses a different device: an egress-only internet gateway takes the place of the NAT gateway. And any of your code that stores or compares addresses has to change. An IPv6 address is 128 bits long, where an IPv4 address is 32. Code must also cope with an IPv4 address written in IPv6 form, such as ::ffff:10.0.0.1 (Chapter 5.9).
6. Network cost, made explicit
Nobody sees network charges until the invoice arrives, so here they are in one table:
| Traffic | Typical charge |
|---|---|
| Internet in | Free |
| Internet out | Per GB, the largest line |
| Between zones, same region | Per GB, each direction |
| Within one zone, private addresses | Usually free |
| Between regions | Per GB, higher |
| Through a NAT gateway | Per GB plus hourly |
| Through a VPC endpoint | Much cheaper or free |
| Via a dedicated connection | Substantially reduced egress |
Three architectural consequences:
Traffic between zones costs real money. A system of many small services, spread across three zones, pays per gigabyte each time one of its services calls another in a different zone. The fix is zone-aware routing. Each service first tries a copy of the other service in its own zone, and crosses to another zone only when that copy is down. This also makes the calls faster.
Move the computing to the data. Do not move the data to the computing. If the data lives in one region, run the job that processes it in the same region. Then nothing is transferred between regions and nothing is charged.
Always use endpoints to reach the provider's own services. An endpoint is cheaper, faster and safer, all at the same time. It is rare for one change to improve all three.
7. Debugging, in order
Chapter 5.10 gave six checks for one network. When the office is connected too, or when the company has many cloud accounts, work through this longer list in order.
- Is it DNS? Resolve the name from the machine that is failing, not from your laptop. Check which resolver gave the answer. With split-horizon DNS and forwarding rules in play, DNS is the first thing to suspect.
- Is there a route? Read the route table on both sides. Often the cloud has a route to the office, but the office has no route back to the cloud. Then a connection starts to open and hangs for ever, because the replies have nowhere to go.
- Is it allowed? Check the security groups in both directions. Check the network ACLs. They are stateless, so the reply needs a rule of its own. A reply comes back to a temporary, high-numbered port on the machine that asked, so that whole range of ports must be allowed. Then check the office firewalls and any egress firewall.
- Is anything listening? A program can be running and still accept connections only from its own machine. Chapter 5.9 covers this bind-address mistake.
- Is the path asymmetric? The request goes out by one path and the reply comes back by another. A stateful device on the return path never saw the request, so it drops the reply.
- Do the ranges overlap? If two networks share a range, some destinations work and others do not, and the pattern looks random.
Flow logs settle most of these questions. A flow log is a record, kept by the provider, of every connection that was accepted or rejected in your network. Each line gives the source, the destination, the port and what happened. So a vague complaint such as "the network is blocking us" turns into one named rule and one named connection.
What the interviewer will push on
"How would you plan addressing for an organisation on a cloud platform?" Central allocation from a large block, generous per-network ranges, big reserved gaps, and never letting a team pick. Then the reason: overlapping ranges cannot be routed and the fix is renumbering a live network, so this is one of very few genuinely irreversible decisions. Include Kubernetes pod ranges and VPN pools in the plan.
"VPN or a dedicated connection?" VPN for speed of provisioning, low bandwidth and as a backup; dedicated for predictable latency, high bandwidth and cheaper egress, which often pays for it. Then the detail people miss: a dedicated connection is private but not encrypted. Production usually runs both, with VPN as failover.
"Hybrid DNS is broken in one direction. What is wrong?" Only one half is configured. Cloud-to-office needs forwarding rules; office-to-cloud needs an inbound resolver endpoint plus conditional forwarding from the on-premises servers. The tell is knowing the two directions are separate configurations.
"Where does cloud network spend actually go?" Internet egress, cross-zone traffic in both directions, and NAT gateway per-gigabyte processing. The fixes are zone-aware routing, VPC endpoints for provider services, and moving compute to the data. Endpoints are the rare change that is cheaper, faster and more secure simultaneously.
"Why did public IPv4 addresses start costing money?" Exhaustion made them a scarce traded asset, and charging is both cost recovery and an incentive to move. The practical response is private subnets behind load balancers and NAT, releasing unattached addresses, and dual-stack — with the honest caveat that IPv6 support in managed services is good and uneven.
"A connection to an on-premises service opens and then hangs. Where do you look?" Asymmetric routing or a missing return route, and a stateful firewall dropping the reply because it never saw the outbound flow. Then reach for flow logs rather than arguing — they name the rule and the packet.
One thing to volunteer: mention that a Kubernetes cluster's internal address range, called the pod CIDR, is part of the address plan. Whoever created the cluster chose it, and it often overlaps a range the company already uses. Then half of the office machines cannot be reached from the cluster. That is one of the hardest hybrid problems to diagnose from the symptom alone.
Recall
- Overlapping address ranges are the one irreversible mistake. Allocate centrally (IPAM), leave large gaps, never reuse a range, and include Kubernetes pod/service ranges and VPN pools in the plan.
- VPN is fast to provision and inherits internet variability; a dedicated connection gives predictable latency, high bandwidth and cheaper egress — and is private, not encrypted. Production runs both.
- Hybrid DNS is two separate configurations: forwarding rules for cloud-to-office, an inbound resolver endpoint plus conditional forwarding for office-to-cloud. Split-horizon explains "it resolves differently on my laptop".
- Inbound path: DNS → CDN → load balancer in public subnets → web application firewall → private targets. A load balancer distributes traffic; an API gateway manages an API — authentication, quotas, transformation, per-route backends.
- Know every path a byte leaves by. NAT gateway is hourly plus per gigabyte; VPC endpoints bypass it, which is cheaper, faster and more secure at once; an egress firewall is where Chapter 8.6.2's filtering is enforced.
- Public IPv4 addresses are charged now, attached or not. Private subnets behind load balancers, release unattached addresses, and move to dual-stack — with the caveat that managed-service IPv6 support is uneven.
- Network cost: in is free, out is charged, cross-zone is charged in both directions. Fix with zone-aware routing, endpoints, and moving compute to the data.
- Debug in order: DNS → route (both sides) → security groups and ACLs → is anything listening → asymmetric routing → overlapping ranges. Flow logs end the argument.
Self-test: Why is an overlapping CIDR unfixable in practice? · What does a dedicated connection not give you? · Which half of hybrid DNS is usually missing? · Name the two charges on a NAT gateway and what removes both. · Why did IPv4 addresses start costing money? · What symptom points at asymmetric routing?
Next: the network is now in place. 13.3 covers what runs on it: virtual machines, containers and serverless functions. It shows how the cost of each changes with traffic, and which fixed limits decide whether serverless fits your work at all.