Appearance
Services, DNS and Pod Networking
358. What problem does a Kubernetes Service solve when Pods are replaced or rescheduled?
The short answer
Pods come and go, and every new pod gets a new IP address, so nobody can rely on a pod's address. A Service gives a group of pods one stable virtual address and one stable DNS name that never change. Traffic sent to the Service is spread across whichever matching pods are ready at that moment. Clients only ever need the Service's name, however often the pods behind it are replaced.
Without a Service: the web app would need the IP addresses of the checkout pods. Every rollout, every crash and every scale-up would change that list, and the web app would have to keep up, or fail.
With a Service:
- A stable virtual IP, the ClusterIP, such as
10.96.14.20, for as long as the Service exists. - A stable DNS name,
checkout-api, or in fullcheckout-api.shop-prod.svc.cluster.local(question 364). - Load balancing across the ready pods that match its selector (question 360).
- Only healthy targets: pods that fail their readiness probe are taken out of the list until they recover (question 402).
The address is virtual. No machine owns 10.96.14.20. Each node's kube-proxy, or network plugin, catches traffic to it and redirects it to a real pod (question 320).
yaml
apiVersion: v1
kind: Service
metadata:
name: checkout-api
namespace: shop-prod
spec:
selector: { app.kubernetes.io/name: checkout-api }
ports:
- name: http
port: 80 # what clients connect to
targetPort: http # the pods' named container port, 8080359. What are ClusterIP, NodePort, and LoadBalancer Services, and when would you use each?
The short answer
A ClusterIP Service, the default, has only an internal address, reachable from inside the cluster. A NodePort Service also opens the same port, between 30000 and 32767, on every node, so traffic to any node's address on that port reaches the Service. A LoadBalancer Service also asks the cloud to create an external load balancer in front of it, with its own public or private address. Use ClusterIP for internal traffic, LoadBalancer or a gateway for traffic from outside, and NodePort mostly as a building block.
| Type | Reachable from | Address | Use for |
|---|---|---|---|
ClusterIP | Inside the cluster | Virtual internal IP | Service-to-service calls, the default |
NodePort | Anywhere that can reach a node | Node IP plus a port in 30000–32767 | Simple set-ups, a base for load balancers |
LoadBalancer | Outside, through the cloud | A cloud load balancer's IP | Exposing one service directly |
yaml
spec:
type: LoadBalancer
externalTrafficPolicy: Local # (1)
selector: { app.kubernetes.io/name: web }
ports: [{ port: 443, targetPort: 8443 }]externalTrafficPolicy: with the default,Cluster, a node that receives traffic may forward it to a pod on another node, which adds a hop and hides the client's real IP address. WithLocal, traffic only goes to pods on the receiving node, so the client's address is kept, but nodes with no matching pod must be taken out of the load balancer, which the cloud does using health checks.
What to use in practice:
- Internal services: always
ClusterIP. The book's shop learned this the hard way, when giving the back end its ownLoadBalancerput it on the internet, bypassing the gateway's protections (see 14.4.3). - Outside traffic: usually one shared entry point, an ingress controller or a Gateway API gateway, which itself is exposed by one
LoadBalancerService, and routes by host name and path to many internalClusterIPServices (question 371). - A
LoadBalancerper service only for traffic that a gateway cannot handle, such as raw TCP or UDP protocols.
Private load balancers. Clouds can create load balancers with only an internal address, using an annotation, for example service.beta.kubernetes.io/azure-load-balancer-internal: "true" on AKS, for services reached from your private network but not from the internet.
360. How does a Service select the Pods that receive traffic?
The short answer
Through its label selector. The Service lists labels, such as app.kubernetes.io/name: checkout-api, and every pod in the same namespace that carries all of those labels and is ready becomes a backend. A controller keeps that list, the Service's endpoints, up to date as pods appear, become ready, stop being ready or disappear. The Service's targetPort says which port on those pods receives the traffic.
yaml
apiVersion: v1
kind: Service
metadata:
name: checkout-api
spec:
selector:
app.kubernetes.io/name: checkout-api # (1)
ports:
- name: http
port: 80 # (2)
targetPort: http # (3)
---
# in the Deployment's pod template:
containers:
- name: api
ports:
- name: http
containerPort: 8080 # (4)- Any ready pod in this namespace with this label becomes a backend.
- The port clients use:
checkout-api:80. - The port on the pod that receives the traffic, given here by name.
- The container's port called
httpis 8080. Using a name means you can change the container port without touching the Service.
Ready is what counts. A matching pod is a backend only while its readiness probe passes. A pod that is starting up, overloaded or shutting down is left out (questions 402 and 488).
Same namespace only. A selector never picks pods from another namespace.
A Service without a selector. If you leave out spec.selector, no endpoints are created automatically. You can then create EndpointSlices by hand, pointing at addresses outside the cluster, such as a database on a virtual machine. Clients use a normal Service name, and you can later move the database into the cluster without changing them.
361. What are Service endpoints or EndpointSlices, and how would you check whether a Service has ready backends?
The short answer
EndpointSlices are the objects that list the actual pod addresses and ports behind a Service, along with whether each is ready, serving or terminating. A controller keeps them up to date, and kube-proxy, network plugins, ingress controllers and gateways read them to route traffic. They replaced the older single Endpoints object, which did not scale to very large Services. To check a Service's backends, list its EndpointSlices, or look at the Endpoints line of kubectl describe service.
bash
kubectl get endpointslices -n shop-prod -l kubernetes.io/service-name=checkout-api
kubectl describe service checkout-api -n shop-prodNAME ADDRESSTYPE PORTS ENDPOINTS AGE
checkout-api-x7k2p IPv4 8080 10.244.1.7,10.244.2.9,10.244.3.4 12dName: checkout-api
Selector: app.kubernetes.io/name=checkout-api
Type: ClusterIP
IP: 10.96.14.20
Port: http 80/TCP
TargetPort: http/TCP
Endpoints: 10.244.1.7:8080,10.244.2.9:8080,10.244.3.4:8080What each EndpointSlice records for every address:
| Condition | Meaning |
|---|---|
ready | Healthy and should receive traffic |
serving | Passing readiness, even if it is shutting down |
terminating | Being deleted |
Those conditions are how traffic is drained from pods during shutdown (question 488).
Why slices? The original Endpoints object held every address of a Service in one object. For a Service with 5,000 pods, one pod changing meant sending the whole huge object to every node. EndpointSlices split the list into pieces of up to 100 addresses each, so a change only updates one small slice.
An empty list is the key clue. If Endpoints: shows <none>, nothing will reach any pod, however healthy the Service itself looks. Question 362 explains the usual causes.
362. Why might a Service exist but have no working endpoints?
The short answer
Usually because the selector does not match the pods' labels, or because the matching pods are not ready. Other causes are pods living in a different namespace from the Service, no pods running at all, or, when endpoints exist but connections fail, a wrong targetPort or a network policy blocking the traffic. Check the selector against the pod labels, then the pods' readiness, then the ports, then the policies.
Work through it in this order:
- Do the labels match?
bash
kubectl get svc checkout-api -n shop-prod -o jsonpath='{.spec.selector}{"\n"}'
kubectl get pods -n shop-prod --show-labels | grep checkoutA typo, such as app: checkout against app: checkout-api, or a label key that changed in a refactor, gives an empty endpoint list (question 347).
- Are the pods ready?
bash
kubectl get pods -n shop-prod -l app.kubernetes.io/name=checkout-api0/1 in the READY column means the readiness probe is failing, so the pod is left out (question 409).
- Same namespace? A Service only selects pods in its own namespace.
- Any pods at all? The Deployment may be scaled to zero, or its pods may all be crash-looping.
- Endpoints present, but connections fail?
- Wrong
targetPort. The Service sends traffic to port 80 on the pods, but the app listens on 8080. Endpoints look fine, and every connection is refused. - A named port that does not exist on the container.
- A network policy that blocks traffic from the caller (question 367).
- The app listening on
127.0.0.1inside the pod instead of all addresses (question 119).
- Wrong
A quick test from inside the cluster:
bash
kubectl run tmp --rm -it --image=busybox:1.36 -n shop-prod -- wget -qO- http://checkout-api/healthzIf this fails while a direct request to one pod's IP works, the problem is in the Service's selector or ports. If both fail, look at the pods themselves.
363. How does service discovery work between Pods in the same cluster?
The short answer
Through the cluster's DNS server, usually CoreDNS. Every pod's /etc/resolv.conf points at it, and it answers names of Services with their ClusterIP, such as checkout-api.shop-prod.svc.cluster.local resolving to 10.96.14.20. Search domains let pods use short names, such as just checkout-api within the same namespace. Kubernetes also injects environment variables for Services that existed when a pod started, but DNS is the method to rely on.
What a pod's DNS settings look like:
bash
kubectl exec -n shop-prod web-6f5c-abcde -- cat /etc/resolv.confsearch shop-prod.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5nameserver 10.96.0.10is the ClusterIP of thekube-dnsService, behind which CoreDNS pods run.searchlists suffixes to try for short names.checkout-apiis tried ascheckout-api.shop-prod.svc.cluster.localfirst, which is the same namespace's Service.ndots:5means "if a name has fewer than five dots, try the search suffixes first". It is why short names work, and also why external names can cause extra lookups (question 375).
What CoreDNS answers:
| Name | Answer |
|---|---|
checkout-api.shop-prod.svc.cluster.local | The Service's ClusterIP |
_http._tcp.checkout-api.shop-prod.svc.cluster.local | An SRV record with the named port's number |
orders-db-0.orders-db.shop-prod.svc.cluster.local | One StatefulSet pod's IP, through a headless Service (question 489) |
Anything else, such as api.stripe.com | Forwarded to upstream DNS servers |
How CoreDNS knows. It watches Services and EndpointSlices through the API server, so a new Service is resolvable within seconds.
The environment variables, such as CHECKOUT_API_SERVICE_HOST=10.96.14.20, are an older mechanism. They are only set for Services that already existed when the pod started, so they go stale. Use DNS names.
364. How would one application call another application using its Service DNS name?
The short answer
By using the Service's name as the host in its URL, with the Service's port. From the same namespace, the short name is enough: http://checkout-api. From another namespace, add the namespace: http://checkout-api.shop-prod. The fully qualified name, checkout-api.shop-prod.svc.cluster.local, always works. Keep the address in configuration, not in code, so each environment can point somewhere different.
| Caller is in | Name to use | Resolves because |
|---|---|---|
shop-prod, same namespace | checkout-api | The first search suffix adds .shop-prod.svc.cluster.local |
shop-qa, another namespace | checkout-api.shop-prod | The search suffix adds .svc.cluster.local |
| Anywhere | checkout-api.shop-prod.svc.cluster.local | It is the full name |
In the shop's web app, written in TypeScript:
ts
const CHECKOUT_URL = process.env.CHECKOUT_URL ?? 'http://checkout-api' // (1)
export async function placeOrder(order: OrderRequest): Promise<OrderResult> {
const res = await fetch(`${CHECKOUT_URL}/orders`, { // (2)
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(order),
signal: AbortSignal.timeout(3_000), // (3)
})
if (!res.ok) throw new Error(`checkout-api replied ${res.status}`)
return (await res.json()) as OrderResult
}- The address comes from configuration, defaulting to the same-namespace Service name. Port 80 is implied for
http://, which matches the Service'sport: 80. - The request goes to the Service. DNS turns
checkout-apiinto the ClusterIP, and the node's rules send it to one ready pod. - A timeout, so a slow or missing dependency fails fast instead of hanging the request.
Common mistakes:
- Using the container port instead of the Service port. Clients use the Service's
port. ThetargetPortis internal to the Service. - Using a pod's IP or name. Both change. Always the Service.
- Forgetting the namespace when calling across namespaces, which fails with a "could not resolve host" error.
365. How do Pod networking and node networking differ conceptually?
The short answer
Node networking is the ordinary network the machines sit on, such as the cloud's virtual network, with one address per node. Pod networking gives every pod its own address, from a separate range, and Kubernetes requires that every pod can reach every other pod's address directly, across nodes, without address translation. Services add a third, purely virtual range of addresses that exist only as rules on the nodes.
The Kubernetes network model sets three rules that every network plugin must provide:
- Every pod gets its own unique IP address in the cluster.
- Every pod can reach every other pod's address directly, on any node, without address translation.
- Agents on a node, such as the kubelet, can reach every pod on that node.
How plugins make pod addresses reachable across nodes (question 366):
- Overlay: pod traffic between nodes is wrapped inside packets addressed node to node, then unwrapped. The pod range is invisible to the underlying network.
- Routed: nodes tell the network which pod range lives behind them, so packets are routed normally.
- Cloud-native addressing: pods get addresses straight from the cloud network, like the nodes. Simple to reason about, but it uses up network addresses quickly (question 614).
Why the ranges must not overlap. If the pod range, the Service range and your company's own networks share addresses, traffic meant for an office server can end up inside the cluster, or the reverse. Choose the ranges when the cluster is created, because changing them later usually means building a new cluster.
366. What is a CNI plugin, and what responsibility does it provide?
The short answer
CNI, the Container Network Interface, is a standard for how a container runtime asks a network plugin to connect a pod. When a pod's sandbox is created, the runtime calls the plugin, which creates the pod's network interface, gives it an IP address, sets up routes, and connects it so it can reach every other pod. When the pod is deleted, the plugin cleans up. Many plugins also enforce network policies, encrypt traffic and record network activity. Common ones are Calico, Cilium, Flannel, Azure CNI and AWS VPC CNI.
What happens when a pod starts:
- The runtime creates the pod's network namespace (question 334).
- It calls the CNI plugin with "add this pod".
- The plugin creates a veth pair, a virtual cable, with one end in the pod as
eth0and the other on the node (question 111). - Its IPAM part, IP address management, hands out a free address from the node's pod range, or from the cloud network.
- It sets routes, so the pod's traffic leaves through its cable and other pods' traffic finds its way in.
- It reports the address back, and Kubernetes records it as the pod's IP.
How plugins differ:
| Plugin | How pods reach other nodes | Network policies | Notes |
|---|---|---|---|
| Flannel | Overlay | No, on its own | Simple |
| Calico | Routed, or overlay | Yes | Widely used, strong policy support |
| Cilium | eBPF, routed or overlay | Yes, including by application protocol | Can replace kube-proxy; detailed visibility |
| Azure CNI | Cloud addresses, or an overlay mode | Through Azure, Calico or Cilium | AKS's main options |
| AWS VPC CNI | Cloud addresses from the VPC | With an add-on or Cilium | EKS default |
The choice matters for: how many pods fit in your network address space, whether network policies are enforced at all, how fast traffic is, and how much you can see when debugging. It is very hard to change on a running cluster, so it is decided at creation time (question 614).
367. How do NetworkPolicies control which Pods can communicate, and what must be true for a policy to take effect?
The short answer
A NetworkPolicy selects some pods and lists the traffic allowed to reach them, their ingress, or leave them, their egress, by pod labels, namespace labels, IP ranges and ports. Pods that no policy selects allow all traffic. As soon as a policy selects a pod for a direction, only the traffic it lists is allowed in that direction. For any of this to work, the cluster's network plugin must enforce network policies. If it does not, the policies are stored and silently ignored.
The usual starting point: deny everything, then allow what is needed.
yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: shop-prod
spec:
podSelector: {} # (1)
policyTypes: [Ingress, Egress] # (2)
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: checkout-api-allow
namespace: shop-prod
spec:
podSelector:
matchLabels: { app.kubernetes.io/name: checkout-api }
policyTypes: [Ingress, Egress]
ingress:
- from:
- podSelector:
matchLabels: { app.kubernetes.io/name: web } # (3)
ports: [{ port: 8080, protocol: TCP }]
egress:
- to:
- namespaceSelector:
matchLabels: { kubernetes.io/metadata.name: kube-system }
podSelector:
matchLabels: { k8s-app: kube-dns } # (4)
ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
- to:
- ipBlock: { cidr: 10.20.5.0/24 } # (5)
ports: [{ port: 5432, protocol: TCP }]- An empty selector selects every pod in the namespace.
- With no rules listed, every pod here may now send and receive nothing.
checkout-apimay receive traffic on port 8080, only fromwebpods in this namespace.- It may reach the cluster's DNS. Forgetting this is the most common mistake: with egress denied, every name lookup fails, and the app reports confusing connection errors.
- It may reach the managed PostgreSQL server's subnet on port 5432.
Rules worth knowing:
- Policies only add permissions. There is no "deny" rule. Several policies selecting the same pod combine, and traffic is allowed if any of them allows it.
- Both ends matter. For a connection from
webtocheckout-apito work,web's egress must allow it, ifwebhas egress policies, andcheckout-api's ingress must allow it. - Replies are automatic. Return traffic of an allowed connection is allowed without a separate rule.
- The plugin must enforce them. Calico, Cilium and Azure's network policy options do. Flannel alone does not. Check with a quick test: apply a deny-all and confirm traffic really stops.
489. What are headless and ExternalName Services, and when would you use them?
The short answer
A headless Service, created with clusterIP: None, has no virtual address and no load balancing. Instead, its DNS name returns the IP addresses of the pods themselves, and with a StatefulSet each pod also gets its own stable DNS name. Use it when clients must reach specific pods, such as a database's primary. An ExternalName Service has no pods at all: its DNS name simply points, as an alias, at a name outside the cluster, such as a managed database's address.
Headless:
yaml
apiVersion: v1
kind: Service
metadata:
name: orders-db
spec:
clusterIP: None
selector: { app: orders-db }
ports: [{ port: 5432 }]bash
kubectl run tmp --rm -it --image=busybox:1.36 -- nslookup orders-dbName: orders-db.shop-prod.svc.cluster.local
Address: 10.244.1.21
Address: 10.244.2.17
Address: 10.244.3.30Three pod addresses, not one virtual address. With the StatefulSet from question 340, each pod also has its own name:
text
orders-db-0.orders-db.shop-prod.svc.cluster.local → 10.244.1.21Use headless when:
- Each pod has a role. "Write to
orders-db-0, read from the others". - Clients balance by themselves. For example, gRPC clients that keep long-lived connections and want to know every backend.
- Peers must find each other, such as members of a database cluster discovering one another.
ExternalName:
yaml
apiVersion: v1
kind: Service
metadata:
name: orders-db
namespace: shop-prod
spec:
type: ExternalName
externalName: shop-orders.postgres.database.azure.comPods connect to orders-db, and DNS answers with an alias, a CNAME record, for the Azure address. No traffic passes through Kubernetes at all.
ExternalName caveats:
- No ports, no load balancing, no selection: it is only a DNS alias.
- TLS names. The client connects to
orders-db, but the server's certificate namesshop-orders.postgres.database.azure.com. Clients that check the certificate against the name they dialled will fail unless configured with the real name. - Not an IP address.
externalNamemust be a DNS name.
490. What is kubectl port-forward, and how is it different from a Service?
The short answer
kubectl port-forward opens a port on your own machine and tunnels connections to it, through the API server and the kubelet, to one pod in the cluster. It is a debugging and admin tool: for example, to open a database or an internal dashboard from your laptop without exposing it. A Service is the cluster's real, permanent way of routing traffic to pods, with a stable address and load balancing. Port-forward is temporary, goes to a single pod, and depends on your session staying open.
bash
kubectl port-forward -n shop-prod svc/checkout-api 8080:80 # (1)
kubectl port-forward -n shop-prod pod/orders-db-0 5432:5432 # (2)
curl http://localhost:8080/healthz # (3)- Forwards your local port 8080 to the Service's port 80. Behind the scenes,
kubectlpicks one pod behind the Service and tunnels to it for the whole session. - Forwards straight to one particular pod, here the database's first instance.
- Your laptop now reaches the pod as if it were local.
kubectl port-forward | Service | |
|---|---|---|
| Purpose | Debugging, admin access | Production routing |
| Lasts | While the command runs | Until deleted |
| Goes to | One pod | Any ready pod, balanced |
| Path | Your machine, API server, kubelet, pod | Node rules straight to pods |
| Who can use it | Anyone with the pods/portforward permission | Anything on the network allowed by policies |
Security points:
- It needs the
createpermission on thepods/portforwardsubresource. Anyone who has it can reach any port of pods in that namespace, bypassing gateways and some network policies. Grant it carefully, especially in production (question 416). - It is fine for an engineer to open a database tunnel briefly, but never as a way to serve real traffic.