Skip to content

Interview PrepKubernetes

Services, DNS and Pod Networking ​

358. What problem does a Kubernetes Service solve when Pods are replaced or rescheduled? ​

The short answer

Pods come and go, and every new pod gets a new IP address, so nobody can rely on a pod's address. A Service gives a group of pods one stable virtual address and one stable DNS name that never change. Traffic sent to the Service is spread across whichever matching pods are ready at that moment. Clients only ever need the Service's name, however often the pods behind it are replaced.

web podcalls checkout-apiService checkout-api10.96.14.20never changespod 10.244.1.7pod 10.244.2.9pod 10.244.3.4new pod10.244.1.12replaced
A fixed front for moving pods. The web pod always calls the same name and address. When a pod is replaced, the Service quietly starts sending traffic to the new one instead.

Without a Service: the web app would need the IP addresses of the checkout pods. Every rollout, every crash and every scale-up would change that list, and the web app would have to keep up, or fail.

With a Service:

  1. A stable virtual IP, the ClusterIP, such as 10.96.14.20, for as long as the Service exists.
  2. A stable DNS name, checkout-api, or in full checkout-api.shop-prod.svc.cluster.local (question 364).
  3. Load balancing across the ready pods that match its selector (question 360).
  4. Only healthy targets: pods that fail their readiness probe are taken out of the list until they recover (question 402).

The address is virtual. No machine owns 10.96.14.20. Each node's kube-proxy, or network plugin, catches traffic to it and redirects it to a real pod (question 320).

yaml
apiVersion: v1
kind: Service
metadata:
  name: checkout-api
  namespace: shop-prod
spec:
  selector: { app.kubernetes.io/name: checkout-api }
  ports:
    - name: http
      port: 80            # what clients connect to
      targetPort: http    # the pods' named container port, 8080

359. What are ClusterIP, NodePort, and LoadBalancer Services, and when would you use each? ​

The short answer

A ClusterIP Service, the default, has only an internal address, reachable from inside the cluster. A NodePort Service also opens the same port, between 30000 and 32767, on every node, so traffic to any node's address on that port reaches the Service. A LoadBalancer Service also asks the cloud to create an external load balancer in front of it, with its own public or private address. Use ClusterIP for internal traffic, LoadBalancer or a gateway for traffic from outside, and NodePort mostly as a building block.

LoadBalancer: adds a cloud load balancer, address 20.61.8.4NodePort: adds port 31080 on every nodeClusterIP: internal address 10.96.14.20, port 80podpodpod
Each type wraps the one inside it. Every Service has a ClusterIP. A NodePort Service adds a port on each node. A LoadBalancer Service adds a cloud load balancer that sends traffic to those node ports, or, on some clouds, straight to the pods.
TypeReachable fromAddressUse for
ClusterIPInside the clusterVirtual internal IPService-to-service calls, the default
NodePortAnywhere that can reach a nodeNode IP plus a port in 30000–32767Simple set-ups, a base for load balancers
LoadBalancerOutside, through the cloudA cloud load balancer's IPExposing one service directly
yaml
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local            # (1)
  selector: { app.kubernetes.io/name: web }
  ports: [{ port: 443, targetPort: 8443 }]
  1. externalTrafficPolicy: with the default, Cluster, a node that receives traffic may forward it to a pod on another node, which adds a hop and hides the client's real IP address. With Local, traffic only goes to pods on the receiving node, so the client's address is kept, but nodes with no matching pod must be taken out of the load balancer, which the cloud does using health checks.

What to use in practice:

  1. Internal services: always ClusterIP. The book's shop learned this the hard way, when giving the back end its own LoadBalancer put it on the internet, bypassing the gateway's protections (see 14.4.3).
  2. Outside traffic: usually one shared entry point, an ingress controller or a Gateway API gateway, which itself is exposed by one LoadBalancer Service, and routes by host name and path to many internal ClusterIP Services (question 371).
  3. A LoadBalancer per service only for traffic that a gateway cannot handle, such as raw TCP or UDP protocols.

Private load balancers. Clouds can create load balancers with only an internal address, using an annotation, for example service.beta.kubernetes.io/azure-load-balancer-internal: "true" on AKS, for services reached from your private network but not from the internet.

360. How does a Service select the Pods that receive traffic? ​

The short answer

Through its label selector. The Service lists labels, such as app.kubernetes.io/name: checkout-api, and every pod in the same namespace that carries all of those labels and is ready becomes a backend. A controller keeps that list, the Service's endpoints, up to date as pods appear, become ready, stop being ready or disappear. The Service's targetPort says which port on those pods receives the traffic.

yaml
apiVersion: v1
kind: Service
metadata:
  name: checkout-api
spec:
  selector:
    app.kubernetes.io/name: checkout-api         # (1)
  ports:
    - name: http
      port: 80                                   # (2)
      targetPort: http                           # (3)
---
# in the Deployment's pod template:
containers:
  - name: api
    ports:
      - name: http
        containerPort: 8080                      # (4)
  1. Any ready pod in this namespace with this label becomes a backend.
  2. The port clients use: checkout-api:80.
  3. The port on the pod that receives the traffic, given here by name.
  4. The container's port called http is 8080. Using a name means you can change the container port without touching the Service.

Ready is what counts. A matching pod is a backend only while its readiness probe passes. A pod that is starting up, overloaded or shutting down is left out (questions 402 and 488).

Same namespace only. A selector never picks pods from another namespace.

A Service without a selector. If you leave out spec.selector, no endpoints are created automatically. You can then create EndpointSlices by hand, pointing at addresses outside the cluster, such as a database on a virtual machine. Clients use a normal Service name, and you can later move the database into the cluster without changing them.

361. What are Service endpoints or EndpointSlices, and how would you check whether a Service has ready backends? ​

The short answer

EndpointSlices are the objects that list the actual pod addresses and ports behind a Service, along with whether each is ready, serving or terminating. A controller keeps them up to date, and kube-proxy, network plugins, ingress controllers and gateways read them to route traffic. They replaced the older single Endpoints object, which did not scale to very large Services. To check a Service's backends, list its EndpointSlices, or look at the Endpoints line of kubectl describe service.

bash
kubectl get endpointslices -n shop-prod -l kubernetes.io/service-name=checkout-api
kubectl describe service checkout-api -n shop-prod
NAME                 ADDRESSTYPE   PORTS   ENDPOINTS                         AGE
checkout-api-x7k2p   IPv4          8080    10.244.1.7,10.244.2.9,10.244.3.4   12d
Name:              checkout-api
Selector:          app.kubernetes.io/name=checkout-api
Type:              ClusterIP
IP:                10.96.14.20
Port:              http  80/TCP
TargetPort:        http/TCP
Endpoints:         10.244.1.7:8080,10.244.2.9:8080,10.244.3.4:8080

What each EndpointSlice records for every address:

ConditionMeaning
readyHealthy and should receive traffic
servingPassing readiness, even if it is shutting down
terminatingBeing deleted

Those conditions are how traffic is drained from pods during shutdown (question 488).

Why slices? The original Endpoints object held every address of a Service in one object. For a Service with 5,000 pods, one pod changing meant sending the whole huge object to every node. EndpointSlices split the list into pieces of up to 100 addresses each, so a change only updates one small slice.

An empty list is the key clue. If Endpoints: shows <none>, nothing will reach any pod, however healthy the Service itself looks. Question 362 explains the usual causes.

362. Why might a Service exist but have no working endpoints? ​

The short answer

Usually because the selector does not match the pods' labels, or because the matching pods are not ready. Other causes are pods living in a different namespace from the Service, no pods running at all, or, when endpoints exist but connections fail, a wrong targetPort or a network policy blocking the traffic. Check the selector against the pod labels, then the pods' readiness, then the ports, then the policies.

Work through it in this order:

  1. Do the labels match?
bash
kubectl get svc checkout-api -n shop-prod -o jsonpath='{.spec.selector}{"\n"}'
kubectl get pods -n shop-prod --show-labels | grep checkout

A typo, such as app: checkout against app: checkout-api, or a label key that changed in a refactor, gives an empty endpoint list (question 347).

  1. Are the pods ready?
bash
kubectl get pods -n shop-prod -l app.kubernetes.io/name=checkout-api

0/1 in the READY column means the readiness probe is failing, so the pod is left out (question 409).

  1. Same namespace? A Service only selects pods in its own namespace.
  2. Any pods at all? The Deployment may be scaled to zero, or its pods may all be crash-looping.
  3. Endpoints present, but connections fail?
    • Wrong targetPort. The Service sends traffic to port 80 on the pods, but the app listens on 8080. Endpoints look fine, and every connection is refused.
    • A named port that does not exist on the container.
    • A network policy that blocks traffic from the caller (question 367).
    • The app listening on 127.0.0.1 inside the pod instead of all addresses (question 119).

A quick test from inside the cluster:

bash
kubectl run tmp --rm -it --image=busybox:1.36 -n shop-prod -- wget -qO- http://checkout-api/healthz

If this fails while a direct request to one pod's IP works, the problem is in the Service's selector or ports. If both fail, look at the pods themselves.

363. How does service discovery work between Pods in the same cluster? ​

The short answer

Through the cluster's DNS server, usually CoreDNS. Every pod's /etc/resolv.conf points at it, and it answers names of Services with their ClusterIP, such as checkout-api.shop-prod.svc.cluster.local resolving to 10.96.14.20. Search domains let pods use short names, such as just checkout-api within the same namespace. Kubernetes also injects environment variables for Services that existed when a pod started, but DNS is the method to rely on.

What a pod's DNS settings look like:

bash
kubectl exec -n shop-prod web-6f5c-abcde -- cat /etc/resolv.conf
search shop-prod.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
  • nameserver 10.96.0.10 is the ClusterIP of the kube-dns Service, behind which CoreDNS pods run.
  • search lists suffixes to try for short names. checkout-api is tried as checkout-api.shop-prod.svc.cluster.local first, which is the same namespace's Service.
  • ndots:5 means "if a name has fewer than five dots, try the search suffixes first". It is why short names work, and also why external names can cause extra lookups (question 375).

What CoreDNS answers:

NameAnswer
checkout-api.shop-prod.svc.cluster.localThe Service's ClusterIP
_http._tcp.checkout-api.shop-prod.svc.cluster.localAn SRV record with the named port's number
orders-db-0.orders-db.shop-prod.svc.cluster.localOne StatefulSet pod's IP, through a headless Service (question 489)
Anything else, such as api.stripe.comForwarded to upstream DNS servers

How CoreDNS knows. It watches Services and EndpointSlices through the API server, so a new Service is resolvable within seconds.

The environment variables, such as CHECKOUT_API_SERVICE_HOST=10.96.14.20, are an older mechanism. They are only set for Services that already existed when the pod started, so they go stale. Use DNS names.

364. How would one application call another application using its Service DNS name? ​

The short answer

By using the Service's name as the host in its URL, with the Service's port. From the same namespace, the short name is enough: http://checkout-api. From another namespace, add the namespace: http://checkout-api.shop-prod. The fully qualified name, checkout-api.shop-prod.svc.cluster.local, always works. Keep the address in configuration, not in code, so each environment can point somewhere different.

Caller is inName to useResolves because
shop-prod, same namespacecheckout-apiThe first search suffix adds .shop-prod.svc.cluster.local
shop-qa, another namespacecheckout-api.shop-prodThe search suffix adds .svc.cluster.local
Anywherecheckout-api.shop-prod.svc.cluster.localIt is the full name

In the shop's web app, written in TypeScript:

ts
const CHECKOUT_URL = process.env.CHECKOUT_URL ?? 'http://checkout-api'       // (1)

export async function placeOrder(order: OrderRequest): Promise<OrderResult> {
  const res = await fetch(`${CHECKOUT_URL}/orders`, {                       // (2)
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify(order),
    signal: AbortSignal.timeout(3_000),                                     // (3)
  })
  if (!res.ok) throw new Error(`checkout-api replied ${res.status}`)
  return (await res.json()) as OrderResult
}
  1. The address comes from configuration, defaulting to the same-namespace Service name. Port 80 is implied for http://, which matches the Service's port: 80.
  2. The request goes to the Service. DNS turns checkout-api into the ClusterIP, and the node's rules send it to one ready pod.
  3. A timeout, so a slow or missing dependency fails fast instead of hanging the request.

Common mistakes:

  • Using the container port instead of the Service port. Clients use the Service's port. The targetPort is internal to the Service.
  • Using a pod's IP or name. Both change. Always the Service.
  • Forgetting the namespace when calling across namespaces, which fails with a "could not resolve host" error.

365. How do Pod networking and node networking differ conceptually? ​

The short answer

Node networking is the ordinary network the machines sit on, such as the cloud's virtual network, with one address per node. Pod networking gives every pod its own address, from a separate range, and Kubernetes requires that every pod can reach every other pod's address directly, across nodes, without address translation. Services add a third, purely virtual range of addresses that exist only as rules on the nodes.

node network: 10.0.0.0/16, the cloud virtual networknode-1 10.0.1.4, node-2 10.0.1.5: real interfaces on real or virtual machinespod network: 10.244.0.0/16node-1 hands out 10.244.1.x, node-2 hands out 10.244.2.x; every pod reaches every pod directlyService network: 10.96.0.0/12virtual addresses such as 10.96.14.20; no interface owns them, only rules on each node
Three kinds of address. Node addresses belong to machines, pod addresses belong to pods, and Service addresses belong to nobody: they are redirected to pod addresses by every node's rules.

The Kubernetes network model sets three rules that every network plugin must provide:

  1. Every pod gets its own unique IP address in the cluster.
  2. Every pod can reach every other pod's address directly, on any node, without address translation.
  3. Agents on a node, such as the kubelet, can reach every pod on that node.

How plugins make pod addresses reachable across nodes (question 366):

  • Overlay: pod traffic between nodes is wrapped inside packets addressed node to node, then unwrapped. The pod range is invisible to the underlying network.
  • Routed: nodes tell the network which pod range lives behind them, so packets are routed normally.
  • Cloud-native addressing: pods get addresses straight from the cloud network, like the nodes. Simple to reason about, but it uses up network addresses quickly (question 614).

Why the ranges must not overlap. If the pod range, the Service range and your company's own networks share addresses, traffic meant for an office server can end up inside the cluster, or the reverse. Choose the ranges when the cluster is created, because changing them later usually means building a new cluster.

366. What is a CNI plugin, and what responsibility does it provide? ​

The short answer

CNI, the Container Network Interface, is a standard for how a container runtime asks a network plugin to connect a pod. When a pod's sandbox is created, the runtime calls the plugin, which creates the pod's network interface, gives it an IP address, sets up routes, and connects it so it can reach every other pod. When the pod is deleted, the plugin cleans up. Many plugins also enforce network policies, encrypt traffic and record network activity. Common ones are Calico, Cilium, Flannel, Azure CNI and AWS VPC CNI.

What happens when a pod starts:

  1. The runtime creates the pod's network namespace (question 334).
  2. It calls the CNI plugin with "add this pod".
  3. The plugin creates a veth pair, a virtual cable, with one end in the pod as eth0 and the other on the node (question 111).
  4. Its IPAM part, IP address management, hands out a free address from the node's pod range, or from the cloud network.
  5. It sets routes, so the pod's traffic leaves through its cable and other pods' traffic finds its way in.
  6. It reports the address back, and Kubernetes records it as the pod's IP.

How plugins differ:

PluginHow pods reach other nodesNetwork policiesNotes
FlannelOverlayNo, on its ownSimple
CalicoRouted, or overlayYesWidely used, strong policy support
CiliumeBPF, routed or overlayYes, including by application protocolCan replace kube-proxy; detailed visibility
Azure CNICloud addresses, or an overlay modeThrough Azure, Calico or CiliumAKS's main options
AWS VPC CNICloud addresses from the VPCWith an add-on or CiliumEKS default

The choice matters for: how many pods fit in your network address space, whether network policies are enforced at all, how fast traffic is, and how much you can see when debugging. It is very hard to change on a running cluster, so it is decided at creation time (question 614).

367. How do NetworkPolicies control which Pods can communicate, and what must be true for a policy to take effect? ​

The short answer

A NetworkPolicy selects some pods and lists the traffic allowed to reach them, their ingress, or leave them, their egress, by pod labels, namespace labels, IP ranges and ports. Pods that no policy selects allow all traffic. As soon as a policy selects a pod for a direction, only the traffic it lists is allowed in that direction. For any of this to work, the cluster's network plugin must enforce network policies. If it does not, the policies are stored and silently ignored.

The usual starting point: deny everything, then allow what is needed.

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop-prod
spec:
  podSelector: {}                           # (1)
  policyTypes: [Ingress, Egress]            # (2)
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-api-allow
  namespace: shop-prod
spec:
  podSelector:
    matchLabels: { app.kubernetes.io/name: checkout-api }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app.kubernetes.io/name: web }        # (3)
      ports: [{ port: 8080, protocol: TCP }]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }                  # (4)
      ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
    - to:
        - ipBlock: { cidr: 10.20.5.0/24 }                       # (5)
      ports: [{ port: 5432, protocol: TCP }]
  1. An empty selector selects every pod in the namespace.
  2. With no rules listed, every pod here may now send and receive nothing.
  3. checkout-api may receive traffic on port 8080, only from web pods in this namespace.
  4. It may reach the cluster's DNS. Forgetting this is the most common mistake: with egress denied, every name lookup fails, and the app reports confusing connection errors.
  5. It may reach the managed PostgreSQL server's subnet on port 5432.

Rules worth knowing:

  1. Policies only add permissions. There is no "deny" rule. Several policies selecting the same pod combine, and traffic is allowed if any of them allows it.
  2. Both ends matter. For a connection from web to checkout-api to work, web's egress must allow it, if web has egress policies, and checkout-api's ingress must allow it.
  3. Replies are automatic. Return traffic of an allowed connection is allowed without a separate rule.
  4. The plugin must enforce them. Calico, Cilium and Azure's network policy options do. Flannel alone does not. Check with a quick test: apply a deny-all and confirm traffic really stops.

489. What are headless and ExternalName Services, and when would you use them? ​

The short answer

A headless Service, created with clusterIP: None, has no virtual address and no load balancing. Instead, its DNS name returns the IP addresses of the pods themselves, and with a StatefulSet each pod also gets its own stable DNS name. Use it when clients must reach specific pods, such as a database's primary. An ExternalName Service has no pods at all: its DNS name simply points, as an alias, at a name outside the cluster, such as a managed database's address.

Headless:

yaml
apiVersion: v1
kind: Service
metadata:
  name: orders-db
spec:
  clusterIP: None
  selector: { app: orders-db }
  ports: [{ port: 5432 }]
bash
kubectl run tmp --rm -it --image=busybox:1.36 -- nslookup orders-db
Name:    orders-db.shop-prod.svc.cluster.local
Address: 10.244.1.21
Address: 10.244.2.17
Address: 10.244.3.30

Three pod addresses, not one virtual address. With the StatefulSet from question 340, each pod also has its own name:

text
orders-db-0.orders-db.shop-prod.svc.cluster.local  →  10.244.1.21

Use headless when:

  1. Each pod has a role. "Write to orders-db-0, read from the others".
  2. Clients balance by themselves. For example, gRPC clients that keep long-lived connections and want to know every backend.
  3. Peers must find each other, such as members of a database cluster discovering one another.

ExternalName:

yaml
apiVersion: v1
kind: Service
metadata:
  name: orders-db
  namespace: shop-prod
spec:
  type: ExternalName
  externalName: shop-orders.postgres.database.azure.com

Pods connect to orders-db, and DNS answers with an alias, a CNAME record, for the Azure address. No traffic passes through Kubernetes at all.

ExternalName caveats:

  • No ports, no load balancing, no selection: it is only a DNS alias.
  • TLS names. The client connects to orders-db, but the server's certificate names shop-orders.postgres.database.azure.com. Clients that check the certificate against the name they dialled will fail unless configured with the real name.
  • Not an IP address. externalName must be a DNS name.

490. What is kubectl port-forward, and how is it different from a Service? ​

The short answer

kubectl port-forward opens a port on your own machine and tunnels connections to it, through the API server and the kubelet, to one pod in the cluster. It is a debugging and admin tool: for example, to open a database or an internal dashboard from your laptop without exposing it. A Service is the cluster's real, permanent way of routing traffic to pods, with a stable address and load balancing. Port-forward is temporary, goes to a single pod, and depends on your session staying open.

bash
kubectl port-forward -n shop-prod svc/checkout-api 8080:80      # (1)
kubectl port-forward -n shop-prod pod/orders-db-0 5432:5432     # (2)
curl http://localhost:8080/healthz                              # (3)
  1. Forwards your local port 8080 to the Service's port 80. Behind the scenes, kubectl picks one pod behind the Service and tunnels to it for the whole session.
  2. Forwards straight to one particular pod, here the database's first instance.
  3. Your laptop now reaches the pod as if it were local.
kubectl port-forwardService
PurposeDebugging, admin accessProduction routing
LastsWhile the command runsUntil deleted
Goes toOne podAny ready pod, balanced
PathYour machine, API server, kubelet, podNode rules straight to pods
Who can use itAnyone with the pods/portforward permissionAnything on the network allowed by policies

Security points:

  • It needs the create permission on the pods/portforward subresource. Anyone who has it can reach any port of pods in that namespace, bypassing gateways and some network policies. Grant it carefully, especially in production (question 416).
  • It is fine for an engineer to open a database tunnel briefly, but never as a way to serve real traffic.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet