Appearance
Managed Kubernetes
611. What does a managed Kubernetes service provide, and what is still your job?
The short answer
In AKS, EKS and GKE, the provider runs the control plane for you: the API server, etcd, the scheduler and controllers, across zones, patched, backed up and monitored, often with an uptime promise. You still own the worker nodes, grouped into node pools, their sizes and upgrades, the cluster's networking choices, add-ons, security settings, and everything you deploy. Some modes, such as GKE Autopilot, EKS Auto Mode and AKS Automatic, also take over the nodes.
| Provider | You | |
|---|---|---|
| Control plane machines, etcd | Yes | |
| Control plane upgrades | Runs them | Choose when, and check compatibility |
| Worker node images and patches | Provides new images | Roll them out, or turn on auto-upgrade |
| Node sizes and counts | Yes, unless automatic nodes are used | |
| Networking model | Offers choices | Choose at creation; hard to change later |
| Identity integration | Provides it | Configure RBAC and workload identity |
| Add-ons | Some managed add-ons | Choose, configure and upgrade them |
| Workloads and their security | Yes |
Cost of the control plane. Each of the three charges a small hourly fee per cluster for a tier with an uptime SLA, typically about ten cents an hour. Long-term or extended support of old versions costs more. The worker nodes are normal virtual machines, billed as such.
Same Kubernetes, different extras. Your manifests work on all three. The differences are in the integration: networking, identity, load balancers, storage drivers, and the upgrade tools.
612. What are node pools, and how should you design them?
The short answer
A node pool is a group of identical worker nodes within one cluster: the same machine size, operating system and settings. Clusters usually have several: a small system pool for Kubernetes' own components, general pools for most apps, and special pools for GPUs, large memory, Windows, or cheap spot capacity. Taints and labels decide which pods may run where. Pools let you scale, upgrade and size each kind of workload separately.
A typical cluster for the shop:
| Pool | Size | Nodes | For |
|---|---|---|---|
system | Small, 3 nodes across zones | 3 | CoreDNS, metrics, the cloud's agents |
general | General purpose | 3 to 20, autoscaled | Web and API services |
spot | General purpose, spot | 0 to 30, autoscaled | Batch jobs, CI runners (question 561) |
gpu | GPU machines | 0 to 4 | Image recognition jobs |
Steering pods to pools:
- Taint the special pool, so ordinary pods stay off it. For example, the spot pool carries a taint that only interruption-tolerant pods tolerate.
- Label the pool, such as
pool=gpu. - Pods that need it get a toleration and a node selector or node affinity for that label (question 394).
Design choices that matter:
- Zones. Spread each pool across zones, so one zone failing does not remove a whole pool (question 396).
- Node size. Bigger nodes waste less on per-node overhead, such as system pods and reserved memory, and fit big pods. Smaller nodes mean losing one hurts less, and scaling steps are finer. Middle-sized general nodes are a common choice.
- Max pods per node is often limited by the networking model, which can make big nodes less useful than expected (question 614).
- Separate system pools. In AKS, the system pool runs critical components. Keeping apps off it stops a busy app from starving cluster DNS.
Upgrades per pool. Each pool is upgraded on its own, which lets you upgrade a small pool first as a canary (question 618).
613. What are serverless or automatic nodes in managed Kubernetes?
The short answer
They remove node management from you. In GKE Autopilot, EKS Auto Mode and AKS Automatic, the provider creates, sizes, patches and removes nodes based on your pods' requests, and you pay roughly for what the pods use. Virtual nodes and EKS on Fargate go further, running each pod on serverless capacity with no node you can see. You give up some control, such as privileged pods, custom node settings and some DaemonSets, in exchange for far less operations work.
| Option | How it works | Trade-off |
|---|---|---|
| GKE Autopilot | Google manages nodes; billed per pod's requests | Restrictions on privileged and host access |
| EKS Auto Mode | AWS manages nodes, using Karpenter-style provisioning | Fewer node-level customisations |
| AKS Automatic | Azure manages nodes and many add-ons with safe defaults | Opinionated settings |
| EKS on Fargate | Each pod runs in its own small VM | No DaemonSets, no GPUs, slower starts |
| AKS virtual nodes | Pods run in Azure Container Instances | Limited features, for bursts |
Node auto-provisioning, which is what Karpenter does, sits in between. You keep normal nodes, but instead of fixed pools, a controller looks at pending pods and starts the cheapest machine that fits them, then removes empty nodes (question 399).
Why teams choose these:
- No node upgrades or patching to schedule.
- Less wasted capacity, since nodes fit the pods.
- Security defaults are already strict.
Why some teams do not:
- Workloads that need privileged access, special kernels or custom agents on nodes.
- Cost at large, steady scale, where well-packed normal nodes with commitment discounts are cheaper.
- Fewer knobs to turn when tuning performance.
614. How does pod networking work in managed Kubernetes, and how do clusters run out of IP addresses?
The short answer
Clusters give pods addresses in one of two ways. In routable or "VNet" mode, such as AWS VPC CNI and Azure CNI with a pod subnet, each pod takes a real address from your cloud network. That is simple to reach and to see, but it uses addresses quickly, so subnets run out. In overlay mode, such as Azure CNI Overlay, pods get addresses from a separate private range that only exists inside the cluster, which saves network addresses but means outside systems reach pods only through services. Choose at creation, because changing later is hard.
The arithmetic of running out. In routable mode, with a /24 subnet, 251 usable addresses, and 30 pods per node: each node uses 31 addresses, one for itself and 30 for pods. After 8 nodes, the subnet is full. The ninth node, or a surge node during an upgrade, cannot start, and pods stay Pending with errors such as "failed to assign an IP address" (question 408).
| Routable | Overlay | |
|---|---|---|
| Pod addresses from | Your network's subnets | A private range inside the cluster |
| Network addresses used | Nodes and every pod | Only nodes |
| Reach pods directly from outside the cluster | Yes | No, only through services and load balancers |
| See pod addresses in network logs | Yes | Outbound traffic shows the node's address |
| Examples | AWS VPC CNI, Azure CNI with pod subnet, GKE VPC-native | Azure CNI Overlay, Cilium overlay modes |
Ways out of address exhaustion:
- Plan big subnets from the start, such as a
/20or larger for pods (question 531). - Use overlay mode when pods do not need to be directly reachable.
- On EKS, use prefix delegation, which gives each node blocks of addresses instead of one at a time, and custom networking with a secondary range, such as
100.64.0.0/10, a range set aside for this kind of internal use. - Add another subnet for a new node pool, where the platform allows it.
- Lower max pods per node, if address use, not node count, is the limit.
Also decided at creation: the service address range, which must not overlap with any network you connect to, and the network policy engine, such as Cilium or Calico (question 367).
615. How does workload identity let pods use cloud services without stored secrets?
The short answer
Each Kubernetes service account is linked to a cloud identity. The cluster signs a short-lived token for the pod's service account, and the cloud, which trusts the cluster's token issuer, swaps that token for its own short-lived cloud credentials for the linked identity. The cloud SDK in the pod does this automatically. Each workload gets exactly its own permissions, there are no keys in Secrets, and nothing long-lived can leak. AKS calls it Microsoft Entra Workload ID, EKS offers IRSA and EKS Pod Identity, and GKE has Workload Identity Federation.
Setting it up on AKS, for the checkout service:
- Turn on the cluster's OIDC issuer and workload identity.
- Create a user-assigned managed identity and give it a role, such as "Key Vault Secrets User" on one vault (question 549).
- Add a federated credential to that identity: "trust tokens from this cluster's issuer, for service account
checkoutin namespaceshop". - Annotate the service account and label the pod:
yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: checkout
namespace: shop
annotations:
azure.workload.identity/client-id: 6c1e... # (1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout
namespace: shop
spec:
selector:
matchLabels: { app: checkout }
template:
metadata:
labels:
app: checkout
azure.workload.identity/use: "true" # (2)
spec:
serviceAccountName: checkout # (3)
containers:
- name: app
image: registry.shop.example/checkout:2.4.1- Which cloud identity this service account maps to.
- Opt this pod in. A webhook adds the token file and environment variables the SDK needs.
- Use the linked service account. Code using
DefaultAzureCredentialthen finds the identity on its own (question 550).
On EKS, EKS Pod Identity links a service account to an IAM role through an association, with an agent on each node. The older IRSA, IAM roles for service accounts, uses an annotation with the role's ARN. On GKE, you bind the Kubernetes service account to an IAM role or a Google service account.
Why not use the node's identity? Every pod on the node would share it. A compromised web pod could then read the payment service's secrets. Workload identity gives each service only its own access (question 417).
616. How do cloud load balancers and gateway controllers work with managed Kubernetes?
The short answer
A Service of type LoadBalancer makes the cloud create a layer 4 load balancer with a public or private address that sends traffic to your nodes or pods. For HTTP routing, you install a controller that reads Ingress or Gateway API resources. It either configures a proxy running in the cluster, or creates a managed layer 7 load balancer, such as Application Gateway for Containers, an AWS Application Load Balancer, or Google's load balancer. Annotations or classes choose the details, such as internal versus internet-facing.
| Cloud | Layer 4 for LoadBalancer Services | Managed layer 7 controller | Gateway API support |
|---|---|---|---|
| AKS | Azure Load Balancer | Application Gateway for Containers | Yes |
| EKS | Network Load Balancer, through the AWS Load Balancer Controller | Application Load Balancer, through the same controller | Yes |
| GKE | Google network load balancer | GKE Gateway controller, Google application load balancer | Yes |
Internal or public. By default a LoadBalancer Service may get a public address. For internal services, use the cloud's annotation:
yaml
apiVersion: v1
kind: Service
metadata:
name: orders-internal
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
selector: { app: orders }
ports:
- port: 443
targetPort: 8443On EKS, the equivalent is service.beta.kubernetes.io/aws-load-balancer-scheme: internal with the AWS Load Balancer Controller.
In-cluster proxy or managed load balancer?
| Proxy in the cluster, such as Envoy Gateway | Managed layer 7 load balancer | |
|---|---|---|
| Runs | As pods you scale and upgrade | In the cloud, managed |
| WAF and TLS certificates | Configure yourself | Built-in integrations |
| Portability | Same everywhere | Cloud-specific |
| Cost | Node resources | Per-hour and per-use charges |
A note on NGINX Ingress. The community ingress-nginx project was retired in March 2026, so new clusters should use a maintained controller, preferably with the Gateway API (question 491).
Traffic straight to pods. Many cloud controllers can send traffic directly to pod addresses, skipping the extra hop through a node port. That works with routable pod networking, and gives better health checks and fewer hops (question 614).
617. How does a managed cluster pull images from a private cloud registry?
The short answer
The nodes pull images with their own identity, the kubelet's identity. So the simplest and safest way is to give that identity read access to the registry. In Azure, that is the AcrPull role, for example by attaching the registry to the cluster. In AWS, it is a read-only ECR policy on the nodes' IAM role. Then no image-pull secrets are needed in your namespaces. For registries in other accounts or outside the cloud, use cross-account permissions, or image-pull secrets refreshed automatically.
On AKS:
bash
az aks update --name aks-shop-prod --resource-group rg-shop --attach-acr crshopprodThis gives the cluster's kubelet identity the AcrPull role on the registry crshopprod. Every node can now pull crshopprod.azurecr.io/checkout:2.4.1 with no secrets in the cluster.
On EKS, the node IAM role gets a read-only ECR policy, such as AmazonEC2ContainerRegistryReadOnly, and the kubelet uses it automatically. For a registry in another AWS account, the repository policy in that account must allow your node role.
When pulls fail, check in order (question 160):
- The exact image name and tag, from
kubectl describe pod. - The node identity's permission on the registry.
- Network access: with a private endpoint, the cluster's network must resolve the registry name to the private address (question 538).
- Registry firewall rules, if public access is restricted.
Image-pull secrets are still needed for registries outside your cloud, such as a vendor's private registry. They are stored per namespace and referenced from the pod or its service account (question 420). Refresh them automatically if their token expires.
Make it faster: keep the registry in the same region as the cluster, and keep images small. Where offered, use image streaming features, such as GKE image streaming or AKS artifact streaming, which start containers before the whole image has downloaded.
618. How do you upgrade a managed Kubernetes cluster safely?
The short answer
Kubernetes releases a new minor version about three times a year, and each is supported for roughly a year, so upgrades are routine work. Read the release notes and find removed APIs your manifests still use. Upgrade a non-production cluster first. Then upgrade the control plane, one minor version at a time, and then the node pools, which are replaced node by node with surge nodes while PodDisruptionBudgets keep services up. Watch errors throughout, and for risky jumps, create a new node pool and move workloads over instead.
Why the order matters. Kubernetes allows nodes to run older versions than the control plane, within a limit, but never newer. So the control plane goes first, then the nodes catch up.
Before upgrading:
- Removed APIs. Each version removes some old API versions. A manifest still using a removed one fails to apply after the upgrade. Tools such as
kubentandplutoscan manifests and Helm releases for them. - Add-ons: ingress or gateway controllers, CNI, CSI drivers, service mesh, policy engines. Check each supports the new version.
- PodDisruptionBudgets for every important service, but none that allow zero disruptions, which would block draining forever (question 400).
Node pool upgrade, as it runs:
- The service adds surge nodes with the new version.
- It cordons an old node, so no new pods land there, then drains it, evicting pods with respect to PodDisruptionBudgets and grace periods (question 356).
- Pods start on the new nodes.
- The old node is deleted, and the next one follows.
Making it safer:
- Max surge settings, such as 33%, trade speed against extra capacity.
- Blue-green node pools: create a new pool on the new version, move workloads, then delete the old pool. Rolling back is moving back.
- Auto-upgrade channels apply patch versions automatically and minor versions on a schedule, inside maintenance windows you choose.
- Stay current. Falling behind means jumping several versions in a hurry, or paying for extended support.
619. What is a private cluster, and how do you secure access to the Kubernetes API?
The short answer
A private cluster's API server has only a private address in your network, so it cannot be reached from the internet at all. Administrators and pipelines reach it from inside the network, through a VPN, a bastion or jump host, or self-hosted runners in the network. If a public API endpoint is kept, restrict it to known address ranges. In all cases, use cloud identity for sign-in, short-lived credentials, Kubernetes RBAC, and turn off local admin accounts.
| Option | API reachable from | Fits |
|---|---|---|
| Public endpoint, open | The whole internet, with sign-in | Avoid for production |
| Public endpoint, authorised ranges | Listed office and runner addresses | Small teams, simple set-ups |
| Private endpoint | Only inside your network | Production, regulated environments |
What changes with a private cluster:
- Pipelines need runners inside the network, or a managed way in, such as AKS
az aks command invoke, which runs a command through Azure's control plane without network access to the API. - People connect through a VPN, a bastion host, or a cloud shell connected to the network (question 543).
- DNS must resolve the API server's name to its private address from every place that connects, similar to private endpoints (question 538).
- GitOps helps a lot: Argo CD or Flux run inside the cluster and pull from Git, so nothing outside needs to reach the API (question 507).
Sign-in and permissions:
- Cloud identity for sign-in, Entra ID on AKS, IAM on EKS, Google identities on GKE, mapped to Kubernetes RBAC groups (question 415).
- Disable local accounts, such as the static admin credentials AKS can generate, so every access is a named, audited person.
- Just-in-time admin, through privileged access management, for cluster-admin rights.
- Audit logs of the API server sent to your central logging.
620. How do you control and reduce the cost of a managed Kubernetes cluster?
The short answer
Most Kubernetes cost is worker nodes, and most waste comes from pods requesting far more CPU and memory than they use, which forces more nodes. Measure real usage per workload, right-size requests, and let the cluster autoscaler or node auto-provisioning remove empty nodes. Use spot node pools for interruption-tolerant work, commitment discounts for the steady base, scale non-production clusters down at night, and show each team the cost of its namespaces with tools such as OpenCost or the cloud's cost analysis.
Where the money goes:
| Item | Typical share | Lever |
|---|---|---|
| Worker nodes | Most of it | Right-size requests, autoscale, spot, commitments |
| Load balancers and public IPs | Small, adds up | Share a gateway instead of one load balancer per service |
| Storage | Medium | Delete unused volumes, right-size disks |
| Logs and metrics | Often surprising | Filter, lower retention, sample (question 633) |
| Data transfer | Varies | Same-zone routing, private endpoints (question 590) |
| Control plane fee | Small per cluster | Fewer, shared clusters |
Requests are what you pay for. The scheduler places pods by their requests, not their real use. If every pod asks for 1 CPU but uses 0.1, nodes fill up on paper while sitting 90% idle (question 389).
Step by step:
- Measure requests against real usage per workload, over a few weeks.
- Right-size requests closer to real use, plus headroom. The Vertical Pod Autoscaler in recommendation mode suggests values (question 495).
- Autoscale nodes, so freed capacity becomes fewer nodes (question 399).
- Bin-pack: use node sizes and autoscaler settings that fill nodes well, and let the autoscaler consolidate half-empty ones.
- Spot pools for batch and CI (question 612).
- Commit to the steady base with reservations or savings plans (question 592).
- Turn off development clusters, or scale their pools to zero, outside working hours.
Show cost per team. OpenCost splits node costs across namespaces and labels by requests and usage. When each team sees what its namespace costs, requests come down without anyone policing them (question 591).
621. When should you choose a simpler container service instead of Kubernetes?
The short answer
Choose Azure Container Apps, AWS ECS with Fargate, or Google Cloud Run when you have a handful of services, a small team, mostly HTTP or queue-driven workloads, and no need for Kubernetes-specific tools. You get rolling deployments, autoscaling, including to zero, HTTPS, secrets and identity, without a cluster to upgrade, secure and pay for. Choose Kubernetes when you have many services and a platform team, need its extension ecosystem, need portability across clouds, or have special workloads it handles better.
| Question | Leans to a container service | Leans to Kubernetes |
|---|---|---|
| Number of services | A few to a few dozen | Dozens to hundreds |
| Platform team | None | Yes |
| Need operators, custom controllers, service mesh | No | Yes |
| Portability across clouds | Not important | Important |
| Special workloads: GPUs, stateful systems, custom networking | Rare | Common |
| Scale to zero | Wanted | Possible, with extras |
| Existing Kubernetes skills | Little | Strong |
What you give up with a container service: fewer controls, provider-specific configuration, and less choice of add-ons. What you gain: no cluster upgrades, no node patching, fewer moving parts, and often a lower bill for small and spiky workloads.
Not a one-way door. A well-built container image runs on either. Many teams start on a container service and move to Kubernetes when the number of services, or a specific need, makes the platform worth running (questions 309 and 559).
622. How do you run multiple Kubernetes clusters, and when do you need more than one?
The short answer
You need more than one cluster for separate environments, for different regions, for strong isolation between teams or customers, for compliance boundaries, and to limit the blast radius of a bad change or upgrade. Running many clusters well means building them all from the same code, deploying to them with GitOps from one repository, managing policies and add-ons centrally, and putting a global load balancer in front of clusters in different regions. Fleet tools help apply changes across many clusters in waves.
Reasons for several clusters:
| Reason | Example |
|---|---|
| Environments | dev, staging and prod never share a cluster |
| Regions | One cluster per region, for latency and disaster recovery (question 589) |
| Isolation | A customer's data must run on its own cluster |
| Blast radius | A broken upgrade or bad admission policy hits one cluster, not all |
| Scale limits | Very large estates split before one cluster becomes unmanageable |
Running them well:
- Same code for every cluster: infrastructure as code, with modules and per-cluster settings (question 602).
- GitOps: each cluster's Argo CD or Flux pulls from Git, with per-cluster overlays, so adding a cluster is adding a folder (question 510).
- Upgrade in waves: development clusters first, then one production cluster, then the rest.
- Central policy and add-ons: the same admission policies, monitoring agents and gateway controllers everywhere.
- Global traffic: a global load balancer or DNS sends users to healthy regional clusters.
- Fleet management: tools such as Azure Kubernetes Fleet Manager, Google Fleet, or Argo CD ApplicationSets apply changes and upgrades across many clusters.
Do not over-split. Every cluster costs a control plane fee, upgrades, monitoring and attention. Namespaces with RBAC, quotas and network policies give enough isolation for most teams inside one cluster (question 325).