Appearance
Helm, GitOps and Service Mesh
503. What is Helm, and what problems does it solve?
The short answer
Helm is the package manager for Kubernetes. A Helm chart is a package: a set of Kubernetes manifests written as templates, plus default settings called values. Installing a chart with your own values creates a release, a named, versioned installation that Helm can upgrade and roll back as a whole. It solves three problems: reusing one set of manifests across environments, installing complex third-party software in one command, and managing the history of what was installed.
The problems it solves:
- Repetition. Without templating, the dev, staging and production manifests are near-copies that drift apart. A chart writes them once and fills in the differences from values.
- Installing other people's software. A monitoring stack can be dozens of objects. With Helm it is one command:
bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace- Releases as a unit. Helm remembers every object a release created, so it can upgrade, roll back or uninstall all of them together (question 505).
Words to know:
| Term | Meaning |
|---|---|
| Chart | The package: templates plus default values |
| Values | The settings that fill in the templates |
| Release | One installation of a chart, with a name, in a namespace |
| Revision | One version of a release; each upgrade adds one |
| Repository | Where charts are published; today often an OCI registry, like images |
Charts in registries. Charts can be pushed to and installed from OCI registries, the same ones that store images:
bash
helm push checkout-api-1.4.2.tgz oci://myacr.azurecr.io/charts
helm install checkout-api oci://myacr.azurecr.io/charts/checkout-api --version 1.4.2A version note. Helm 4 was released in November 2025, after six years of Helm 3. Most commands are unchanged, but some flags were renamed, such as --atomic, which became --rollback-on-failure.
504. What are a Helm chart, release, values, and templates, and how does Helm render and install them?
The short answer
A chart is a folder with a Chart.yaml describing it, a values.yaml with defaults, and a templates folder of Kubernetes manifests containing placeholders. When you install, Helm merges the default values with yours, fills in every template using Go's template language, and sends the resulting plain manifests to the cluster. That installation, with its values and rendered manifests, is stored as a release. helm template shows the rendered output without installing anything.
A chart's layout:
text
checkout-api/
Chart.yaml name, version, appVersion, dependencies
values.yaml default settings
templates/
deployment.yaml
service.yaml
_helpers.tpl shared snippets, such as standard labels
charts/ charts this one depends onA template, with placeholders:
yaml
# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }} # (1)
labels: {{- include "checkout-api.labels" . | nindent 4 }} # (2)
spec:
replicas: {{ .Values.replicas }} # (3)
template:
spec:
containers:
- name: api
image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"
resources: {{- toYaml .Values.resources | nindent 12 }} # (4)
{{- if .Values.debug }} # (5)
env: [{ name: LOG_LEVEL, value: debug }]
{{- end }}- The release's name, chosen at install time.
- A shared label snippet from
_helpers.tpl, indented into place. - A value from the values files.
- A whole block of values turned into YAML.
- A condition: this part appears only when
debugis true.
Values, and how they combine, lowest priority first:
- The chart's own
values.yaml. - Each
-ffile, in the order given. - Each
--seton the command line.
bash
helm template checkout-api ./checkout-api -f values-prod.yaml --set image.digest=sha256:41c9... # render only
helm upgrade --install checkout-api ./checkout-api -n shop-prod -f values-prod.yaml --set image.digest=sha256:41c9...Always look at the rendered output in reviews or CI, with helm template or helm diff (question 349). Templates are easy to get subtly wrong, especially indentation, and the rendered YAML is what actually reaches the cluster.
505. How does Helm upgrade and roll back releases, and where does it store release history?
The short answer
helm upgrade renders the chart with the new values, compares it with the previous revision, applies the differences, and records a new revision. helm rollback re-applies an earlier revision's stored manifests as a new revision. Each revision is stored as a Secret in the release's namespace, containing the chart, the values and the rendered manifests, compressed and encoded. Helm keeps the last 10 revisions by default.
bash
helm upgrade --install checkout-api ./checkout-api -n shop-prod \
-f values-prod.yaml --wait --timeout 10m --rollback-on-failure # (1)
helm history checkout-api -n shop-prod # (2)
helm rollback checkout-api 41 -n shop-prod # (3)
kubectl get secrets -n shop-prod -l owner=helm,name=checkout-api # (4)- Install if missing, upgrade if present.
--waitwaits until the release's resources are ready.--rollback-on-failure, called--atomicbefore Helm 4, rolls back automatically if the upgrade fails or times out. - Lists revisions with their status, chart version and description.
- Goes back to revision 41. This creates a new revision, 44 for example, whose content equals 41.
- The stored history: Secrets named like
sh.helm.release.v1.checkout-api.v43.
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
41 Fri Oct 24 14:02:11 2026 superseded checkout-api-1.4.0 1.4.0 Upgrade complete
42 Mon Oct 27 09:30:45 2026 superseded checkout-api-1.4.1 1.4.1 Upgrade complete
43 Mon Oct 27 11:12:03 2026 deployed checkout-api-1.4.2 1.4.2 Upgrade completeThings worth knowing:
- A Helm rollback includes everything the chart manages, such as ConfigMaps and Services, not just the pod template, unlike
kubectl rollout undo(question 427). - History is in the cluster. Delete the namespace, and the history goes with it.
--history-maxsets how many revisions to keep. - Hooks. Templates with annotations such as
helm.sh/hook: pre-upgraderun as Jobs at set moments, which is a common place to run database migrations (question 357). - A failed upgrade can leave a release in a
failedorpending-upgradestate. With rollback-on-failure, Helm cleans up itself. Without it, you may need to roll back by hand before the next upgrade works. - With GitOps, Argo CD usually renders charts itself and applies the result, so Helm's own history is not used. Rollback is then a Git revert (question 512).
506. What is the difference between Helm and Kustomize, and when would you combine them?
The short answer
Helm uses templates: manifests with placeholders, filled in from values, plus packaging and release history. Kustomize uses no templates: you keep plain, valid YAML as a base, and each environment adds small overlay files that patch it, such as changing the replica count or the image. Kustomize is built into kubectl. Use Helm to package and distribute software, especially third-party charts, and Kustomize for your own apps' environment differences. You can combine them, for example by patching a chart's rendered output with Kustomize.
Kustomize, base and overlay:
text
deploy/
base/
deployment.yaml
service.yaml
kustomization.yaml
overlays/
prod/
kustomization.yaml
replicas.yamlyaml
# overlays/prod/kustomization.yaml
resources: [../../base]
namespace: shop-prod
images:
- name: checkout-api
newName: myacr.azurecr.io/shop/checkout-api
digest: sha256:41c9b0e8... # (1)
patches:
- path: replicas.yaml # (2)
configMapGenerator:
- name: checkout-api-config
literals: [LOG_LEVEL=info] # (3)- Replace the image everywhere it appears, pinned by digest.
- A small patch file that changes only
replicas. - Generates a ConfigMap with a hash in its name, so a configuration change rolls the pods (question 380).
bash
kubectl apply -k deploy/overlays/prod| Helm | Kustomize | |
|---|---|---|
| Approach | Templates and values | Plain YAML plus patches |
Built into kubectl | No | Yes, kubectl apply -k |
| Packaging and versioned releases | Yes | No |
| Release history and rollback | Yes | No, use Git |
| Readability | Templates can get complex | Base files are normal YAML |
| Best for | Distributing apps, third-party software | Your own apps across environments |
Combining them. Two common ways:
- Install a third-party chart, then patch it. Kustomize can render a Helm chart with its
helmChartsfield and apply patches on top, for a change the chart does not offer as a value. - Let the GitOps tool do both. Argo CD can render Helm charts and Kustomize overlays, so a repository can use whichever fits each application.
507. What is GitOps, and how is pull-based deployment different from push-based deployment?
The short answer
GitOps means a Git repository holds the complete desired state of what should run, and software inside the cluster continuously makes the cluster match it. To change production, you change Git, through a reviewed pull request. In push-based deployment, the CI pipeline connects to the cluster and applies changes, so CI needs cluster credentials. In pull-based deployment, an agent such as Argo CD or Flux runs inside the cluster, pulls from Git and applies changes itself, and also undoes any change made outside Git.
The four principles of GitOps, as described by the OpenGitOps project:
- Declarative: the desired state is described, not scripted (question 324).
- Versioned and immutable: it is stored in Git, with full history.
- Pulled automatically: agents fetch it themselves.
- Continuously reconciled: agents keep comparing and correcting, not just once per deploy.
| Push-based | Pull-based | |
|---|---|---|
| Who applies changes | The CI pipeline | An agent in the cluster |
| CI needs cluster credentials | Yes | No |
| Detects manual changes | No | Yes, and can undo them |
| Record of what should run | Pipeline logs | Git history |
| Rollback | Re-run an old pipeline | Revert a commit |
| Firewall direction | Into the cluster | Out from the cluster to Git |
Why pull-based is popular:
- Security: CI, which runs a lot of code, never holds production cluster credentials (question 271).
- Drift correction: a manual
kubectl editin production is detected and, with self-heal, reverted. - Audit: every production change is a reviewed commit.
- Private clusters: the cluster only needs outbound access to Git, never inbound access from CI.
The usual split of work: CI builds and checks the image, then opens a pull request, or commits directly, changing the digest in the configuration repository. The GitOps tool does the rest (question 510).
508. How does Argo CD work, including applications, sync, self-heal, pruning, and drift?
The short answer
Argo CD runs in the cluster and manages Application objects. Each Application points at a source, a Git repository, a path and a revision, and a destination, a cluster and namespace. Argo CD renders the source, compares it with what is live, and reports Synced or OutOfSync, plus the health of every resource. Syncing applies the differences. With automated sync, it does that by itself. Self-heal undoes manual changes, which is called drift, and prune deletes resources that were removed from Git.
yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: checkout-api-prod
namespace: argocd
spec:
project: shop # (1)
source:
repoURL: https://github.com/shop/deploy.git
targetRevision: main
path: apps/checkout-api/overlays/prod # (2)
destination:
server: https://kubernetes.default.svc
namespace: shop-prod
syncPolicy:
automated:
selfHeal: true # (3)
prune: true # (4)
syncOptions: [ServerSideApply=true]- An AppProject limits which repositories, clusters and namespaces this Application may use, which is how teams are kept apart.
- A Kustomize overlay, rendered by Argo CD. Helm charts and plain folders of YAML work too.
- Self-heal: if someone changes a managed resource by hand, Argo CD puts the Git version back.
- Prune: if a resource is deleted from Git, Argo CD deletes it from the cluster. Without this, removed resources would linger.
What Argo CD shows for each Application:
| Status | Meaning |
|---|---|
| Synced | Live state matches Git |
| OutOfSync | They differ, through a new commit or drift |
| Healthy | Resources are working: Deployments available, pods ready |
| Progressing | A rollout is under way |
| Degraded | Something is failing, such as crash-looping pods |
Ordering and extras:
- Sync waves, set with the annotation
argocd.argoproj.io/sync-wave, apply resources in order, for example the namespace and secrets before the Deployment. - Hooks, such as
PreSync, run Jobs before a sync, which suits database migrations (question 357). - ApplicationSets generate many Applications from one template, for example one per service, or one per cluster.
509. How do Argo CD and Flux compare?
The short answer
Both are mature GitOps tools that graduated in the Cloud Native Computing Foundation, and both continuously apply Git to Kubernetes. Argo CD is one application with a rich web interface, central management of many clusters, and its own Application model with single sign-on and role-based access. Flux is a set of separate controllers, one per job, driven mostly from the command line and Git, running inside each cluster, and it includes automation that updates image versions in Git. The choice usually comes down to whether a central visual interface or a composable, Git-only toolkit suits the team.
| Argo CD | Flux | |
|---|---|---|
| Shape | One application with a server and UI | A toolkit of controllers |
| Web interface | Rich, built in | Basic, or from third parties |
| Main objects | Application, ApplicationSet, AppProject | GitRepository, Kustomization, HelmRelease |
| Many clusters | Often one central Argo CD managing many | Usually Flux in each cluster |
| Helm | Renders charts, applies the output | Runs real Helm releases through its Helm controller |
| Updating image versions in Git | Separate Image Updater tool | Built-in image automation controllers |
| Access control | Its own RBAC and single sign-on | Kubernetes RBAC |
Both do the essentials well: pull-based reconciliation, drift detection, pruning, health assessment, Helm and Kustomize support, and notifications.
What tends to decide it:
- Teams who want to see applications, their health and their differences, and give developers a UI, often choose Argo CD.
- Teams who want everything in Git, small composable parts, and automatic image updates committed back to Git, often choose Flux.
510. How do you promote a release between environments with GitOps?
The short answer
Keep each environment's desired state in its own folder of the configuration repository, such as an overlay per environment. Promotion is then a change to the next environment's folder, usually setting the new image digest, made by a pull request. CI or a promotion tool opens that pull request automatically once the previous environment's checks pass, and the required approvals gate it. When it merges, the GitOps tool deploys it. Avoid a separate branch per environment.
How it runs:
- CI builds the image and commits its digest to
overlays/dev/(question 133). - Argo CD syncs dev. Checks run against dev.
- When they pass, automation opens a pull request changing
overlays/staging/to the same digest. - Merged, Argo CD syncs staging. Longer checks run.
- When they pass, a pull request for
overlays/prod/is opened, which needs approval from the right people throughCODEOWNERSand branch protection (question 186). - Merged, Argo CD syncs production.
Why folders, not branches. Environment branches drift apart, conflict on merges, and make "what differs between staging and production?" hard to answer (question 194). With folders on one branch, the difference is just a diff between two folders.
Tools that automate this: Argo CD Image Updater and Flux's image automation can update digests in Git automatically, and Kargo is a tool designed specifically to model and automate promotion between stages.
511. How do you handle secrets in a GitOps workflow?
The short answer
Never commit plain Kubernetes Secrets, because base64 is not encryption and Git history is forever. Instead, you have two choices. The usual best choice is to commit only references, such as an ExternalSecret that points at a value kept in Azure Key Vault or AWS Secrets Manager. The other is to commit encrypted secrets that only the cluster can decrypt, with Sealed Secrets or SOPS. The secret manager or the decryption key then becomes the thing to protect, back up and rotate.
| Approach | What is in Git | Where the real value lives | Notes |
|---|---|---|---|
| External Secrets Operator | A reference, such as "checkout-db-password in Key Vault" | The outside secret manager | Easy rotation; nothing secret in Git |
| Secrets Store CSI Driver | A list of which vault secrets to mount | The outside secret manager | Values only ever reach pod files |
| Sealed Secrets | An encrypted Secret | Git, encrypted | Back up the controller's private key |
| SOPS | YAML with encrypted values | Git, encrypted with a cloud key | Flux decrypts natively; Argo CD needs a plugin |
Question 492 shows the External Secrets Operator and Sealed Secrets in detail.
SOPS in a sentence. SOPS encrypts the values inside a YAML file, leaving the keys readable, using a key from a cloud key service or a personal encryption key. The file can be reviewed in pull requests, because you can see which setting changed, but not its value.
Why references usually win:
- Rotation happens in the vault, and reaches the cluster on the next refresh, with no Git change.
- Access is controlled and audited by the vault, not by who can read the repository.
- Leaks: even a full copy of the repository reveals nothing secret.
Whatever you choose, scan the configuration repository for secrets in every pull request (question 215), because the most common GitOps secret leak is someone committing a plain Secret by mistake.
512. How do you roll back a deployment in a GitOps setup?
The short answer
Revert the commit in the configuration repository that introduced the bad version, and let the GitOps tool sync the cluster back to the previous state. That keeps Git as the truth, and the rollback itself is recorded and reviewed. Do not use kubectl rollout undo on resources the GitOps tool manages, because it makes the cluster differ from Git, and self-heal will put the bad version straight back. Argo CD also offers a rollback button, which pauses automatic sync until Git is fixed.
bash
git revert 7a1c9e2 # the commit that changed the prod digest to the bad image
git push origin main # through a fast-tracked pull request if branch rules requireThe GitOps tool sees the new commit and applies the previous digest, with a normal rolling update.
Why kubectl rollout undo misbehaves here:
- You undo the rollout by hand. The cluster now runs the old version.
- Git still says the new version.
- Argo CD reports OutOfSync, and with self-heal on, applies Git again, bringing the bad version back within minutes.
Argo CD's built-in rollback. The UI or argocd app rollback can re-apply an earlier synced revision. Because that also makes the cluster differ from Git, Argo CD turns off automated sync for that Application while it is in effect. Use it for speed in an emergency, then fix Git and re-enable automated sync.
Same limits as any rollback: it restores what Git describes. It does not undo database changes, and if the bad release ran an incompatible migration, rolling back the app may not be enough (question 286).
513. What is a service mesh, and what problems does it solve?
The short answer
A service mesh is a layer of network proxies that handles traffic between services, so the services themselves do not have to. Every request between services passes through these proxies, which a central control plane configures. That gives you several things without changing application code. Every call gets encryption and identity through mutual TLS. You get traffic control, such as retries, timeouts and weighted canary splits. Every call gets detailed metrics and traces. And you can set rules about which service may call which. Istio, Linkerd and Cilium are common meshes.
Two parts:
- The data plane: the proxies that carry every request. In Istio these are Envoy proxies, and in Linkerd its own lightweight proxy.
- The control plane: tells every proxy its configuration, issues certificates, and distributes policies.
What it gives you, without code changes:
| Feature | Example |
|---|---|
| Encryption and identity | Every call encrypted with mutual TLS; each service proves who it is (question 514) |
| Authorisation | "Only web may call checkout-api on POST /orders" |
| Reliability | Retries, timeouts, circuit breakers that stop calls to a failing service |
| Traffic shaping | 90/10 canary splits, request mirroring (question 482) |
| Observability | Request rate, errors and latency for every pair of services, plus traces |
Where the mesh fits beside other pieces: the gateway handles traffic into the cluster, sometimes called north-south traffic. The mesh handles traffic between services inside, called east-west traffic.
514. How does mutual TLS work in a service mesh?
The short answer
The mesh gives every workload a cryptographic identity based on its service account, in the form of a short-lived certificate issued by the mesh's own certificate authority and renewed automatically. When one service calls another, their proxies perform a TLS handshake in which both sides present certificates and check each other's, which is what makes it mutual. The connection is encrypted, each side knows exactly which service is on the other end, and policies can then allow or deny calls by identity.
Ordinary TLS versus mutual TLS:
- In ordinary TLS, as when a browser visits a website, only the server proves its identity with a certificate. The client stays anonymous.
- In mutual TLS, both sides present certificates. The server knows exactly which client is calling, as well as the other way round.
How a mesh makes it automatic:
- Identity. Each workload's identity comes from its namespace and service account, often written as a SPIFFE ID, a standard format for workload identities, such as
spiffe://cluster.local/ns/shop-prod/sa/checkout-api. - Certificates. The control plane acts as a certificate authority. It issues each proxy a certificate for its workload's identity, valid for a short time, often around a day, and renews it well before expiry. No person ever handles these certificates.
- The handshake.
web's proxy connects tocheckout-api's proxy. Each presents its certificate, each checks that the other's was signed by the mesh's authority, and they agree an encrypted connection. - The app sees none of it. Each app talks plain HTTP to its own local proxy.
Using identities in rules, for example with Istio:
yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: checkout-api-callers
namespace: shop-prod
spec:
selector:
matchLabels: { app.kubernetes.io/name: checkout-api }
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/shop-prod/sa/web"] # only the web service's identity
to:
- operation: { methods: ["POST"], paths: ["/orders"] }Permissive and strict modes. While moving to a mesh, permissive mode accepts both encrypted and plain connections, so services not yet in the mesh still work. Strict mode accepts only mutual TLS, which is the goal once everything has joined.
515. What is the difference between sidecar-based and sidecarless, or ambient, service meshes?
The short answer
In the classic sidecar model, every pod gets its own proxy container, injected next to the app, which handles all of the pod's traffic. That gives fine-grained control, but costs CPU and memory in every pod, and upgrading the mesh means restarting every pod. In the sidecarless model, such as Istio's ambient mode, a shared per-node proxy handles encryption and basic traffic for all pods on that node, and full HTTP features are added only where needed, through separate proxies. It is cheaper and simpler to operate, with somewhat different trade-offs.
| Sidecar | Ambient, or sidecarless | |
|---|---|---|
| Where the proxy runs | In every pod | One per node, plus optional shared HTTP proxies |
| Resource cost | Per pod | Per node, much lower overall |
| Joining the mesh | Inject the sidecar, restart pods | Label the namespace, no restart |
| Upgrading the mesh | Restart every pod | Upgrade the node proxies |
| HTTP features, retries, routing | Everywhere | Where a waypoint proxy is added |
| Start-up and shutdown ordering | Historically tricky; native sidecars fix it | Not an issue |
Istio's ambient mode, stable since late 2024, splits the work:
- ztunnel, a small per-node proxy, gives every pod mutual TLS and layer 4 policies, meaning rules about who may connect.
- Waypoint proxies, deployed only for the services that need them, add layer 7 features: HTTP routing, retries and request-level authorisation.
Cilium's mesh uses eBPF in the kernel for much of the work, with a per-node proxy for HTTP features.
The sidecar ordering problem, and its fix. With classic sidecars, the app sometimes started before its proxy was ready, so its first network calls failed, and at shutdown the proxy sometimes stopped first, dropping the app's last requests. Native sidecars, stable in Kubernetes 1.33, start before and stop after the app, which fixes this for sidecar meshes too (question 355).
516. When should you not adopt a service mesh?
The short answer
When its cost is larger than its benefit. Picture a handful of services, no requirement for encryption between them, and no team with time to run another complex system. There, a mesh adds proxies to every request, new failure modes and new things to upgrade. Debugging gets much harder, for little gain. Simpler tools often cover the actual need: network policies for who may talk to whom, TLS in the app or the gateway for encryption, and OpenTelemetry for tracing.
Signs you do not need one yet:
- Few services. A monolith and two helpers do not need a mesh to manage their traffic.
- No mandate for encryption in transit inside the cluster, and no need for strong per-service identity.
- No capacity to operate it. A mesh is a critical system: if its control plane or certificates break, every call can fail.
- Tight latency budgets. Each proxy hop adds a little time, typically around a millisecond, which matters for some systems.
- Debugging is already hard. A mesh adds another layer where a request can be dropped, retried or delayed.
Simpler alternatives for each common need:
| Need | Simpler option |
|---|---|
| Restrict which services may talk | NetworkPolicies (question 367) |
| Encrypt external traffic | TLS at the gateway (question 372) |
| Encrypt internal traffic | TLS in the app, or the network plugin's built-in encryption |
| Traces and metrics per call | OpenTelemetry in the apps (question 632) |
| Canary traffic splits | Gateway API weights (question 370) |
| Retries and timeouts | Client libraries in the apps |
Signs you do need one: many services owned by many teams, compliance rules requiring encrypted, identity-checked calls everywhere, and a need for uniform traffic policies that no single team could build into every app.
If you adopt one, start small: one namespace in permissive mode, measure the overhead, then grow, and prefer the lightest option that meets the need, such as Linkerd or an ambient mesh.