Appearance
13.7 — Pipelines: From Commit to Production
Getting code from a developer's laptop to the servers that users reach takes several steps. The code is compiled or packaged, checked, and then installed on the servers. Installing a new version on the servers is called deploying. A pipeline is an automated sequence that carries out these steps every time somebody saves a change. This page uses Git words such as commit, branch and pull request. If they are new to you, read 13.11.1 to 13.11.3 first.
Here is the failure this page is built around. A release fails in production. To investigate, the team builds the software again from exactly the same commit. The new build behaves differently. The reason is that the software uses a library, and that library uses another library. That second library published a small update between the two builds. So the second build quietly picked up different code.
The packaged output of a build is called an artefact. In this story, the artefact that was tested is not the artefact that was deployed. The rule that prevents it is: build once, then move that exact artefact through every environment.
1. The words, used precisely
The letters CI/CD stand for three practices, and people often mix them up.
Continuous integration means every developer merges their work into the shared main branch often, at least daily, and each merge is checked automatically. "Integration" means joining everyone's work together. The alternative is for each developer to work apart on their own branch for weeks. The branches drift far from each other, and joining them at the end is slow and painful.
Continuous delivery means every commit that passes the checks is ready to deploy. A person still decides when to deploy it.
Continuous deployment means every commit that passes the checks is deployed, with no person involved.
Most companies practise continuous delivery and call it continuous deployment. The difference matters. Deploying with no person involved is safe only when the automatic checks are strong enough to catch a bad release, and when a bad release can be undone automatically. A team has to build those things first.
2. Build once, promote many
There are two pipelines, and they have different jobs. The most common design mistake is to merge them into one.
The build pipeline runs on every commit. It compiles the code, checks it, packages it and publishes the result as an artefact with a version label. The artefact is immutable, which means it is never changed after it is made. The build pipeline knows nothing about environments.
The release pipeline takes an artefact that already exists and deploys it to one environment, such as development, staging or production. It handles the approvals and supplies the settings that belong to that environment.
commit ──▶ build ──▶ artefact:sha-0e1b ──▶ deploy dev ──▶ deploy staging ──▶ deploy prod
(built once) (same bytes, every time)Always build first and deploy the same artefact everywhere. Suppose you built again for each environment. Each build could pick up different library versions, run on a different build machine and stamp a different time into the output. What you tested in staging would then not be what you shipped to production. Moving one artefact from one environment to the next is called promoting it. You promote the artefact. You never rebuild it.
So the differences between environments must be settings, supplied when the program starts. They must not be inputs to the build. You make one container image, and each environment hands it different settings when it runs (Chapter 14.2.3). If a build option differs per environment, you get a different artefact per environment, and you are back to the original mistake.
An artefact repository is where artefacts are kept. A container registry and a package feed are two kinds. It never alters an artefact once stored. It has rules for how long old ones are kept. Ideally each artefact is also signed when it is built, and the signature is checked before it is deployed (Chapter 8.6.2).
3. Pipeline anatomy
trigger → checkout → restore cache → install → build
→ unit tests → lint & types → security scan
→ package → publish artefact
→ deploy → smoke test → (approval) → promoteA few words in that diagram need explaining. Checkout fetches the code. A lint is an automatic check for common mistakes and untidy style. A type check confirms that values are used consistently, for example that a number is never treated as text. A smoke test is a quick check after deploying that the new version starts and answers at all.
Put the quick, cheap steps first. Lint and type checks take seconds and catch a large share of mistakes. Slower checks take minutes. Suppose a pipeline runs a twenty-minute step first and the formatting check after it. A misplaced comma then costs twenty minutes before anyone hears about it.
A fast pipeline matters. Aim to give the developer an answer within ten minutes. When it takes longer, people stop waiting for it. They move on to other work and save up several changes to send together. That undoes the point of continuous integration, which is small changes merged often.
The machine that runs a pipeline's steps is called a runner. Five things make a pipeline faster.
- Cache the dependencies. Downloading every library on every run is slow, so keep a saved copy. The exact versions in use are listed in a file called the lockfile. Label the saved copy with a fingerprint of the lockfile. Then the copy is replaced exactly when the dependencies change, and never otherwise.
- Split slow steps across several runners so that they run at the same time.
- Run only what changed. When many projects live in one repository, called a monorepo, work out which projects a commit affects and build only those.
- Reuse container layers from earlier builds (Chapter 14.2.2).
- Use a runner of the right size. A larger runner that cuts a twenty-minute build to ten usually costs less than the engineers' time it saves.
Use a fresh runner for every job unless you have a reason not to. A runner that is created for one job and then thrown away is called ephemeral. Nothing from one build can leak into the next. A runner that is kept is faster, because its caches are already filled. But leftovers pile up on it, and an attacker who gets into it once can reach every later build.
4. Branching and merging
Trunk-based development keeps branches short. A developer creates a branch, works on it for a day or two, and merges it into the main branch, which is also called the trunk. Work that is not finished is merged too, but it is hidden behind a feature flag, explained just below. This is what continuous integration looks like in practice. Nothing is left waiting to be joined up later.
GitFlow uses several branches that live for a long time, named develop, release and hotfix. It was designed for software that ships in numbered versions on a schedule, such as a desktop program. It suits a continuously deployed service badly, because work piles up on the long-lived branches and arrives in large, risky merges. Chapter 13.11.8 covers both in detail.
A feature flag is a switch in the code that turns a feature on or off without deploying again. It separates two things that used to happen together. Deploying puts the code on the servers. Releasing lets users see the feature. With a flag, the code is deployed switched off. Later it is switched on for one user, for 5% of users, or for everyone. Flags need looking after. Give every flag an owner and a date by which it will be removed. Every flag doubles the number of ways the program can behave. With two hundred old flags, nobody can reason about what the program will do.
A merge queue catches two changes that are each fine alone and break when combined. Suppose one pull request renames a function and another adds a new call to the old name. Each one passes its checks against the main branch. Once both are merged, the main branch is broken. A merge queue lines the pull requests up and checks each one together with those ahead of it before merging. Add one as soon as a busy repository starts breaking its main branch.
Make the checks and the review compulsory. Git hosting services let you set branch protection rules, which refuse a merge until the checks pass and a reviewer approves. Use those rules. Do not rely on people remembering.
5. Deploying
| Strategy | Downtime | Cost | Rollback |
|---|---|---|---|
| Recreate | Yes | Low | Redeploy |
| Rolling | No | Low | Roll forward or back |
| Blue-green | No | 2× capacity | Instant — flip back |
| Canary | No | Slightly more | Shift traffic back |
| Feature flag | No | None | Toggle |
Rollback in the table means going back to the previous version after a bad release. Take the strategies one at a time.
Recreate stops the old version and then starts the new one. Users see an outage in between.
Rolling replaces the copies of the application a few at a time, so some copies are always running. It is the usual choice, and it is what Kubernetes does by default (Chapter 14.3.5).
Blue-green runs a complete second environment. The old version is "blue" and the new one is "green". You start green beside blue, switch all the traffic over to green, and keep blue running for a while. If green misbehaves, you switch back in an instant. That instant rollback is what you pay double for. Choose blue-green when a bad release would be very costly and a slow recovery is not acceptable.
Canary sends a small share of the traffic, say 5%, to the new version. You watch its error rate and its response time. If they look healthy, you send more. The name comes from the canaries that miners carried to warn them of bad air. Progressive delivery is the same thing done by a tool. It moves 5%, compares the measurements with your targets, moves more, and switches back by itself if the targets are missed. This is what makes continuous deployment safe enough to use.
Two rules apply to every strategy.
Every new version must work alongside the version it replaces. During a rollout both versions run at the same time. So the new one must still accept the same requests, read the same messages and work with the same database tables as the old one. This is called being backward compatible.
Change the database separately, and before the code that needs the change. A change to the structure of a database is called a migration. Migrations follow the expand-and-contract pattern of Chapter 7.2.4. If a migration is bundled into a release, rolling the release back does not undo the migration. The old code then meets a database it does not understand, and a simple rollback turns into an outage.
6. Security in the pipeline
Chapter 8.6.2 covers the reasoning; this is the checklist.
Use OIDC federation for cloud credentials. A pipeline needs permission to deploy to your cloud account. The old way was to store a long-lived cloud key in the pipeline's settings, where a leak would hand over the whole account. OIDC federation removes the stored key. On each run, the pipeline service gives the pipeline a signed note saying which repository, branch and workflow it is. The cloud checks the signature and hands back a credential that works for a few minutes. No cloud key is stored in the pipeline at all. In most companies that key was the most valuable secret they had.
Never let pull requests from forks see your secrets. A fork is an outsider's copy of your repository. Anyone can open a pull request from one, and its code runs in your pipeline. If secrets were available to it, a stranger could read them.
Refer to third-party actions by their full commit ID, not by a tag. An action is a ready-made pipeline step written by someone else. A tag such as v4 is a movable label, and its owner can point it at different code tomorrow. A commit ID always means exactly the same code.
Give the pipeline's own token as little permission as possible. Make it read-only by default, and grant write access only to the jobs that need it.
Separate build from deploy privileges, and require approval for changes to deployment workflows.
Sign artefacts at build and verify at deploy, so a compromised registry is not a compromised production.
7. GitHub Actions
yaml
name: ci
on:
push: { branches: [main] }
pull_request:
permissions:
contents: read # (1)
id-token: write # (2)
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true # (3)
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix: { node: [22, 24] } # (4)
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # (5) v7.0.1
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: { node-version: '${{ matrix.node }}', cache: npm }
- run: npm ci
- run: npm run lint && npm test
deploy:
needs: test
if: github.ref == 'refs/heads/main'
environment: production # (6)
runs-on: ubuntu-latest
steps:
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: arn:aws:iam::123456789012:role/deploy # (7)
aws-region: eu-west-2(1) Least privilege by default, granting only what is needed. (2) Required for OIDC. (3) Cancel superseded runs on the same branch — a large and free saving on a busy repository. (4) A matrix runs the job once per combination in parallel. Here it is the two Node.js versions still supported in 2026: Node 20 reached end of life in April 2026, so testing on it would only prove you work on a version nobody should run. (5) Pinned by SHA, with the human-readable version in a comment so a reviewer can see what it is. A tag such as v7 can be moved to different code by whoever controls the action. A full commit SHA cannot. (6) An environment carries protection rules — required reviewers, wait timers, and environment-scoped secrets. This is where the deployment approval lives. (7) No stored key — the OIDC token is exchanged for a role.
Reusable workflows and composite actions remove duplication across repositories, which matters as soon as you have more than a handful.
Self-hosted runners are needed for private network access or specialised hardware, and they are a security boundary: never run untrusted pull requests on a self-hosted runner with network access to anything you care about.
8. Jenkins
Jenkins predates all of this, and it is still everywhere — because it runs anywhere, has a plugin for everything, and a great deal of institutional pipeline logic lives in it.
Architecture: a controller schedules and holds configuration; agents execute. Never run builds on the controller — a build with controller access is a full compromise of the automation system.
groovy
pipeline {
agent { label 'linux' } // (1)
options { timeout(time: 30, unit: 'MINUTES') }
environment { REGISTRY = 'registry.example.com' }
stages {
stage('Checkout') { steps { checkout scm } } // (2)
stage('Build') { steps { sh 'make build' } }
stage('Test') {
steps { sh 'make test' }
post { always { junit 'reports/*.xml' } } // (3)
}
stage('Deploy') {
when { branch 'main' } // (4)
steps {
withCredentials([string(credentialsId: 'deploy-token', variable: 'TOKEN')]) {
sh 'make deploy' // (5)
}
}
}
}
post { failure { slackSend(message: "Build failed: ${env.BUILD_URL}") } }
}(1) Which agent pool. (2) checkout scm is the idiom worth understanding. In a multibranch pipeline, Jenkins discovers branches and pull requests in a repository and creates a job per branch, injecting the source configuration into the scm variable automatically. So checkout scm means "check out the exact repository, branch and commit that triggered this job" — with no URL or branch hard-coded, which is what makes one Jenkinsfile work for every branch. (3) post blocks run regardless of outcome — the place for test reports and cleanup. (4) Conditional stages. (5) Credentials are injected as variables scoped to a block and masked in logs, rather than being in the environment for the whole build.
Shared libraries put common pipeline code in a separate repository, loaded with @Library. This is what stops fifty Jenkinsfiles duplicating the same forty lines, and it is the main thing separating a maintained Jenkins estate from an unmaintainable one.
The honest assessment. Jenkins is powerful, self-hosted, and it requires ongoing care: plugin updates that break, a controller that becomes a single point of failure, and Groovy pipeline code that few people want to maintain. New projects generally choose a hosted service; existing Jenkins estates are usually cheaper to maintain than to migrate.
9. Azure DevOps and GitLab
Azure DevOps Pipelines — YAML with stages → jobs → steps, agent pools (Microsoft-hosted or self-hosted), service connections holding credentials for external systems (and workload identity federation so those are not stored secrets), environments with approvals and gates, and Artifacts feeds for packages. Classic drag-and-drop pipelines still exist; YAML in the repository is the right choice because the pipeline is versioned with the code.
GitLab CI — .gitlab-ci.yml with stages and jobs, an integrated container registry, environments and review apps. Review apps are the notable idea: a full ephemeral deployment per merge request, with a URL, torn down on merge.
The concepts transfer completely. Every one of these has: a trigger, a unit of work on a runner, caching, artefacts, environments with approvals, and secret injection. Learn the model once and the syntax is a lookup.
10. GitOps
The repository is the desired state, and an agent inside the cluster pulls it.
Instead of a pipeline holding cluster credentials and pushing changes, Argo CD or Flux runs in the cluster, watches a Git repository, and reconciles — the same control-loop idea as Kubernetes itself (Chapter 14.3.8).
What it buys:
- No cluster credentials in CI, which removes a whole class of risk.
- Git is the audit log: who changed what, when, reviewed by whom.
- Drift detection — a manual change is reverted or flagged.
- Rollback is a revert, and recovery from a lost cluster is pointing the agent at the repository.
What it costs:
- Two repositories — application code and deployment manifests — and a step that updates the second when the first builds. Rendering the manifests and committing them is the usual approach.
- Debugging gains a layer: is it the pipeline, the manifest repository, or the agent?
- Secrets need a solution — sealed secrets, an external store with an operator, or a provider's secret store — because plaintext secrets cannot go in Git.
Worth adopting when you have several clusters or environments. For a single cluster and a small team, a push-based pipeline is simpler and fine.
11. A complete free lab on one machine
Everything above, running on one machine, with no cloud account. This is the fastest way to understand the whole chain, because you build every link yourself and can see each one.
What you will deploy is a small order system with two services that talk through a queue, the shape Chapter 10.8.3 describes:
orders-api, written in Python with FastAPI. It accepts an order over HTTP and puts a message on a queue.orders-worker, written in TypeScript on Node.js. It takes each message off the queue and does the slow work, here just logging that stock is reserved.- RabbitMQ in between, so the API can answer "accepted" at once even when the worker is busy or down.
What you need: a Linux machine, or Windows with WSL2, with 8 GB of memory, plus Docker, kubectl, k3d, Node.js 24 and Python 3. Every component is free and open source. The versions below were current in October 2026. Pin the versions you use, for the same reason section 6 pins actions.
The pieces and how they connect
you ──git push──▶ Gitea ──runs──▶ runner ──docker push──▶ registry
▲ │ │
│ commits new image tag │ pulls image
│ ▼ ▼
└──── orders-deploy repo ◀──watches── Argo CD ──applies──▶ k3d cluster- Gitea is a self-hosted Git server, like a small GitHub. It runs Gitea Actions, which uses the same workflow syntax as GitHub Actions.
- The runner is the program that actually executes the workflow steps. Here it runs directly on your machine, so the steps can use your Docker.
- The registry stores the built container images.
- k3d runs a real Kubernetes cluster inside Docker, in about thirty seconds.
- Argo CD runs inside the cluster and keeps it matching the
orders-deployrepository. This is the GitOps model from section 10.
There are two repositories, exactly as section 10 described. orders holds the code. orders-deploy holds the Kubernetes files that say which image version should be running.
Step 1: start Gitea and create the repositories
bash
docker run -d --name gitea -p 3000:3000 -v gitea-data:/data gitea/gitea:28.2Open http://localhost:3000, accept the defaults on the setup page, and create a user called lab. Then:
- Create a repository called
orders. In its Settings, turn on Enable Repository Actions. Actions are on for the server but off for each new repository. - Create a repository called
orders-deploy, and make it public, so Argo CD can read it without a password. - Under your user Settings → Applications, create an access token with write access to repositories. In the
ordersrepository's Settings → Actions → Secrets, save it as a secret namedDEPLOY_TOKEN.
Step 2: start a runner
In orders → Settings → Actions → Runners, create a registration token. Download the Gitea Runner binary for your system from its releases page, then:
bash
./gitea-runner register --no-interactive \
--instance http://<your-machine-ip>:3000 \
--token <registration-token> \
--name laptop --labels lab:host
./gitea-runner daemon--labels lab:host does two things. lab is the name a workflow asks for with runs-on: lab. :host tells the runner to run those jobs directly on this machine instead of inside a fresh container, so the job can use your Docker to build images. That is fine on a lab machine and wrong on a shared one, for the reason in section 7: any workflow can run any command with your machine's access. Use your machine's network address rather than localhost, because the runner and Gitea must be able to reach each other.
Step 3: create the registry and the cluster
bash
k3d registry create registry.localhost --port 5000
k3d cluster create lab --registry-use k3d-registry.localhost:5000The first command starts a registry container called k3d-registry.localhost, listening on port 5000. The second creates the cluster and tells every node to pull images from that registry. Your machine reaches it by the same name, because any name ending in .localhost points back at your own machine.
Step 4: the service code
orders/api/main.py:
python
import json
import os
import uuid
import pika
from fastapi import FastAPI
from pydantic import BaseModel, Field
RABBIT_URL = os.environ.get("RABBIT_URL", "amqp://guest:guest@rabbitmq:5672/") # (1)
QUEUE = "orders.created"
app = FastAPI()
class OrderIn(BaseModel):
customer_id: str
sku: str
quantity: int = Field(gt=0) # (2)
def publish(message: dict) -> None:
connection = pika.BlockingConnection(pika.URLParameters(RABBIT_URL))
try:
channel = connection.channel()
channel.queue_declare(queue=QUEUE, durable=True) # (3)
channel.basic_publish(
exchange="",
routing_key=QUEUE,
body=json.dumps(message),
properties=pika.BasicProperties(delivery_mode=2), # (4)
)
finally:
connection.close()
@app.post("/orders", status_code=202) # (5)
def create_order(order: OrderIn) -> dict:
order_id = str(uuid.uuid4())
publish({"order_id": order_id, **order.model_dump()})
return {"order_id": order_id, "status": "accepted"}
@app.get("/healthz") # (6)
def healthz() -> dict:
return {"ok": True}(1) The broker's address comes from the environment, so the same image works in every environment. This is the build-once rule from section 2. (2) FastAPI rejects an order with a quantity of zero or less before your code runs, and answers 422 with the reason. (3) durable=True makes the queue survive a RabbitMQ restart. (4) delivery_mode=2 writes each message to disk, so the message survives a restart too. A durable queue full of non-persistent messages comes back empty. (5) 202 Accepted means "received, not finished yet". That is the honest answer, because the worker has not reserved the stock yet. (6) A cheap endpoint Kubernetes calls to decide whether this copy is ready for traffic.
Opening a new broker connection for every order is simple and slow. It is fine for a lab. A real service keeps one connection open and reuses it.
orders/api/requirements.txt and orders/api/Dockerfile:
text
fastapi==0.140.0
uvicorn==0.54.0
pika==1.4.4dockerfile
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
USER 10001
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]requirements.txt is copied and installed before main.py, so that a code change reuses the cached layer with the libraries in it (Chapter 14.2.2). USER 10001 runs the process as a user with no special rights, instead of root.
Running the API locally and calling it gives:
GET /healthz → 200 {"ok": true}
POST /orders quantity 2 → 202 {"order_id": "…", "status": "accepted"}
POST /orders quantity 0 → 422 "Input should be greater than 0"orders/worker/src/worker.ts:
ts
import amqp from "amqplib";
const RABBIT_URL = process.env.RABBIT_URL ?? "amqp://guest:guest@rabbitmq:5672/";
const QUEUE = "orders.created";
const DEAD_LETTERS = "orders.dead";
type OrderCreated = { order_id: string; customer_id: string; sku: string; quantity: number };
function isOrderCreated(value: unknown): value is OrderCreated { // (1)
const v = value as Partial<OrderCreated>;
return typeof v?.order_id === "string" && typeof v.sku === "string"
&& typeof v.quantity === "number" && v.quantity > 0;
}
const connection = await amqp.connect(RABBIT_URL);
const channel = await connection.createChannel();
await channel.assertQueue(QUEUE, { durable: true });
await channel.assertQueue(DEAD_LETTERS, { durable: true });
await channel.prefetch(10); // (2)
await channel.consume(QUEUE, (msg) => {
if (msg === null) return;
let body: unknown;
try { body = JSON.parse(msg.content.toString()); } catch { body = null; }
if (!isOrderCreated(body)) {
channel.sendToQueue(DEAD_LETTERS, msg.content, { persistent: true }); // (3)
channel.ack(msg);
return;
}
console.log(`reserving ${body.quantity} x ${body.sku} for order ${body.order_id}`);
channel.ack(msg); // (4)
});
process.on("SIGTERM", async () => { // (5)
await channel.close();
await connection.close();
process.exit(0);
});(1) A message from a queue is just bytes. TypeScript cannot know its shape, so this function checks it at runtime and tells the compiler what it found. (2) prefetch(10) lets RabbitMQ hand this worker at most ten unacknowledged messages at a time, so one slow worker cannot hoard the whole queue. (3) A message that is not valid JSON, or not an order, goes to a separate dead-letter queue for a human to look at. Without this, a bad message is retried for ever and blocks the ones behind it (Chapter 10.8.1). (4) The ack tells RabbitMQ the work is done and the message can be deleted. If the worker crashes before this line, RabbitMQ gives the message to another worker. That is at-least-once delivery, which is why real work here must be idempotent (Chapter 10.4). (5) Kubernetes sends SIGTERM before stopping a pod. Closing cleanly returns unacknowledged messages to the queue at once, instead of after a timeout.
orders/worker/package.json and orders/worker/Dockerfile:
json
{ "name": "orders-worker", "private": true, "type": "module",
"scripts": { "start": "node src/worker.ts" },
"dependencies": { "amqplib": "^0.10.9" },
"devDependencies": { "@types/amqplib": "^0.10.7", "@types/node": "^24.0.0", "typescript": "^5.9.0" } }dockerfile
FROM node:24-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY src ./src
USER node
CMD ["node", "src/worker.ts"]Node.js 24 runs a .ts file directly. It removes the type annotations as it loads the file, so there is no build step. "type": "module" is what allows the top-level await in the worker. Run npm install once on your machine to create package-lock.json, and commit it, so the image installs exactly the versions you used.
Step 5: the pipeline
orders/.gitea/workflows/ci.yaml:
yaml
name: ci
on:
push:
branches: [main]
jobs:
release:
runs-on: lab # (1)
env:
REGISTRY: k3d-registry.localhost:5000
TAG: ${{ github.sha }} # (2)
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Fast checks
run: |
python3 -m py_compile api/main.py # (3)
cd worker && npm ci && npx tsc --noEmit --strict --module nodenext --target es2022 src/worker.ts
- name: Build and push the images
run: |
docker build -t "$REGISTRY/orders-api:$TAG" api
docker build -t "$REGISTRY/orders-worker:$TAG" worker
docker push "$REGISTRY/orders-api:$TAG" # (4)
docker push "$REGISTRY/orders-worker:$TAG"
- name: Point the deployment repository at the new images
env:
DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }} # (5)
run: |
git clone "http://lab:${DEPLOY_TOKEN}@localhost:3000/lab/orders-deploy.git" deploy
cd deploy
sed -i "s|newTag: .*|newTag: ${TAG}|" kustomization.yaml # (6)
git -c user.name=ci -c user.email=ci@lab.local commit -am "Deploy ${TAG}"
git push(1) Ask for the runner registered with the lab label in step 2. (2) Gitea fills in the github.* names too, for compatibility, so github.sha is the commit being built. The image tag is the commit, so you can always tell exactly which code is running. (3) The cheap checks come first, as section 3 said: a Python syntax check and a strict TypeScript type check take seconds. (4) Each image is built once and pushed under its commit tag. Nothing later rebuilds it. (5) The token is stored as a secret, so it is masked in the job log. (6) The pipeline's last act is not to deploy. It is to edit one line in the other repository and commit. Argo CD does the deploying.
Step 6: the deployment repository
orders-deploy/kustomization.yaml lists the files and pins the image versions. Kustomize, which is built into kubectl and Argo CD, reads it and rewrites each image: in the files below:
yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: orders
resources:
- namespace.yaml
- rabbitmq.yaml
- api.yaml
- worker.yaml
images: # (1)
- name: orders-api
newName: k3d-registry.localhost:5000/orders-api
newTag: initial
- name: orders-worker
newName: k3d-registry.localhost:5000/orders-worker
newTag: initial(1) The deployment files below say only image: orders-api. This block turns that into the full registry address plus a tag. The newTag lines are the only thing the pipeline changes, so every deployment in the history of this repository is a one-line commit.
orders-deploy/namespace.yaml creates the orders namespace. orders-deploy/rabbitmq.yaml:
yaml
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq
stringData: # (2)
username: orders
password: lab-only-password
url: amqp://orders:lab-only-password@rabbitmq:5672/
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq
spec:
replicas: 1
selector:
matchLabels: { app: rabbitmq }
template:
metadata:
labels: { app: rabbitmq }
spec:
containers:
- name: rabbitmq
image: rabbitmq:4-management
ports:
- containerPort: 5672
- containerPort: 15672
env:
- name: RABBITMQ_DEFAULT_USER
valueFrom: { secretKeyRef: { name: rabbitmq, key: username } }
- name: RABBITMQ_DEFAULT_PASS
valueFrom: { secretKeyRef: { name: rabbitmq, key: password } }
readinessProbe:
exec: { command: ["rabbitmq-diagnostics", "-q", "ping"] }
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
spec:
selector: { app: rabbitmq }
ports:
- { name: amqp, port: 5672 }
- { name: management, port: 15672 }(2) A password in Git is acceptable only in a lab, and this is it. Section 10's options replace it in real use. A user is created at all because RabbitMQ's built-in guest user may only connect from inside its own container.
orders-deploy/api.yaml:
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 2
selector:
matchLabels: { app: orders-api }
template:
metadata:
labels: { app: orders-api }
spec:
containers:
- name: api
image: orders-api # (3)
ports:
- containerPort: 8000
env:
- name: RABBIT_URL
valueFrom: { secretKeyRef: { name: rabbitmq, key: url } }
readinessProbe: # (4)
httpGet: { path: /healthz, port: 8000 }
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: orders-api
spec:
selector: { app: orders-api }
ports:
- { port: 80, targetPort: 8000 }(3) A short name that Kustomize replaces. (4) Kubernetes sends traffic to a new copy only after /healthz answers. During a rolling update, that is the gate that stops a broken version from receiving requests (Chapter 14.3.5). orders-deploy/worker.yaml is the same shape without the port, the probe and the Service.
Step 7: install Argo CD and point it at the repository
bash
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
-f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.5/manifests/install.yaml
kubectl apply -f argocd-application.yamlargocd-application.yaml:
yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: orders
namespace: argocd
spec:
project: default
source:
repoURL: http://host.k3d.internal:3000/lab/orders-deploy.git # (5)
targetRevision: main
path: .
destination:
server: https://kubernetes.default.svc
namespace: orders
syncPolicy:
automated: { prune: true, selfHeal: true } # (6)(5) Inside a k3d cluster, host.k3d.internal is the name for your own machine, which is where Gitea is listening. (6) automated means Argo CD applies changes without being asked. prune deletes things removed from the repository. selfHeal undoes changes made by hand. Those three words are the whole GitOps promise in configuration.
Step 8: push, and watch the chain
Commit the code to orders and the files to orders-deploy, then push both. In order, you will see:
- Gitea shows a workflow run on
orders. The fast checks pass, two images are pushed, and a new commit appears inorders-deploycalled "Deploy" followed by the commit SHA. - Within about three minutes, Argo CD notices the new commit.
kubectl get pods -n orders -wshows new pods starting, becoming ready, and the old ones stopping. - Send an order, by forwarding a port to the API:
bash
kubectl port-forward -n orders svc/orders-api 8080:80 &
curl -s -X POST localhost:8080/orders -H 'content-type: application/json' \
-d '{"customer_id":"c_42","sku":"SKU-9","quantity":2}'
kubectl logs -n orders deploy/orders-workerThe worker's log shows a reserving 2 x SKU-9 for order … line for the order you just sent.
Step 9: break it on purpose
This is where the understanding is. Each break shows one idea from this chapter working:
- Ship a broken API. Change
/healthzto return an error and push. The rollout stops with the old pods still serving. The readiness probe kept the broken version out of traffic. - Change the cluster by hand. Run
kubectl scale deploy/orders-api -n orders --replicas=5. Within moments Argo CD sets it back to 2. That isselfHeal, and it is also why hand fixes do not survive in a GitOps cluster. - Stop the broker. Scale
rabbitmqto 0 replicas inorders-deployand push. Orders now fail at the API, because publishing needs the broker. The worker crashes and Kubernetes restarts it until the broker returns. Scale it back to 1 and watch both recover on their own. - Send a bad message. Open the RabbitMQ management page with
kubectl port-forward -n orders svc/rabbitmq 15672, log in asorders, and publishnot jsontoorders.created. It lands inorders.deadinstead of blocking the queue. - Roll back. Revert the last "Deploy" commit in
orders-deployand push. Argo CD puts the previous images back. Rollback is agit revert, exactly as section 10 promised.
Grafana and Prometheus are a good next addition, installed with their Helm chart, to see request rate and queue depth (Chapter 10.10). The chain above is complete without them.
What the interviewer will push on
"Why build once and promote?" Because rebuilding per environment means you did not test what you shipped — dependency resolution and build environments drift. The artefact is the unit of promotion, and environment differences must be runtime configuration, not build inputs, or you have per-environment artefacts by another name.
"How do you deploy a database migration safely?" Expand and contract, deployed ahead of and separately from the code, so both old and new versions work against the intermediate schema. Then the reason: a migration bundled into a release cannot be rolled back with it, which is how a rollback becomes an outage.
"Blue-green or canary?" Blue-green for instant rollback at the cost of double capacity; canary to limit blast radius and catch what tests missed, ideally progressive with automated rollback on objective breach. Feature flags are the cheapest rollback of all — a toggle rather than a deploy.
"How does your pipeline get cloud credentials?" OIDC federation — a short-lived token proving the repository and branch, exchanged for a role. No stored key. Then the fork rule: never expose secrets to pull requests from forks, and never run untrusted code on a self-hosted runner with network access.
"What does checkout scm mean in a Jenkinsfile?" In a multibranch pipeline, Jenkins injects the triggering repository, branch and commit into the scm variable, so one Jenkinsfile works for every branch and pull request with nothing hard-coded. It is the mechanism that makes multibranch pipelines work.
"What does GitOps actually give you?" No cluster credentials in CI, Git as the audit log, drift detection, and rollback as a revert. Then the honest costs: a second repository and an update step, an extra layer when debugging, and a real secrets problem because plaintext cannot go in Git. Worth it at several clusters; unnecessary for one.
One thing to volunteer: point out that pipeline feedback time is a productivity metric, not a technical detail. Past about ten minutes people stop waiting and start batching commits, which reverses the point of continuous integration — so cancelling superseded runs, sharding tests and caching on the lockfile hash are engineering-throughput work, not tidying.
Recall
- Build once, promote the same artefact. Rebuilding per environment means you did not test what you shipped. Environment differences are runtime configuration, never build inputs.
- Build pipeline (per commit, environment-agnostic) and release pipeline (per environment, approvals) are different jobs.
- Order the pipeline cheap and broad first, and aim for ten minutes to feedback — past that, people batch commits and continuous integration stops working. Cache on the lockfile hash, shard tests, cancel superseded runs, and prefer ephemeral runners.
- Trunk-based plus feature flags is what makes continuous integration real; every flag needs an owner and a removal date. A merge queue fixes semantic conflicts between independently-passing pull requests.
- Every deploy must be backward compatible with the version it replaces, and migrations go ahead of and separately from the code — expand and contract — or a rollback becomes an outage.
- Security: OIDC federation instead of stored cloud keys, no secrets to fork pull requests, actions pinned by SHA, least-privilege pipeline token, build and deploy privileges separated, artefacts signed and verified.
checkout scmworks because a multibranch pipeline injects the triggering repository, branch and commit — oneJenkinsfilefor every branch. Never build on the Jenkins controller, and use shared libraries.- GitOps removes cluster credentials from CI and makes Git the audit log with drift detection and revert-based rollback, at the cost of a second repository, an extra debugging layer and a real secrets problem.
Self-test: What breaks if you rebuild per environment? · Why must a migration deploy separately from the code that uses it? · What does an "environment" object give you in a pipeline? · Which credential mechanism removes the highest-value secret from CI? · What does scm contain in a multibranch Jenkins job? · What is the productivity cost of a twenty-minute pipeline?
Next: 13.8 covers the other half of reproducibility — describing infrastructure as code, what state actually is, and why the most dangerous line in Terraform is a plan nobody read.