Skip to content

Interview PrepCI/CD

Artifacts, Migrations and Verified Releases ​

216. What is an artifact repository, and why not rely only on build outputs stored on a CI runner? ​

The short answer

An artifact repository is a dedicated, durable store for versioned build outputs, such as JAR files, npm packages, Python wheels and container images. It adds access control, checksums and retention rules. It also gives other tools a standard way to download them. A CI runner is temporary: it may be deleted after the job, and the CI system's own "artifact" storage is short-lived and not designed for other systems to install from. Release outputs need a permanent, trustworthy home.

Why not keep outputs on the runner, or in CI's artifact storage?

  1. Runners disappear. An ephemeral runner is destroyed after each job (question 183). Its files go with it.
  2. CI storage expires. CI systems keep job artifacts only for a limited time, often days or weeks, and are built for passing files between jobs, not for distributing releases.
  3. No standard way to consume them. npm install, pip install, Maven and Kubernetes all expect a proper repository or registry. None of them can install from "the files of pipeline run 9137".
  4. No guarantees. A release must be immutable, findable by version, and checked by checksum. A folder on a runner offers none of that.

What an artifact repository gives you:

FeatureWhy it matters
Versioned, immutable releasesVersion 2.3.1 is always the same bytes
Standard protocolsEvery package manager and runtime can download from it
Checksums and signaturesCorruption and tampering are detected
Access controlWho may publish, who may download
Retention rulesOld snapshots are cleaned up, releases are kept
Proxying public sourcesFaster, and protected from outages and removed packages

Examples: JFrog Artifactory and Sonatype Nexus host almost every format. GitHub Packages, GitLab's package registry, Azure Artifacts and AWS CodeArtifact are built into their platforms. Container images go to an OCI registry (question 34).

Proxying is worth a sentence. Pointing builds at your repository, which caches the public registries, means a public package being deleted, or the public registry being down, does not break your builds.

217. How would you store and version JARs, npm packages, Python wheels, and container images? ​

The short answer

Each in a repository of its own format, published once per version, and never overwritten. JARs go to a Maven repository under a group, an artifact name and a version. npm packages go to an npm registry under a scoped name such as @shop/money. Python wheels go to a Python package index. Container images go to an OCI registry, with tags for people and digests for deployments. Release versions are immutable everywhere.

FormatRepository typeIdentified byPublish with
JARMavencom.shop:money:2.3.1mvn deploy
npm packagenpm registry@shop/money@2.3.1npm publish
Python wheelPython package indexshop_money-2.3.1-py3-none-any.whltwine upload
Container imageOCI registryshop/money-api:2.3.1 and its digestdocker push

Format details worth knowing:

  1. Maven separates snapshots from releases. A version ending in -SNAPSHOT, such as 2.4.0-SNAPSHOT, is a moving development build that can be republished. A release version, such as 2.3.1, must never change once published. Repositories are usually configured to reject overwriting releases.
  2. npm uses scopes, such as @shop/, to keep a company's packages apart from public ones. A .npmrc file tells npm which registry serves which scope. dist-tags, such as latest and next, are movable labels, like image tags.
  3. Python wheels are pre-built package files. Their file names encode the package name, the version, and which Python versions and platforms they suit. Pip downloads from an index whose address you can set, so a private index can serve private wheels and proxy public ones.
  4. Container images use tags and digests (question 35). Turn on tag immutability for release tags.

The same three rules for all four:

  1. Publish once, from the pipeline, never from a laptop.
  2. Never overwrite a release. A fix gets a new version number.
  3. Keep the link to the source: the commit SHA in metadata or labels (question 220).

218. What is the difference between caching a dependency and publishing a release artifact? ​

The short answer

A cache is a speed-up: a copy of something you could fetch or rebuild anyway, kept nearby so the next job is faster. It can be thrown away at any moment, and nothing should depend on it existing. A release artifact is an official product: published once, immutable, versioned, kept for as long as anyone might need it, and used as the source of truth by deployments and other teams.

CacheRelease artifact
PurposeSpeedDistribution and record
If it disappearsJobs get slowerDeployments and rollbacks break
Can changeYes, replaced freelyNever
Identified byA hash of its inputs, such as the lock fileA version and a digest
Kept forUntil evicted or expiredLong, by policy
Trusted as source of truthNoYes

The classic mistake is to use one as the other:

  • A cache treated as an artifact: a deploy job expects to find the build output in the CI cache. One day the cache was evicted, because it filled up, and the deploy fails, or worse, picks up an older entry.
  • An artifact treated as a cache: an image tag such as latest is overwritten on every build "to save space", and the version needed for a rollback is gone.

The rule: anything a deployment or another team needs must be a published artifact. Caches only ever make things faster, and the pipeline must still work, slowly, with every cache empty.

219. Why should a pipeline build an artifact once and promote that same artifact across environments? ​

The short answer

So that what you tested is exactly what you ship. Every rebuild is a chance for something to differ: a newer base image, a dependency resolved differently, another tool version. Promoting one artifact by its digest means every check in every environment applies to the bytes production will run. It also makes releases faster, keeps one signature and one provenance record valid everywhere, and makes rollback as simple as pointing back at the previous digest.

This is the same principle as question 134, applied to any kind of artifact, not only container images. In short:

  1. Tested equals shipped. Two builds of one commit can produce different bytes (question 85). Only promotion guarantees they are the same.
  2. Faster. Promotion is a pointer change. No compile, no download, no build queue.
  3. One proof. The SBOM, scan results, signature and provenance created at build time stay true for every environment (question 146).
  4. Simple rollback. The previous version's digest is still in the registry, already tested and proven.
  5. Clear history. "This digest went to staging at 11:00 and production at 14:00" is a complete, simple story.

What must change per environment is configuration, delivered separately at deploy time (question 226). If an artifact contains environment-specific values, you are forced to build it once per environment, which is the thing this rule exists to avoid.

220. How would you label an image so that its source commit and build can be identified later? ​

The short answer

Tag it with the commit SHA, and add the standard OCI labels: the source repository, the exact revision, the version, the creation time and a link to the build. Pass the values in from the pipeline as build arguments. Then anyone holding the image can read its origin with one docker inspect, and a signed provenance statement adds a record that cannot be faked.

dockerfile
ARG GIT_SHA=unknown
ARG VERSION=dev
ARG BUILD_URL=unknown
ARG CREATED=unknown
LABEL org.opencontainers.image.source="https://github.com/shop/checkout-api" \
      org.opencontainers.image.revision="${GIT_SHA}" \
      org.opencontainers.image.version="${VERSION}" \
      org.opencontainers.image.created="${CREATED}" \
      org.opencontainers.image.url="${BUILD_URL}"

These ARG lines are placed near the end of the Dockerfile, just before the LABEL. They change on every build, so declaring them early would break the build cache for everything after them (question 64).

bash
docker buildx build \
  --build-arg GIT_SHA="$(git rev-parse HEAD)" \
  --build-arg VERSION=1.4.2 \
  --build-arg CREATED="$(git log -1 --format=%cI)" \
  --build-arg BUILD_URL="https://github.com/shop/checkout-api/actions/runs/9137" \
  -t myacr.azurecr.io/shop/checkout-api:3f9c2ab \
  -t myacr.azurecr.io/shop/checkout-api:1.4.2 --push .

Using the commit's own time for created, instead of "now", keeps the label identical across rebuilds of the same commit (question 85).

Reading it back:

bash
docker image inspect myacr.azurecr.io/shop/checkout-api:1.4.2 \
  --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'
3f9c2ab8e71d4c0a9f5b2e6d1c3a7f8b9e0d4c21

Labels are easy to fake, because anyone building an image can write any label. They are for convenience. For proof, use a signed provenance statement generated by the pipeline (question 146), which records the same facts and is signed by the pipeline's identity.

221. Why is a mutable tag such as latest risky for production releases? ​

The short answer

Because the same name can point at different images over time, so it does not tell you, or your tools, what is actually running. Machines that pulled at different moments run different code under one name. A deployment that reuses the tag may not roll out at all, since the configuration did not change. Two pipelines pushing at once can overwrite each other. And rollback has nothing to point at. Use unique tags per build, deploy by digest, and make release tags immutable in the registry.

Question 87 covers the general risks. In a pipeline, three more appear:

  1. Races between pipelines. Two builds finish within seconds of each other and both push latest. Whichever pushed last wins. If the older commit's build was slower, latest now points at older code.
  2. Caches keep the wrong one. Nodes and caches that already hold latest may keep using their copy, so different parts of the system disagree about what latest means.
  3. Audit is impossible. "We deployed latest on Tuesday" cannot be checked later, because latest has moved since.

The fix, as a rule set:

  1. Every build gets a unique tag: the commit SHA.
  2. Releases get a semantic version tag, such as 1.4.2, which is never moved.
  3. Deployments reference the digest.
  4. The registry refuses to overwrite tags in production repositories.

If some tool must have a moving tag, such as staging-current, use it only as a convenience pointer that nothing deploys from.

222. How would you prevent two concurrent pipeline runs from deploying out of order? ​

The short answer

Allow only one deployment to an environment at a time, and refuse to deploy anything older than what is already there. Concurrency groups in GitHub Actions, resource groups in GitLab and exclusive locks in Azure Pipelines queue deployments per environment. A check before deploying confirms the new commit is newer than the one running. With GitOps, the order of commits in the configuration repository decides, which makes ordering much easier.

The problem, as a timeline:

timerun A, commit 101: slow checks...deploys 101run B, commit 102deploys 102102 is overwritten by older 101
The out-of-order deploy. The newer change reaches production first, and the older one, finishing later, quietly replaces it. Nobody asked for a rollback, but one happened.

Fix 1: one deployment per environment at a time.

yaml
jobs:
  deploy-prod:
    environment: production
    concurrency:
      group: deploy-production       # (1)
      cancel-in-progress: false      # (2)
  1. All runs that use this group form a queue. Only one runs at a time.
  2. A running deployment is never cancelled halfway. A newer run waits. In GitHub Actions, if several runs are waiting, only the newest pending one is kept, which is usually what you want.

Other platforms have the same idea: GitLab's resource_group, Azure Pipelines' exclusive lock check on an environment, and Jenkins' lock step or disableConcurrentBuilds.

Fix 2: never deploy older over newer. A queue alone still runs the older job if it was queued first. So, before deploying, check that the new commit comes after the one currently deployed:

bash
current=$(kubectl get deploy checkout-api -n shop-prod -o jsonpath='{.metadata.annotations.shop/commit}')
if ! git merge-base --is-ancestor "$current" "$GITHUB_SHA"; then
  echo "refusing: $GITHUB_SHA is not newer than deployed $current"; exit 1
fi

git merge-base --is-ancestor A B succeeds only if commit A is in B's history, meaning B is the newer one.

Fix 3: let Git decide, with GitOps. Deployments can happen by committing a new digest to a configuration repository. Then Git itself puts those commits in a single order, and the GitOps tool always applies the latest one (question 507).

223. How do you handle dependency caching without accidentally reusing stale or untrusted dependencies? ​

The short answer

Key each cache on exactly what determines its contents, mainly the hash of the lock file plus the operating system and tool version, so any dependency change creates a new cache. Keep verifying integrity on install, using lock-file hashes, so a tampered cache is caught. Isolate caches so code from untrusted pull requests cannot write into the cache that main and releases read. And let caches expire.

1. Key the cache correctly.

yaml
- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ runner.os }}-node20-${{ hashFiles('package-lock.json') }}   # (1)
    restore-keys: |
      npm-${{ runner.os }}-node20-                                            # (2)
  1. The key includes the lock file's hash. Change any dependency and the key changes, so a fresh cache is built.
  2. A fallback: if no exact match exists, start from the most recent cache with this prefix. That is safe here because npm ci still installs exactly what the lock file says, and only reuses downloads that match.

2. Verify on every install. The cache only provides speed. Correctness comes from the lock file:

  • npm ci checks every package against the integrity hash recorded in package-lock.json.
  • pip install --require-hashes refuses any package whose hash is not listed.
  • Go checks downloads against go.sum.

So even a corrupted or tampered cache entry is rejected.

3. Keep untrusted code away from trusted caches. A pull request from a stranger could write a poisoned entry into a shared cache, which a release build would later trust. Defences:

  • GitHub Actions scopes caches by branch: a pull request can read caches from the base branch, but caches it creates are only visible to that pull request, never to main.
  • Do not cache build outputs across trust levels, only downloaded dependencies that are verified on install.
  • Release builds can skip caches entirely, if speed matters less than certainty.

4. Expire caches. Most CI caches evict unused entries after some days. Old caches can also hide that a dependency is no longer available upstream, which you want to find out about sooner rather than later.

224. How would you speed up a slow pipeline using caching, parallelism, test splitting, or incremental builds? ​

The short answer

Measure first, then attack the critical path, the longest chain of steps that must run in order. Cache downloads and build layers. Run independent jobs in parallel. Split long check suites into parallel shards balanced by past timings. Build and check only what a change affects. Skip work that does not apply to the change. Reuse outputs between jobs instead of rebuilding. Use faster machines where the step is truly compute-bound.

Step 1: measure. Find where the minutes go: per job, per step, including time spent waiting in the queue for a runner. Many teams discover the slow part is not what they assumed.

Step 2: apply the right technique:

TechniqueWhat it doesTypical saving
Dependency cacheSkip re-downloading packages1 to 5 minutes
Docker layer cacheReuse unchanged image layers (question 77)Most of the image build
Parallel jobsRun independent checks at onceTotal becomes the longest job
Check splittingSplit one long suite into shards20 min becomes 4 × 5 min
Affected-only buildsMonorepo builds only what changed (question 193)Often most of the pipeline
Path filtersSkip pipelines for documentation-only changesWhole runs
Build once, reusePass the built output to later jobs as an artifactRepeated builds
Faster runnersMore CPU and faster disksCompute-heavy steps
Shallow cloneFetch only the latest commitSeconds on big repositories
Pre-built runner imagesTools already installedSetup time per job

Check splitting, a little more: give each shard a set of checks so that all shards take about the same time, using recorded timings from previous runs. Many CI tools and check runners support this directly.

yaml
strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npx playwright test --shard=${{ matrix.shard }}/4

What not to do: delete slow checks or security scans to save time. Speed bought by removing safety is a trade, not an improvement (question 295).

225. How do you make builds reproducible across developers' machines and CI runners? ​

The short answer

Make the inputs and the tools identical everywhere. Pin tool versions in files that live in the repository, such as .nvmrc or a tool-version file, and use wrappers such as the Gradle wrapper. Pin dependencies with lock files. Run the same commands locally and in CI, through one script or make target. Best of all, run the build inside a pinned container image, so every machine uses exactly the same environment.

The usual sources of "works on my machine" builds:

  1. Different tool versions. Node.js 18 on one laptop, 22 in CI.
  2. Unpinned dependencies. A library resolves to a newer version on a fresh install.
  3. Different commands. CI runs npm run build:prod, a developer runs npm run build.
  4. Different operating systems. File name case, line endings and paths differ (question 84).
  5. Hidden global tools. The build quietly relies on something installed globally on one machine.

The fixes:

  1. Pin the toolchain in the repository:
    • .nvmrc or .node-version for Node.js.
    • A .tool-versions file, read by version managers such as asdf or mise, for several tools at once.
    • The Gradle or Maven wrapper, which downloads the exact build tool version the project declares.
  2. Commit lock files, and install with exact-install commands such as npm ci (question 63).
  3. One set of commands. Put the steps in a Makefile or package.json scripts, and have CI call exactly those, so local and CI builds do the same thing.
  4. Build in a container. A pinned builder image, or a dev container definition that editors such as VS Code can open, gives every developer and every runner the same operating system, tools and versions.
  5. Remove time and machine details from outputs, as in question 85, for bit-for-bit reproducibility.

A good check: a new developer clones the repository, runs one documented command, and gets the same result as CI. If they need a page of setup notes first, something is not pinned.

226. How do you manage build-time configuration separately from environment-specific runtime configuration? ​

The short answer

Build-time configuration is anything that must be the same in every environment, such as compiler options, the app's version and which code is included, and it is fixed when the artifact is built. Runtime configuration is anything that differs per environment, such as database addresses, external service URLs, feature flags and secrets, and it is injected when the artifact is started, through environment variables, mounted configuration files or a configuration service. Keeping runtime values out of the artifact is what lets one artifact run everywhere.

Build-time configurationRuntime configuration
ExamplesCompiler flags, version number, bundled codeDatabase URL, API endpoints, log level, flags, secrets
Same in every environmentYesNo
Set whendocker buildContainer start
Lives inThe Dockerfile, build argumentsEnvironment variables, ConfigMaps, secret stores
Changing it meansA new buildA new deployment of the same image

This split is one of the ideas in the Twelve-Factor App, a well-known list of principles for building services. Its rule: configuration that varies between deploys belongs in the environment, not in the code.

The front-end trap. Single-page web apps are often built with values baked in, for example VITE_API_URL=https://staging-api.shop.com read at build time. That forces one build per environment. A common fix: the app fetches a small file at start-up, such as /config.json, and each environment serves its own version of that file, for example from a ConfigMap mounted into nginx.

ts
type RuntimeConfig = { apiUrl: string; checkoutEnabled: boolean }

export async function loadConfig(): Promise<RuntimeConfig> {
  const res = await fetch('/config.json', { cache: 'no-store' })   // (1)
  if (!res.ok) throw new Error(`config.json: HTTP ${res.status}`)  // (2)
  return (await res.json()) as RuntimeConfig                      // (3)
}
  1. Fetches the environment's configuration when the app starts, bypassing the browser cache so a configuration change takes effect at once.
  2. Fails loudly if the file is missing, instead of silently running with the wrong settings.
  3. Returns the settings. The same built files now work in every environment.

Secrets are runtime configuration too, and they need their own protected delivery (question 144).

227. How do you handle database migration scripts as part of a release pipeline? ​

The short answer

Keep migrations as numbered, versioned scripts in the repository, managed by a tool such as Flyway, Liquibase, Prisma or Alembic, which records which have run. Run them as their own pipeline step, or as a one-off job, before the new application version rolls out, never from every application copy at start-up. Write them to be backward compatible, so the old and new application versions both work during the rollout. Run them in staging first, watch their duration on large tables, and take a backup before anything destructive.

How a migration tool works:

  1. Each change is a file with an order number, such as V12__add_saved_cards.sql.
  2. The tool keeps a history table in the database listing which files have been applied.
  3. On each run it applies, in order, only the files not yet in the history.
  4. Running it twice does nothing the second time, so it is safe to retry.

Where it fits in the release:

text
build image  →  scan  →  deploy to staging:  run migrations  →  roll out app  →  verify
                         deploy to prod:     run migrations  →  roll out app  →  verify

Rules that keep it safe:

  1. A separate step or job, not inside the app's start-up. Suppose ten copies of the app start at once, and each tries to migrate. They race each other, and a slow migration delays every pod becoming ready (question 357).
  2. Backward compatible. During a rolling update, old and new versions of the app run at the same time against the same database. A migration that renames a column breaks the old version immediately. Use the expand and contract method: add first, switch over, remove later (question 287).
  3. Watch the clock. Adding an index or changing a column on a table with fifty million rows can lock the table for minutes. Use the database's online options, such as PostgreSQL's CREATE INDEX CONCURRENTLY, or online schema change tools for MySQL.
  4. Set timeouts. A migration waiting for a lock should give up after a few seconds, not block all traffic while it waits.
  5. Back up before destructive steps. Dropping a column or table cannot be undone without a backup.
  6. Staging first, with production-sized data where possible, so duration surprises appear there.

228. What happens if a database migration succeeds but the application deployment fails? ​

The short answer

If the migration was backward compatible, very little: the old application version still works with the new schema, so you roll back only the application and keep the schema, then fix and redeploy. If the migration was not backward compatible, the old version may now be broken too, so a rollback does not help. You must fix forward quickly. As a last resort, you restore the database, which risks losing recent data. This is exactly why migrations should always be backward compatible.

Case 1: a backward-compatible migration, such as adding a new nullable column.

  • The old app version ignores the new column, and keeps working.
  • Roll back the app to the previous image.
  • Leave the schema as it is. The next attempt will need it anyway.
  • Investigate why the new version failed, fix it, and deploy again.

Case 2: an incompatible migration, such as renaming or dropping a column the old version uses.

  • The old app version now fails too, because the column it expects is gone.
  • Rolling back the app is no longer a way out.
  • Options, best first:
    1. Fix forward: get the new version working, quickly.
    2. Undo the schema change with a reverse migration, if one exists and it is safe, for example renaming the column back.
    3. Restore the database from a backup. This loses every write since the backup, such as new orders, so it is a last resort.

How to never be in case 2:

  1. Expand and contract (question 287): the schema change that the new version needs is always one that the old version can live with.
  2. Separate the steps in time. Destructive changes, such as dropping the old column, happen in a later release, after the new version has proven itself in production.
  3. Check compatibility before deploying. Run the old application version's checks against the new schema in CI.

229. How can a pipeline test backward compatibility between a new service version and older clients? ​

The short answer

Check the new version against what existing clients actually expect. Consumer contract checks record each client's expectations and verify the new provider version against all of them, and a broker can answer "is it safe to deploy this version?". API schema comparison tools flag breaking changes in an OpenAPI or protobuf definition. You can also run the previous client version's checks against the new service. And roll out gradually, so old clients meet the new version on a small share of traffic first.

1. Consumer contract checks. Each client team publishes what it relies on: which endpoints, which fields, which status codes. The provider's pipeline verifies the new version against every published contract, including those of client versions still in use, such as last month's mobile app. The Pact tool's broker stores the contracts and offers a can-i-deploy check that answers yes or no for a specific version and environment.

2. Schema comparison. If the API is described in a file, a tool can compare the old and new descriptions and list breaking changes:

bash
oasdiff breaking api/openapi-main.yaml api/openapi-pr.yaml --fail-on ERR

Typical breaking changes it catches: a removed endpoint, a removed field in a response, a new required field in a request, a changed field type. For gRPC services described with protobuf, buf breaking does the same.

3. Old clients against the new server. Keep the previous client version's checks and run them against the new service in CI.

4. Design for compatibility:

  • Add, do not change. New optional fields are safe. Renaming or removing fields is not.
  • Version the API when a breaking change is truly needed, such as /v2/orders, and keep /v1/ working for a published period.
  • Tolerant readers. Clients should ignore fields they do not know, so providers can add fields freely.

5. Gradual rollout. A canary release exposes the new version to a small share of real clients first, including old ones you may not have checks for (question 282).

230. How do you verify a deployment after it completes rather than treating a successful deployment command as success? ​

The short answer

"The command succeeded" only means the cluster accepted the change. Verify that the change actually works. Wait for the rollout to finish with every new pod ready. Confirm the running version is really the new one. Run smoke checks against the real endpoint. Then watch error rate, latency and saturation for a bake period, compared with before the release, plus the logs and key business numbers. Roll back automatically if any of them gets worse.

What a successful kubectl apply really means: the API server stored the new desired state. Pods may not have started yet. They may be crashing. Traffic may still go to the old version.

A proper verification sequence:

bash
kubectl rollout status deployment/checkout-api -n shop-prod --timeout=5m       # (1)
curl -fsS https://shop.example.com/api/version | grep -q "$GITHUB_SHA"          # (2)
./smoke.sh https://shop.example.com                                             # (3)
  1. Waits until every new pod is created and ready and the old ones are gone. It fails if that does not happen within five minutes, or if the rollout is stuck (question 428).
  2. Asks the live service which commit it is running. This catches the classic "pipeline green, old version still running" problem (question 299).
  3. Smoke checks: a few critical paths, such as loading the home page, searching, and adding to the cart with a test account.

Then a bake period, often 10 to 30 minutes, watching:

SignalCompare againstRoll back if
Error rateThe hour before the releaseClearly higher
Latency, p95 and p99Before the releaseClearly slower
Saturation: CPU, memory, connectionsBefore the releaseNear limits
New error messages in logsPrevious versionNew errors appear
Business metric, such as orders per minuteSame time last weekUnusual drop

p95 and p99 are percentiles: 95% or 99% of requests are faster than this. They show the slow experience that averages hide.

Automate it. Progressive delivery tools run this comparison and the rollback by themselves (question 481).

231. What is a quality gate, and who should be able to bypass it? ​

The short answer

A quality gate is an automatic checkpoint with clear pass or fail rules, such as "no critical vulnerabilities, all checks passed, coverage on new code at least 80%", that a change must pass to move forward. Ideally nobody bypasses it in normal work. For real emergencies, there should be a documented break-glass process: a small set of named people, approval from someone other than the author, a recorded reason, an expiry, and a follow-up to fix the underlying issue.

What makes a good gate:

  1. Clear rules that everyone can read.
  2. Automatic, so it is applied the same way every time.
  3. Fast enough not to become the bottleneck.
  4. Few false alarms, so people trust it instead of resenting it.

Who may bypass, and how:

RuleWhy
Nobody, in normal workA gate that is routinely bypassed is not a gate
A short list of named people for emergenciesAccountability
Never the change's own author aloneSeparation of duties: one person cannot approve their own risk
A written reason, recorded automaticallyAudit, and learning
Time-limitedThe exception expires, it does not become permanent
A follow-up ticketThe cause, a flaky check or a false positive, gets fixed

Watch the bypass rate. If gates are bypassed every week, either the gates are wrong or the process is broken. Both are worth fixing. Question 302 walks through an emergency where a security gate blocks an urgent fix.

232. How should pipeline failure notifications identify the failed stage, commit, owner, and next diagnostic step? ​

The short answer

A good failure message answers, at a glance: what failed, which stage and step, on which commit and branch, who is responsible, what the error said, and what to do next. It links straight to the failing log line, to the pull request and to a runbook. It goes to the right people: the author for a pull request, the team for main, the on-call engineer for production. And it is sent when things change, first failure and recovery, not on every repeat.

An example message to a team chat channel:

text
❌ checkout-api · main · deploy-staging FAILED
Step:     smoke checks (3 of 4 passed)
Error:    POST /api/orders returned 500: "relation saved_cards does not exist"
Commit:   3f9c2ab "feat(checkout): saved cards" by asha · PR #482
Owner:    @shop/checkout-team  (on-call: ravi)
Likely:   the migration step was skipped; check the migrate job logs
Logs:     https://github.com/shop/checkout-api/actions/runs/9137/job/4412#step:6:88
Runbook:  https://wiki.shop/runbooks/failed-staging-deploy

What each line is for:

  1. The headline: service, branch, stage and result, readable in a notification preview.
  2. The step: no hunting through a 2,000-line log.
  3. The error excerpt: often enough to understand the failure without opening anything.
  4. The commit, the author and the pull request: who changed what.
  5. The owner: who should act, from the code owners file or the on-call schedule.
  6. A hint: a known cause or the most useful next check, if the pipeline can tell.
  7. Links: straight to the failing line and the runbook.

Sending it to the right place:

FailureWho gets told
Pull request checksThe author, on the pull request itself
Main branch buildThe owning team's channel
Staging deployThe team's channel
Production deployThe team and the on-call engineer, with a page if it affects users

Avoiding noise. Notify on the first failure and on recovery, not on every repeat. Mark failures caused by infrastructure, such as a runner losing its connection, differently from code failures, so people learn which ones need their attention (question 264).

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet