Skip to content

Interview PrepCI/CD

Release Strategies ​

280. What is the difference between blue-green, canary, rolling, and recreate deployment strategies? ​

The short answer

Recreate stops the old version completely, then starts the new one, so there is downtime. Rolling replaces copies of the old version with the new one a few at a time, so both run together for a while. Blue-green runs a complete second copy of the system with the new version, then switches all traffic to it at once, keeping the old copy ready for an instant switch back. Canary sends a small share of real traffic to the new version first, and increases it step by step while the numbers stay healthy.

Recreategap: no serviceRollingBlue-greenblue: v1, full setgreen: v2, full setrouterall traffic switchesat once; blue keptfor instant switch backCanaryv1: 95% of trafficv2: 5%, then 25%, 50%, 100%Blue boxes: old version. Green boxes: new version.
Four ways to switch versions. They differ in whether there is downtime, whether old and new run together, how much extra capacity they need, and how many users a bad release can reach before you notice.
RecreateRollingBlue-greenCanary
DowntimeYesNoNoNo
Old and new run togetherNoYes, brieflyNo, a clean switchYes, on purpose
Extra capacity neededNoneA littleDoubleA little
Users exposed to a bad releaseEveryoneGrowing shareEveryone, after the switchA small share first
RollbackRedeploy oldRoll back the replica setSwitch back, instantlySend traffic back to old
ComplexityLowestLow, Kubernetes defaultMediumHighest

Recreate suits cases where two versions must never run at the same time, such as a single job that holds an exclusive lock, and where a short outage is acceptable.

Rolling is Kubernetes' default for Deployments (question 350). It is cheap and smooth, but during the rollout both versions serve users, so they must be compatible with each other and with the database.

Blue-green gives a clean, instant switch and an instant rollback (question 281).

Canary limits the damage of a bad release to a small slice of users, and gives you real-traffic evidence before going further (questions 282 and 283).

281. When would you choose blue-green deployment rather than a rolling update? ​

The short answer

When you need an instant, all-at-once switch and an equally instant way back, or when the old and new versions must not serve users at the same time. Blue-green also lets you check the new version fully on real production infrastructure before any customer touches it. The costs are running a second full copy of the system during the switch, and the same database problems as any deployment, because both copies usually share one database.

Choose blue-green when:

  1. Two versions cannot serve together. For example, the new version changes a message format or a session format that the old version cannot read, so a rolling update's mixed period would break users.
  2. Rollback must be instant. Switching the router back to blue takes seconds, while rolling back a rolling update takes as long as the rollout itself.
  3. You want to check the new version on real infrastructure first. Green is fully deployed and can be checked, with smoke checks and internal users, before the switch.
  4. The platform does not support gradual rollouts well, for example some older systems behind a load balancer that can only point at one group of servers.

Stay with rolling updates when:

  • Old and new versions are compatible, which you should aim for anyway.
  • Doubling capacity, even briefly, is too expensive.
  • The service is one of many small services deploying many times a day, where rolling plus good readiness probes is simple and enough.

The database caveat. Blue and green usually share one database, so blue-green does not avoid the need for backward-compatible schema changes. If green runs a migration that blue cannot handle, switching back to blue does not work (question 286).

How the switch happens, for example in Kubernetes: two Deployments, checkout-api-blue and checkout-api-green, and one Service whose selector points at one of them. Changing the selector's version: blue to version: green moves all traffic at once.

282. How does a canary deployment limit the impact of a faulty release? ​

The short answer

By letting only a small share of real traffic reach the new version first, such as 5%. If the new version is faulty, only those users are affected, and only for as long as it takes to notice, which automated comparison makes a matter of minutes. Rolling back means sending that small share back to the old version. A bug that would have hit every customer hits a few, briefly.

A worked example. The shop serves 40,000 users an hour. A release contains a bug that breaks checkout for everyone who reaches it.

  • Without a canary: all 40,000 users an hour hit the bug until someone notices and rolls back. If that takes 30 minutes, 20,000 users have a broken checkout.
  • With a 5% canary and automated checks: 2,000 users an hour reach the new version. The checkout error rate on the canary jumps within minutes, the analysis halts the rollout after, say, 6 minutes, and traffic returns to the old version. About 200 users were affected.

Why it works:

  1. Real traffic is the most honest check. No staging environment has production's exact data, users and load (question 13).
  2. Comparison against a baseline. The canary's error rate and latency are compared with the old version's at the same moment, so ordinary ups and downs in traffic do not confuse the result.
  3. Small blast radius. Blast radius means how much is affected when something goes wrong. A canary keeps it small by design.

What canaries need:

  • Traffic splitting that can send a precise share to the new version: a service mesh, an ingress controller or Gateway API with weights, or a cloud load balancer (question 481).
  • Good metrics per version, so the canary's numbers can be separated from the old version's.
  • Enough traffic. With very little traffic, 5% may be only a handful of requests, too few to judge (question 283).

283. How do you decide when to increase canary traffic or automatically halt a rollout? ​

The short answer

Write the rules before the release. Move through fixed steps, such as 5%, 25%, 50% and 100%, with a waiting time at each step. At each step, compare the canary with the old version on error rate, latency, saturation and key business numbers. Continue only if every metric stays within its threshold and there is enough traffic to be confident. Halt and roll back automatically as soon as any threshold is crossed.

A typical plan:

StepCanary shareWaitMust hold
15%10 minutesError rate no worse than old + 0.5 points; p99 latency within 10%
225%10 minutesSame, plus no rise in restarts
350%15 minutesSame, plus checkout success rate unchanged
4100%DoneOld version scaled down after a final bake

Compare, do not just measure. "Error rate below 1%" can fail for reasons that have nothing to do with the release, such as a partner outage that affects both versions. "Canary error rate no higher than the old version's" asks the right question.

Be sure there is enough evidence. If the canary received 40 requests in ten minutes, one error looks like a 2.5% error rate. Set a minimum number of requests before a step can pass, and lengthen waiting times for low-traffic services. Some analysis tools use statistical tests to decide whether a difference is real.

Halt rules, which act immediately:

  • Error rate or latency over the threshold.
  • Pods of the new version crashing or failing readiness.
  • A sharp drop in a business metric, such as orders per minute.

Automate the decision. Progressive delivery tools run these checks against your monitoring data and promote or abort by themselves (question 481). A person watching dashboards is slower, less consistent, and not always watching.

284. What health signals should a release controller evaluate before considering a rollout successful? ​

The short answer

The four golden signals for the new version, compared with the old one: error rate, latency at the slow end such as p95 and p99, traffic, and saturation such as CPU, memory and connection pools. Plus the platform's view of the pods: readiness, restarts and memory kills. Plus the health of what the service calls. Plus business numbers such as checkout success. Plus synthetic checks that walk through key user journeys. All measured against a baseline, not just against fixed numbers.

GroupSignalsWhy
Errors5xx rate, failed requests, exceptions per minuteThe most direct sign of breakage
Latencyp50, p95, p99 response timeSlowness often appears before errors
TrafficRequests per second reaching the new versionConfirms it is really serving, and gives the sample size
SaturationCPU, memory, connection pools, queue depthShows problems that will appear under more load
Pod healthReadiness, restarts, OOM killsThe platform's own verdict
DependenciesError rate and latency of downstream callsA new version may hurt the services it calls
BusinessCheckout success, sign-ups, payment approval rateCatches bugs that return 200 but do the wrong thing
SyntheticScripted user journeysCovers paths real traffic may not exercise during the rollout

The first four are Google's four golden signals, from its site reliability engineering book (question 629).

Two subtle points:

  1. "200 OK" is not the same as correct. A release that shows an empty product list returns successful responses and breaks the shop. Business metrics and synthetic checks catch what error rates miss.
  2. Look at the service's neighbours. A new version that makes twice as many database queries may look fine itself while slowing the database for every other service.

285. What is a feature flag, and how is it different from deploying a new application version? ​

The short answer

A feature flag is a switch, read by the code at run time, that turns a piece of behaviour on or off without changing the deployed code. Deploying puts new code into production. A flag decides who actually sees the new behaviour, and when. With flags, you can deploy code switched off, turn it on for staff, then 10% of users, then everyone, and switch it off in seconds if it misbehaves, all without another deployment.

ts
import { flags } from './flags'

export async function checkout(user: User, cart: Cart): Promise<Order> {
  if (await flags.isEnabled('saved-cards', { userId: user.id, country: user.country })) {   // (1)
    return checkoutWithSavedCards(user, cart)                                               // (2)
  }
  return classicCheckout(user, cart)                                                        // (3)
}
  1. Asks the flag service whether saved-cards is on for this user. The answer can depend on who the user is, their country, a percentage rollout or anything the flag's rules say.
  2. The new path, for users the flag is on for.
  3. The old path, still there, for everyone else.

Deploying versus releasing:

DeployRelease through a flag
What changesThe code running in productionWho experiences new behaviour
HowThe pipelineChanging a flag's rules
How fastMinutesSeconds
UndoRoll back the deploymentTurn the flag off

Kinds of flags:

  1. Release flags: hide unfinished work, so it can be merged and deployed early (question 189).
  2. Experiment flags: show different versions to different users and measure the results (question 482).
  3. Operational flags, or kill switches: turn off an expensive or risky feature during an incident.
  4. Permission flags: turn features on for certain customers or plans.

The cost of flags. Every flag doubles the paths through the code. Old flags that stay forever make code hard to read and hard to check. Treat flags as temporary: give each an owner and an expiry, and delete it once the feature is fully on. OpenFeature is an open standard interface for flags, and LaunchDarkly, Unleash and Flagsmith are common tools.

286. How do you roll back an application when the database schema has already changed? ​

The short answer

It depends on whether the schema change is backward compatible. If the previous application version still works with the new schema, roll back only the application and leave the schema alone. If it does not, rolling back the application alone breaks it too, so you must either fix forward, or undo the schema change, which can lose data written since. The real answer is to plan for this before deploying, by only ever making schema changes the previous version can live with.

Case 1: compatible change, the safe case. The new release added a column, saved_card_id, which can be empty. The old version simply ignores it. Roll back the application; the extra column does no harm. Data the new version wrote into it stays there, ready for the next attempt.

Case 2: incompatible change, the dangerous case. The new release renamed name to full_name. The old version still queries name, which no longer exists. Rolling back the application gives you an old version that fails on every request. Your options are now all bad:

  1. Fix forward, quickly, under pressure.
  2. Reverse the migration, renaming the column back, if a tested reverse migration exists, and if no data has been written that only fits the new shape.
  3. Restore from a backup, losing every order and change made since.

Rules that keep you in case 1:

  1. Every schema change must work with the previous application version. That is the expand and contract approach (question 287).
  2. Separate the schema change from the code that depends on it, in time: deploy the schema first, the code that uses it next, the clean-up of old columns much later.
  3. Decide the rollback plan before deploying. "If this fails, we roll back the app and keep the column" is a plan. "We will figure it out" is not.
  4. Never write destructive "down" migrations in a panic. Dropping a column during an incident is how incidents become data loss.

287. What is the expand-and-contract approach for backward-compatible database migrations? ​

The short answer

It splits a breaking schema change into several safe steps spread over separate releases. First expand: add the new structure alongside the old one, without removing anything. Then migrate: write to both, copy existing data across, and move reads to the new structure. Finally contract: once nothing uses the old structure any more, remove it. At every step, both the current and the previous application versions work, so any single release can be rolled back.

Example: renaming the column name to full_name in the customers table.

① expandadd full_name, emptyapp writes bothapp still reads name② backfillcopy name intofull_name for old rows,in small batches③ switchapp reads full_nameand writes onlyfull_name④ contractdrop the oldname column,weeks laterAfter every step, the previous release still works,so every step can be rolled back on its own.
A rename in four safe moves. Nothing is ever removed while something still uses it. The destructive step comes last, long after the new version has proven itself.

Release 1, expand.

sql
ALTER TABLE customers ADD COLUMN full_name text;   -- nullable, so instant and harmless

The application now writes the customer's name into both columns, and still reads name. Rolling this release back is safe: the old version ignores full_name.

Release 2, backfill. A one-off job copies existing values in small batches, so it never locks the table for long:

sql
UPDATE customers SET full_name = name
WHERE id IN (SELECT id FROM customers WHERE full_name IS NULL LIMIT 5000);

It repeats until no rows are left.

Release 3, switch. The application reads and writes only full_name. Release 2's version still works, because it also wrote full_name.

Release 4, contract. Weeks later, once release 3 has been stable and nothing reads name:

sql
ALTER TABLE customers DROP COLUMN name;

Why so many steps? Because during any rolling deployment, two application versions run at once against one database. Each step only ever asks the schema to support two adjacent versions, which is always possible. The alternative, one big rename, breaks whichever version is not the newest.

288. How do you handle a release that is partly deployed across many services? ​

The short answer

First stop the rollout, so the situation does not change while you look. Then check compatibility: if each service works with the others' old and new versions, a partial state is safe, and you can either finish or roll back calmly. If not, restore a consistent state quickly, usually by rolling back the services that already updated, in reverse order. The lasting fix is to design every service to work with the previous version of its neighbours, so that partial states are always safe.

In the moment:

  1. Pause. Stop the pipeline, or stop the GitOps tool from syncing further changes.
  2. Map the state. Which services run the new version, which the old?
  3. Check compatibility. Does web version 2 work with checkout-api version 1? If the APIs are backward compatible, yes.
  4. Choose:
    • Compatible: finish the rollout once the cause of the stop is understood, or roll back the ones that updated. Either is safe.
    • Not compatible: customers may already see errors. Roll back the updated services, in the reverse order they were deployed, or push forward the rest if that is faster and safe.

Designing so that partial deployments are always safe:

  1. Backward-compatible APIs. Providers add fields and endpoints; they do not remove or rename them without a deprecation period (question 229).
  2. Deploy providers before consumers. The new checkout-api field must exist before web starts using it.
  3. Feature flags across services. Deploy all services with the new behaviour switched off, then turn the flag on once every service is updated (question 285).
  4. Independent deployability. If several services must always be deployed together, in a fixed order, at the same moment, they are really one system split across several pieces, often called a distributed monolith. That coupling is the real problem, and worth removing.

For the rare release that truly spans services, coordinate it explicitly: a release plan with the order, a check after each step, and a written rollback order.

289. How do you prevent two releases from overwriting one another or creating an unsafe deployment order? ​

The short answer

Serialize deployments per environment with locks or queues, so only one release at a time changes an environment. Refuse to deploy anything older than what is running. Keep one source of truth for each environment's desired state, ideally a Git repository with GitOps, so concurrent changes become ordered commits. Encode required ordering between services explicitly in the pipeline. And block manual changes that go around the process.

  1. One deployment at a time per environment. Concurrency groups, resource groups or environment locks (question 222).
  2. Newer only. A check that the commit being deployed comes after the one already running (question 222).
  3. One source of truth. With GitOps, each environment's desired state lives in Git. Two releases become two commits, in a definite order, and conflicting edits must be merged by a person before anything happens (question 507).
  4. Merge queues. Changes to main are tested and merged one by one in a fixed order, so releases built from main come out in order too.
  5. Explicit ordering between services. If checkout-api must be deployed before web for a release, the pipeline encodes that with dependencies, rather than relying on luck.
  6. No side doors. If engineers can kubectl apply changes by hand, the pipeline's ordering means nothing. Remove routine write access to production, and let the GitOps tool undo unexpected manual changes.

290. How should manual approval be used in production delivery without creating a slow bottleneck? ​

The short answer

Use approvals only where a human adds something a machine cannot, and make them quick and informed. Require them for production and for risky changes, not for every environment or every change. Show approvers a clear summary: what changed, which checks passed, the risk. Let a policy approve low-risk changes automatically. Keep a rota of approvers with a response time, so nobody waits on one busy person. And keep moving checks from humans to automation.

Why approvals become bottlenecks:

  • One or two people approve everything, and they are in meetings.
  • Approvers do not know what they are approving, so they either rubber-stamp, which adds delay but no safety, or investigate from scratch, which takes hours.
  • Every small change waits, so people batch changes together, making each release bigger and riskier.

What works:

  1. Approve where it matters. Production only, and often only for changes marked risky, such as database migrations or changes to payment code. Routine changes that passed every check can flow automatically.
  2. Make approvals informed. The approval request shows the commits, linked pull requests and their reviewers, check and scan results, and the planned rollout. Approving should take a minute.
  3. Policy-based auto-approval. "If every check passed, no migration, no change under payments/, and outside a freeze window: approve automatically." The rule itself is reviewed once, instead of each change.
  4. A rota with a response time, such as "approvals answered within 30 minutes during working hours", with several approvers per team.
  5. Approve release candidates, not every commit, if you batch releases at all.
  6. Separation of duties still holds: the author never approves their own change (question 231).

Watch the waiting time. Track how long deployments wait for approval. If it grows, the process needs fixing, not the people.

291. What is the difference between continuous delivery, a manual promotion gate, and a fully automatic production deployment? ​

The short answer

Continuous delivery is the overall practice: every change is built, checked and kept ready to release. A manual promotion gate is one way to finish that journey: a person decides when a ready change moves to production. A fully automatic production deployment is the other way, also called continuous deployment: the pipeline moves every passing change to production without waiting for anyone.

Continuous delivery with a manual gateContinuous deployment
Every change is releasableYesYes
Who decides "now"A person, by approvingThe pipeline, when checks pass
Time from merge to productionWaits for approvalMinutes
What makes it safeChecks plus human judgmentChecks, canaries, automatic rollback
Common inRegulated industries, high-risk systemsTeams with strong automated checks

Both rest on the same foundation: one artifact built once, automated deployment, and strong checks (question 172). The gate is a policy choice added at the end, not a different kind of pipeline. Many organisations mix them: automatic deployment for most services, and a manual gate for a few, such as anything touching payments (question 174).

292. What are deployment freeze windows, change approvals, and audit trails intended to control? ​

The short answer

Freeze windows stop non-urgent changes during periods when a failure would hurt most, or when fewer people are around to fix it, such as a big sale or the holidays. Change approvals make sure risky changes are seen and agreed by the right people before they happen. Audit trails record who changed what, when, why, and who approved it, which compliance rules require and every incident investigation needs. All three exist to control risk and to prove control.

Freeze windows:

  • Why: a broken checkout during the year's biggest sale costs far more than on an ordinary Tuesday. And during holidays, the people who could fix it may be away.
  • The catch: changes pile up during the freeze, and the first release afterwards is huge and risky. Keep freezes short, allow emergency fixes through an expedited path, and avoid a "big bang" release right after.

Change approvals:

  • Traditional companies use a change advisory board, a meeting where proposed changes are reviewed, often weekly. It is slow, and studies by the DORA research group found that heavyweight external approval did not reduce failure rates.
  • Modern practice keeps the intent and changes the method: peer review on every change, automated checks, automatic approval for low-risk changes, and human approval only where it adds value (question 290).

Audit trails:

  • What they record: every deployment, its artifact, its commit and pull request, its approvals, who triggered it, and the result (question 196).
  • Why: regulations and standards such as SOX for financial reporting, PCI DSS for card data and ISO 27001 for information security require proof that changes to production are controlled. During an incident, the trail answers "what changed?" in seconds.
  • Best form: produced automatically by the pipeline and the platform, not written by hand afterwards.

293. How do you implement rollback when a deployment is technically successful but key business metrics deteriorate? ​

The short answer

Decide in advance which business metrics guard each release, such as checkout success rate or orders per minute. Also decide what counts as a real drop compared with a baseline, such as the same hour last week, or the old version during a canary. Watch them during the canary and the bake period, ideally automatically. When they cross the line, roll back first, by switching a feature flag off or reverting the deployment, and investigate afterwards. Give the on-call engineer clear authority to do that without a meeting.

Why this needs special care: every technical signal can be green. Pods are ready, error rates are flat, latency is fine. But the new checkout page has a button that does not work on one popular phone, and orders drop 15%.

1. Choose guardrail metrics per release. For the shop:

  • Checkout success rate.
  • Orders per minute.
  • Payment approval rate.
  • Add-to-cart rate.

2. Compare against the right baseline. Business numbers move with the time of day and the day of the week. Compare against:

  • The old version, at the same moment, during a canary. This is the best comparison.
  • The same hour on the same weekday last week, when the change goes to everyone.

3. Automate where possible. Progressive delivery tools can include business metrics in their analysis, just like error rates (question 481).

4. Roll back fast, by the fastest route.

  • A feature flag off, in seconds, if the change is behind a flag (question 285).
  • A deployment rollback, in minutes, otherwise.

5. Give people authority. The on-call engineer should be able to say "orders are down 15% since the release, rolling back" without asking permission. The cost of a mistaken rollback is a delayed feature. The cost of a slow rollback is lost revenue and trust.

6. Then investigate, with the evidence: which users, which pages, which devices. Roll forward again with the fix, behind a canary.

294. What are the DORA metrics, and what can deployment frequency, lead time, change failure rate, and recovery time tell you? ​

The short answer

The DORA metrics come from DevOps Research and Assessment, a research programme that studied thousands of teams and found a small set of numbers that track software delivery performance. Deployment frequency and lead time for changes measure speed. Change failure rate and failed deployment recovery time measure stability. The research's key finding is that the best teams are strong at both at once: speed and stability go together. Recent DORA reports add a fifth metric, rework rate.

MetricQuestion it answersHow to measure
Deployment frequencyHow often do we deliver?Production deployments per day or week
Lead time for changesHow fast does a change reach users?Time from commit to running in production, median
Change failure rateHow often does a deployment hurt?Share of deployments needing a fix, rollback or hotfix
Failed deployment recovery timeHow fast do we recover from a bad deployment?Time from the failure to service restored
Rework rateHow much unplanned work do deployments cause?Share of deployments that are unplanned fixes for production problems

What each tells you:

  1. Deployment frequency reflects batch size. Teams that deploy often make each change small, which makes each one lower-risk.
  2. Lead time shows how much waiting is in the process: reviews, queues, approvals, environments. It usually reveals that most time is spent waiting, not working (question 8).
  3. Change failure rate shows how good the checks and the release methods are.
  4. Recovery time shows how good monitoring, rollback and incident response are. Recovery time used to be called mean time to recovery, MTTR.

Speed and stability together. The surprising result of the research is that teams do not have to trade one for the other. Small, frequent, well-checked changes are both faster and safer than large, rare ones.

How to use them well:

  • Measure automatically, from the pipeline and incident data, not by survey.
  • Compare a team with itself over time, not teams with each other. Different systems have different constraints.
  • Do not set them as targets for individuals. A team told to "deploy more" can split changes meaninglessly. Use them to find bottlenecks, then fix the bottleneck (question 212).

295. How do you improve pipeline duration without weakening test coverage or security controls? ​

The short answer

Make the same work faster instead of doing less of it. Cache, parallelise and split. Run only what a change affects on pull requests, with the full suite still running on main before release. Make slow scans incremental or move them off the critical path while keeping them as gates before promotion. Buy faster machines where work is compute-bound. And measure the critical path first, so effort goes where the time really is.

TechniqueFasterSafety kept because
CachingDependencies, layers, build outputsCaches verified by hashes (question 223)
Parallel jobs and shardsTotal time drops to the longest jobSame checks, run side by side
Affected-only checks on pull requestsLess work per pull requestFull suite still runs after merge
Incremental scansScan only changed code and new dependenciesFull scans still run on a schedule
Slow scans after the build, gating promotionPull request feedback is quickerPromotion is blocked until they pass
Faster runnersCompute-bound stepsSame work
Fixing slow checksThe checks themselvesSame coverage, better written
Removing duplicate workBuilding once, reusing artifactsNothing removed but repetition

The key distinction is between feedback and gating. A developer needs fast feedback on a pull request. Production needs complete gating before promotion. These do not have to be the same pipeline run. A fast pull request pipeline, plus a thorough post-merge pipeline that must pass before anything is promoted, gives both.

What not to do: delete checks, lower thresholds, or mark security scans as allowed to fail. Those make the number smaller and the risk bigger (question 203).

296. How do you measure pipeline reliability, queue time, failure rate, mean time to recovery, and flaky-test rate? ​

The short answer

Collect every pipeline run's events, such as start, finish, result, job and step durations, the runner, and any retries, through the CI platform's API or webhooks, into a store you can query. Then compute four numbers. Queue time is the gap between a job being created and starting. Failure rate is failed runs over all runs, split into infrastructure and code causes. Recovery time is how long main stays red. Flaky rate is the share of checks that failed and then passed on the same commit.

MetricDefinitionWatch
Queue timeJob start time minus job creation timep50 and p95
Pipeline failure rateFailed runs ÷ total runs, per pipelineSplit by cause
Infrastructure failure rateRuns failed by runner, network or platform errors ÷ totalShould be near zero
Main branch recovery timeTime from main turning red to green againMedian per week
Flaky rateCheck runs that failed then passed on the same commit ÷ all check runsTrend
DurationRun time, per pipelinep50 and p90, per week

Getting the data. CI platforms emit events: GitHub sends webhooks for workflow runs and jobs, GitLab and Azure Pipelines have similar APIs. A small collector stores them in a database, or the platform's own analytics are used where they are good enough.

Classifying failures. Tag each failure:

  • Infrastructure: known patterns such as "runner lost communication", "no space left on device", registry timeouts.
  • Flaky: the same check passed on a retry of the same commit.
  • Real: everything else.

Even a rough classification tells you whether a bad week was the platform's fault or the code's.

Recovery time for main. When main's pipeline goes red, every team that merges is blocked or at risk. Measuring how long main stays red, and alerting when it is red for more than, say, 30 minutes, keeps the habit from question 171 alive.

297. What is the difference between deployment frequency and release frequency? ​

The short answer

Deployment frequency is how often new code goes into production. Release frequency is how often new behaviour becomes visible to users. With feature flags, the two separate: a team can deploy twenty times a day with features switched off, and release a finished feature to customers once a month by turning its flag on. Deployment is a technical event. Release is a business decision.

An example. The saved-cards feature takes three weeks to build.

  • It is merged and deployed in 30 small changes over those weeks, behind a flag that is off. Customers see nothing. That is 30 deployments.
  • On launch day, marketing announces it and the flag is turned on for everyone. That is one release.

Why separating them is useful:

  1. Small, safe deployments, without exposing half-built features.
  2. Releases on the business's schedule, such as with a marketing campaign, without a risky big deployment on the same day.
  3. Gradual releases: staff first, then 5% of customers, then everyone (question 285).
  4. Dark launches: new code runs in production, perhaps processing real requests in the background, before anyone sees its results (question 482).

For metrics: DORA's deployment frequency counts deployments. If you want to measure how fast value reaches customers, also measure the time from "feature finished" to "feature released".

298. How do you handle an urgent production fix while maintaining traceability and required quality checks? ​

The short answer

Use the normal pipeline, made faster, never a side door. Make a small, focused change in a pull request with a fast reviewer. Run every mandatory gate: the build, the fast checks and the security scans. If a slow, non-critical suite must be skipped, record it as a time-limited exception with an approver. Roll out with a shortened canary, watch closely, communicate throughout, and follow up afterwards with the skipped checks and a review of why the fix was needed.

The expedited path:

  1. Small change. Fix only the problem. An urgent fix is the worst moment to refactor.
  2. Pull request with a quick review, from an on-call reviewer who is expecting it.
  3. The mandatory gates run: build, unit and integration checks, security scans. Most of these take minutes.
  4. If a long suite must be deferred, such as a 40-minute end-to-end suite, the deferral is approved by a named person, recorded with a reason, and the suite runs right after the deployment (question 231).
  5. Priority in the queue, so the fix does not wait behind routine builds.
  6. A shortened canary: still a small share first, for a shorter time, with someone watching.
  7. Communication: incident channel updates, and a status page update if customers are affected.
  8. Follow-up: run anything deferred, write the incident review, and add a check that would have caught the original bug (question 455).

What not to do: log in to a server and change files by hand, or build an image on a laptop and push it straight to production. It feels faster. It is untraceable, it is unchecked, and the next normal deployment quietly removes it, bringing the bug back. The same pipeline, faster, is both safer and almost as quick.

481. What is progressive delivery, and how do tools like Argo Rollouts or Flagger automate canaries? ​

The short answer

Progressive delivery means releasing a change to users gradually, while automatically checking real metrics at each step and rolling back by itself if something looks wrong. Argo Rollouts and Flagger bring this to Kubernetes. You describe the steps, such as 10% then 50% then 100% with pauses, and the checks, written as queries against your monitoring system. The tool then shifts traffic step by step through an ingress, a gateway or a service mesh, runs the checks, and either promotes or aborts.

An Argo Rollouts example. A Rollout replaces a Deployment, with the same pod template plus a strategy:

yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout-api }
  template:
    metadata:
      labels: { app: checkout-api }
    spec:
      containers:
        - name: api
          image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
  strategy:
    canary:
      canaryService: checkout-api-canary       # (1)
      stableService: checkout-api-stable
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:            # (2)
            httpRoute: checkout-api
            namespace: shop-prod
      steps:
        - setWeight: 10                        # (3)
        - analysis:
            templates: [{ templateName: success-rate }]
        - setWeight: 50
        - pause: { duration: 10m }
        - analysis:
            templates: [{ templateName: success-rate }]
  1. Two Services, one for the canary pods and one for the stable pods, so traffic can be split precisely between them.
  2. Traffic is split by changing weights on a Gateway API route (question 370). Ingress controllers and service meshes are also supported.
  3. Send 10% of traffic to the new version, then run the analysis.

The analysis, a query against Prometheus:

yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      successCondition: result[0] >= 0.99            # (1)
      failureLimit: 1                                # (2)
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="checkout-api",role="canary",code!~"5.."}[1m]))
            /
            sum(rate(http_requests_total{app="checkout-api",role="canary"}[1m]))
  1. Each measurement must show at least 99% of canary requests succeeding.
  2. One failed measurement aborts the rollout. Argo Rollouts then sends all traffic back to the stable version and scales the canary down.

Flagger, from the Flux project, does the same job differently: you keep your normal Deployment, and Flagger creates and manages the canary copy and the traffic split around it.

Why teams adopt these tools: they turn the canary rules from question 283 into code that runs the same way every time, at any hour, without someone staring at a dashboard.

482. What is the difference between A/B testing, canary releases, and shadow traffic? ​

The short answer

A canary release is about safety: a small random share of traffic tries a new version, and you check that it is healthy before sending everyone. A/B testing is about learning: users are split into groups that consistently see different variants, and you measure which variant gives better business results. Shadow traffic, also called mirroring, copies real requests to a new version in the background, so you can see how it behaves under real load while users only ever receive the old version's answers.

CanaryA/B testShadow traffic
GoalCatch bad releases earlyLearn which variant works betterTest with real traffic, at no risk to users
Who sees the new versionA small, random shareA defined group, consistentlyNobody; responses are discarded
Same user, same variant every timeNot necessarilyYes, by designNot applicable
MeasuredErrors, latency, saturationConversion, engagement, revenueErrors, latency, correctness of results
DurationMinutes to hoursDays to weeksAs long as needed
Usually done withDeployment and traffic toolsFeature flags or experiment platformsA mesh, proxy or gateway that can mirror

A/B testing in more depth. Users are assigned to variant A or B, usually by a hash of their user ID, so each person always sees the same one. The experiment runs until enough data has been collected for a statistically meaningful answer, often days. It is a product decision tool, not a release safety tool.

Shadow traffic in more depth. A proxy or service mesh sends each request to the current version, whose answer goes to the user, and also sends a copy to the new version, whose answer is thrown away. You compare the two versions' errors, speed and even outputs.

The big caution with shadowing: side effects. If the shadow version charges cards, sends emails or writes orders, real customers are charged twice. Shadowing is safe for read-only requests, or when the shadow version's writes go to a separate, throwaway store.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet