Skip to content

Interview PrepCI/CD

Shared Pipelines at Scale ​

258. What are YAML anchors, templates, reusable workflows, and shared libraries intended to solve? ​

The short answer

They all solve copy and paste. YAML anchors remove repetition inside one file. Templates, such as GitLab's include and extends or Azure Pipelines templates, reuse pieces across files and repositories. Reusable workflows and composite actions do the same in GitHub Actions. Jenkins shared libraries do it with code. The goal in every case is that a pipeline pattern is written once, fixed once, and improved once, for everyone who uses it.

ToolPlatformReuse withinReuse across repositories
YAML anchorsMany YAML-based tools, such as GitLabOne fileNo
extendsGitLabOne file or included filesWith include
includeGitLabNot applicableYes, from other projects or URLs
Templates and extendsAzure PipelinesYesYes, from a template repository
Composite actionsGitHub ActionsYesYes
Reusable workflowsGitHub ActionsYesYes
Shared librariesJenkinsYesYes

YAML anchors, inside one file:

yaml
.node-defaults: &node-defaults            # (1)
  image: node:20-slim
  before_script: [npm ci]
  tags: [linux]

lint:
  <<: *node-defaults                      # (2)
  script: [npm run lint]

unit:
  <<: *node-defaults
  script: [npm run unit]
  1. &node-defaults gives this block a name, an anchor. In GitLab, a job name starting with a dot is a hidden job that never runs on its own.
  2. <<: *node-defaults copies the anchored block into this job, and the job adds its own script.

GitLab include across repositories:

yaml
include:
  - project: shop/pipeline-templates
    ref: v3.2.0
    file: /templates/node-service.yml

This pulls a template from another project, pinned to a tag, so the template's owners can improve it without breaking everyone at once.

Questions 238, 243 and 247 show the Jenkins and GitHub Actions equivalents. Question 259 covers using them across many repositories.

259. How do you avoid duplicating almost identical pipeline definitions across many repositories? ​

The short answer

Put the shared pipeline in one central place, as a versioned template, reusable workflow or shared library, with inputs for the few things that really differ, such as the service name and its language. Each repository keeps only a tiny file that calls it with a pinned version. New repositories start from a generator that already includes that file. A bot opens pull requests when a new template version is released, so updates spread without anyone copying anything.

shop/pipeline-templatesnode-service.ymltags v3.1, v3.2build, scan, sign, deploycheckout-apiuses: node-service@v3.2catalog-apiuses: node-service@v3.2notification-workeruses: node-service@v3.1, bot PR open
Write once, pin everywhere. Each service calls a version of the central pipeline. Upgrading is an ordinary pull request in each repository, so every team sees and checks the change.

The whole pipeline file in a service repository:

yaml
# .github/workflows/ci.yml in shop/checkout-api
name: ci
on:
  pull_request:
  push:
    branches: [main]
jobs:
  pipeline:
    uses: shop/pipeline-templates/.github/workflows/node-service.yml@v3.2.0
    with:
      service: checkout-api
      node-version: '20'
    secrets: inherit

What makes this work well:

  1. Inputs only for real differences. Service name, language version, maybe a flag for an extra step. If every repository needs ten inputs, the template is trying to be too many things.
  2. Pinned versions. Repositories use @v3.2.0, or at least @v3, never a moving branch, so template changes never surprise anyone (question 263).
  3. Automated upgrades. Renovate or Dependabot can update template versions in every repository, which turns a hundred manual updates into a hundred reviewed, automatically checked pull requests.
  4. A generator for new services. A template repository, or a portal such as Backstage, creates new services with the standard pipeline already wired in, so new duplication never starts.
  5. Conventions over configuration. If every service puts its Dockerfile in the same place and uses the same script names, the template needs almost no inputs.

260. What are the trade-offs between a centralized pipeline template and team-owned pipelines? ​

The short answer

A central template gives consistency, built-in security controls, less duplication and easier audits, but it can become a bottleneck, fit some teams badly, and break everyone at once when it has a bug. Team-owned pipelines give flexibility and ownership, but drift apart, repeat the same work, and protect security unevenly. The usual answer is a middle path: a central core that enforces the non-negotiable steps, with clear places where teams can add their own.

Central templateTeam-owned pipelines
ConsistencyHighLow
Security controlsEnforced for everyoneUp to each team
Speed of a fleet-wide fixOne releaseDozens of pull requests
FlexibilityLimitedComplete
OwnershipPlatform teamEach team
Risk of one bad changeBreaks everyoneBreaks one team
BottleneckChanges queue behind one teamNone

The middle path that works for most companies:

  1. A required core. Security scans, image signing, provenance and the deployment method are in the central template, and cannot be removed. Azure Pipelines' extends templates and GitLab's compliance pipelines are designed for this: the template defines the structure, and the team fills in allowed sections.
  2. Extension points. Inputs, or named "before" and "after" hooks, where teams add their own steps, such as extra checks.
  3. An escape hatch, with rules. A team with a real need may own its pipeline, as long as it still runs the required controls, which are checked independently, for example by the cluster refusing unsigned images (question 147).
  4. Inner source. Teams can propose changes to the template through pull requests, so the platform team is not the only one who can improve it.

A sign the balance is wrong: if teams regularly copy the template to change one line, the template is too rigid. If every team's pipeline is different and some skip scanning, it is too loose.

261. How do you test a pipeline definition before allowing it to change production deployments? ​

The short answer

Treat pipeline changes like code changes. Lint and validate the file automatically. Require review from the pipeline's code owners. Run the changed pipeline on a branch where production steps cannot run, using dry runs or a sandbox environment. For shared templates, run the new version against a set of real consumer repositories before tagging it, then roll it out gradually by version.

1. Static checks on every change.

bash
actionlint .github/workflows/*.yml          # (1)
  1. actionlint checks GitHub Actions files for syntax errors, unknown properties, wrong expression types, and some security problems, such as script injection. GitLab has a built-in CI configuration check in its editor and API, and Azure Pipelines validates YAML when you run or preview it.

2. Review by the right people. A CODEOWNERS rule makes the platform or security team approve changes to pipeline files and templates (question 186).

3. Run it where it cannot hurt production.

  • Production steps only run from main, with environment protection, so a branch run of the changed pipeline simply cannot deploy there (question 246).
  • Dry runs for deployment steps: kubectl diff, helm upgrade --dry-run, terraform plan. They show what would change, without changing it.
  • A sandbox environment that mirrors production's setup, for the full deploy path.

4. For shared templates: test on real consumers. Before tagging a new template version, run it against a handful of representative service repositories, using the release candidate tag. Some teams keep a set of small "fixture" repositories purely for this. Then tag the release, and let repositories adopt it gradually (question 263).

5. Local tools, with limits. Tools such as act can run GitHub Actions workflows locally inside containers. They are handy for quick checks, but they do not reproduce hosted runners, permissions or secrets exactly, so they never replace a real run.

262. How would you debug a pipeline that succeeds manually but fails when triggered by a pull request? ​

The short answer

The two runs see different contexts, so compare them. A pull request run usually builds a temporary merge of the branch into its target, not the branch alone. It gets a read-only token and, from forks, no secrets. Its event variables are different, such as the reference name. It may skip or add jobs through conditions and path filters, use another runner group, or have too shallow a clone to compare changes. Print the context in both runs and look at the first step whose inputs differ.

The usual differences:

  1. Different code. For pull requests, GitHub and GitLab often check out a merge commit: your branch merged into the latest target. Your branch alone may build fine while its combination with main does not. Reproduce locally with git merge origin/main on your branch.
  2. Missing secrets. Pull requests from forks get none, and any step that needs one fails, often with a confusing error such as an empty login (question 250).
  3. Fewer permissions. The built-in token is read-only, so steps that write, such as commenting on the pull request, pushing a tag or publishing a package, are refused.
  4. Different variables. On GitHub, github.ref is refs/pull/482/merge for a pull request, not a branch name. A script that expects refs/heads/... breaks.
  5. Conditions. A job with if: github.event_name == 'push' runs on manual runs but not on pull requests, and a later job that needs it then skips or fails.
  6. Shallow clones. Pull request checkouts often fetch only one commit. A step that compares with main, such as changed-files detection or version calculation, finds no history.
  7. Other runners. Pull requests may run on a different, more restricted runner group, with other tools or network access.

How to compare quickly:

yaml
- name: show context
  run: |
    echo "event=$GITHUB_EVENT_NAME ref=$GITHUB_REF sha=$GITHUB_SHA head=$GITHUB_HEAD_REF base=$GITHUB_BASE_REF"
    git log --oneline -3

Run it in both kinds of run, and the difference is usually obvious. Then fix the step to work in both contexts, instead of making pull requests behave like manual runs, because the pull request's restrictions are there for good reasons.

263. How do you manage pipeline versioning and compatibility when a shared template changes? ​

The short answer

Version the template like a library, with semantic versions: patch for fixes, minor for new optional features, major for anything that could break callers. Consumers pin a version and upgrade on purpose. Never move a released tag. Add new behaviour behind inputs with safe defaults, mark old inputs as deprecated before removing them, publish a changelog, support the previous major version for a while, and test each release against real consumers first.

Version numbers with a promise:

ChangeExampleVersion
Bug fix, no behaviour change for callersFix a cache key3.2.0 to 3.2.1
New optional input or step, off by defaultAdd run-e2e: false3.2.1 to 3.3.0
Removed or renamed input, new required stepRename service to name3.3.0 to 4.0.0

How consumers pin:

  • @v3.2.1 or a full commit SHA: exact and immutable. Updates come only through pull requests from a bot.
  • @v3, a moving major tag that the maintainers advance to each new 3.x release: automatic fixes, but small changes arrive without review. Some teams accept that for templates they trust.
  • Never a branch such as @main in production pipelines.

Making changes safely:

  1. New behaviour behind inputs, defaulting to the old behaviour, so a minor release changes nothing for existing callers.
  2. Deprecate before removing. Print a warning in pipeline logs when an old input is used, for at least one minor version, before the next major removes it.
  3. Changelog for every release: what changed, and what callers must do.
  4. Overlap. Keep fixing the previous major version for a few months, so teams can upgrade at their own pace.
  5. Test first against real consumer repositories (question 261).
  6. Never re-tag. Moving v3.2.1 to new code breaks the promise that pinned versions never change.

264. How should the pipeline behave when a job is cancelled, times out, or loses its runner? ​

The short answer

Fail clearly, clean up, and leave every environment in a known state. Every job needs a timeout, so a hung job does not block a queue forever. Clean-up steps should run whatever the outcome. Steps should be safe to retry. A deployment interrupted halfway should either finish, roll back, or be detected and repaired by the next run. Infrastructure failures, such as a lost runner, can be retried automatically, while code failures should not.

1. Timeouts everywhere.

yaml
jobs:
  deploy:
    timeout-minutes: 20                 # whole job
    steps:
      - run: ./deploy.sh
        timeout-minutes: 15             # one step

Without a timeout, a stuck step can hold a runner, and a deployment lock, for hours. GitHub's default job timeout is six hours, far too long for most jobs.

2. Clean-up that always runs. GitHub Actions has if: always() on a step, Jenkins has post { always { } }, and GitLab has after_script. Use them to release locks, delete temporary resources, and upload logs. Remember that a runner that died cannot run its own clean-up, so anything critical, such as releasing a lock, also needs an expiry time.

3. Safe retries. A step is idempotent when running it twice has the same effect as running it once. kubectl apply and helm upgrade --install are idempotent. "Insert this row" or "increase the version number" usually are not. Write deployment steps to be safe to run again.

4. Interrupted deployments. What happens if a deploy is cancelled after updating half the services?

  • Use deployment methods that are safe to interrupt. A Kubernetes rolling update keeps going on its own once started. A GitOps tool keeps applying the desired state, even if the pipeline that changed it was cancelled (question 507).
  • Hold a lock per environment, so the next deployment does not start on top of an unfinished one (question 222).
  • Make the next pipeline run check the environment's real state and repair it, rather than assuming the previous run finished.

5. Retry infrastructure failures, not code failures.

yaml
# GitLab: retry only when the runner itself failed
build:
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]

A real code failure retried is just a slower failure, and it hides problems.

6. Say what happened. Notifications should distinguish "cancelled", "timed out" and "runner lost" from "checks failed", so people know whether to look at their code or at the platform (question 232).

265. How would you keep CI/CD platform credentials and runner permissions isolated across teams? ​

The short answer

Give every team, and every trust level, its own runners, its own credentials and its own cloud permissions. Use separate runner groups or pools per team, so one team's jobs never run on machines that hold another team's access. Scope secrets to a project or environment, not to the whole organisation. With OIDC, map each repository and environment to its own cloud role. Put different teams and environments in separate cloud accounts or subscriptions, segment self-hosted runner networks, and audit who uses what.

1. Runners per team and trust level.

  • GitHub: runner groups, each available only to selected repositories.
  • GitLab: project or group runners, instead of instance-wide shared runners, for sensitive work.
  • Azure Pipelines: agent pools, with permissions per project.
  • Jenkins: labels and folder-level restrictions on which jobs may use which agents.

A self-hosted runner that deploys to the payments team's production must never accept jobs from other teams' repositories.

2. Secrets with the smallest possible scope.

  • Environment secrets over repository secrets, and repository secrets over organisation-wide ones (question 246).
  • Folder-scoped credentials in Jenkins (question 239).
  • No "one deploy key for everything".

3. Cloud identity per repository and environment. With OIDC, the token states exactly which repository, branch and environment it came from. The cloud's trust rules can then say: tokens from shop/payments in the production environment may use the payments production role, and nothing else may (question 269). A leak in one team's pipeline cannot reach another team's resources.

4. Separate accounts or subscriptions. Put production and non-production, and often each team, in separate cloud accounts or subscriptions. That boundary is the cloud's strongest one (question 527).

5. Network segmentation. Self-hosted runners for different teams sit in separate networks, with access only to what each team needs.

6. Audit. Platform audit logs record who used which credentials and runners, and alerts flag unusual use, such as a production credential used by an unexpected repository.

266. What would you monitor to detect an overloaded or unreliable CI/CD platform? ​

The short answer

Watch the platform the way you would watch any production service. Queue time tells you if there are enough runners. Runner availability and provisioning errors tell you if runners are healthy. Job duration trends show slowdowns. Failure rate, split into code failures and infrastructure failures, shows reliability. Also watch retries and flaky checks, timeouts, cache hit rate, the speed of registries and artifact stores, API rate-limit errors, the controller's own health, and cost.

SignalWhat it tells youExample alert
Queue time, p50 and p95Enough runners?p95 above 5 minutes for 15 minutes
Runner availabilityRunners online and accepting jobsFewer than 80% online
Provisioning errorsAutoscaling working?Any burst of failures
Job duration trendSlowdowns creeping inMedian up 50% this week
Failure rate by causeCode problems versus platform problemsInfrastructure failures above 2%
Retries and flaky rateTrust in resultsRising week over week
Timeouts and cancellationsHangs, overloaded dependenciesSudden spike
Cache hit rateCaches still workingBelow 60%
Registry and artifact latencyPull and push speedp95 pull time doubled
Rate-limit errorsHitting provider or registry limitsAny 429 or toomanyrequests
Controller healthJVM memory, disk, executor use for JenkinsDisk above 85%
CostMinutes and machines per dayOver budget

Set targets, like a service level objective. For example: "95% of jobs start within two minutes", or "fewer than 1% of runs fail because of the platform". Then alert on missing the target, rather than on every small blip (question 630).

Split failures by cause. A rise in failures caused by code is the teams' business. A rise in failures caused by runners, networks or the platform is the platform team's. Classifying them, even roughly, for example by matching known infrastructure error messages, tells you who should act.

Listen to people too. A quick regular survey, or simply counting complaints in the support channel, often notices "the pipeline feels slow" before the dashboards do.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet