Skip to content

Interview PrepCI/CD

Pipeline Security and the Supply Chain ​

267. How do you store, rotate, scope, and revoke secrets used by pipelines? ​

The short answer

Store them in the CI platform's secret store or an external vault, never in code. Scope each one as narrowly as possible: to one environment, one repository and one job. Prefer secrets that create themselves on demand and expire on their own, such as OIDC tokens or a vault's dynamic credentials. Rotate the rest on a schedule, using an overlap so nothing breaks. Revoke immediately when a secret leaks or a person leaves. Keep an inventory with an owner and an expiry date for every secret.

Store. The platform's own store, such as GitHub environment secrets or GitLab protected variables, or a vault such as HashiCorp Vault, Azure Key Vault or AWS Secrets Manager, which the pipeline reads at run time.

Scope. The smallest reach that works:

  1. Environment over repository, repository over organisation (question 246).
  2. Bound only into the steps that use it (question 239).
  3. With the smallest permissions on the other side, such as a registry token that can push to one repository only.

Prefer secrets that need no rotation. The best secret is one that does not exist until it is needed and expires minutes later:

  • OIDC federation to clouds (question 269).
  • Dynamic secrets from a vault, which creates, for example, a database user with a one-hour lifetime for each job.

Rotate the rest, without breaking things. A safe rotation overlaps old and new:

  1. Create the new secret, while the old one still works.
  2. Update every consumer to use the new one.
  3. Confirm nothing still uses the old one, from access logs.
  4. Revoke the old one.

Revoke immediately when:

  • A secret appears in a log, an image, a commit or a chat message.
  • A person with access leaves, or a laptop is lost.
  • A third-party tool that held the secret reports a breach.

Rotating after a leak is not optional. Bots scan public code for leaked keys within minutes, so assume a leaked secret has been seen.

Keep an inventory: every secret, what it is for, who owns it, where it is used, and when it expires. Unowned secrets are the ones that never get rotated.

268. Why should long-lived cloud access keys be avoided when short-lived identity federation is available? ​

The short answer

A long-lived key works for anyone who has it, from anywhere, until someone remembers to revoke it. Keys stored in CI get copied, logged, leaked through compromised tools, and kept by people who left. With identity federation, the pipeline proves who it is each time and gets credentials that expire within an hour, are limited to one repository, branch or environment, and are never stored. There is nothing worth stealing, and nothing to rotate.

What goes wrong with long-lived keys:

  1. They leak. Into logs, into images, into Git history, onto laptops. A compromised build tool or action can read them from the environment. In March 2025, a widely used GitHub Action, tj-actions/changed-files, was compromised and printed many repositories' secrets into their build logs.
  2. They last. A leaked key stays valid until someone revokes it, which may be never.
  3. They are anonymous. Logs show the key was used, not which pipeline run or person used it.
  4. They are often too powerful. One key ends up used by many pipelines, so it gets the union of everyone's permissions.
  5. They need rotating, which is manual work that is usually skipped.

What federation changes:

Long-lived keyFederated, short-lived token
Stored anywhereYes, as a CI secretNo
LifetimeUntil revokedMinutes to an hour
Usable byAnyone holding itOnly the trusted repository, branch or environment
TraceableTo the keyTo the exact workflow run
RotationManualNot needed

All major clouds support it: AWS with an OIDC identity provider and an IAM role, Azure with federated credentials on an app registration or a managed identity, and Google Cloud with workload identity federation. Question 269 explains how it works step by step.

269. How does OpenID Connect (OIDC) let a CI system obtain short-lived cloud credentials? ​

The short answer

The CI system issues each job a signed identity token that says exactly who the job is: which repository, which branch or environment, which workflow. The job hands that token to the cloud's token service. The cloud checks the signature using the CI provider's public keys. It checks the token's claims against a trust rule you configured, such as "only the main branch of shop/checkout-api". If everything matches, it returns credentials that expire within an hour.

CI jobCI token servicethe issuercloud token serviceAWS STS, Entra ID① give me an identity token② signed token: repo, branch, workflow③ exchange this token for credentials④ fetch public keys, check signatureand the trust rule⑤ credentials valid for one hour⑥ deploy with them; they expire on their own
Trading an identity for credentials. Nothing secret is stored anywhere. The CI provider vouches for the job, the cloud checks that vouching against its own rule, and the credentials it hands back expire quickly.

Step by step, with GitHub Actions and AWS:

  1. The job asks for a token. The workflow needs permissions: id-token: write. The job asks GitHub's token service for an OIDC token.
  2. GitHub returns a signed token. It is a JSON Web Token, a small signed piece of JSON. Its claims, the facts it states, include:
    • iss, the issuer: https://token.actions.githubusercontent.com
    • aud, the audience: who the token is meant for, such as sts.amazonaws.com
    • sub, the subject: for example repo:shop/checkout-api:ref:refs/heads/main, or repo:shop/checkout-api:environment:production
  3. The job sends the token to AWS, asking to assume a role.
  4. AWS checks it. It downloads GitHub's public signing keys from the issuer's address, verifies the signature, checks the expiry, and compares the claims with the role's trust policy.
  5. AWS returns temporary credentials, valid for up to an hour.
  6. The job uses them, and they expire by themselves.

The trust policy is where security lives:

json
{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:shop/checkout-api:environment:production"
    }
  }
}

Only jobs from shop/checkout-api that run in its production environment can use this role. A pull request, another branch or another repository has a different sub, and is refused. A dangerous mistake is a loose condition, such as a wildcard that matches every repository in the organisation, or worse, every repository on GitHub.

On Azure, the same idea is a federated identity credential on an app registration or a user-assigned managed identity, with the issuer and the exact subject. GitLab, Azure Pipelines and Jenkins with plugins can all issue OIDC tokens in the same way.

270. How do you prevent secrets from being exposed through logs, shell tracing, artifacts, caches, or pull-request code? ​

The short answer

Block each route on purpose. Never print secrets, and never turn on shell tracing in steps that handle them, because log masking only hides exact matches. Keep secret files out of anything uploaded as an artifact or saved in a cache. Never give secrets to jobs that run unreviewed pull request code. Pass untrusted text through environment variables, not pasted into scripts. Give secrets only to the steps that need them, and be careful which third-party actions run where secrets exist.

RouteHow secrets leakPrevention
Logsecho $TOKEN, error messages, verbose tool outputDo not print; masking as a last line only
Shell tracingset -x prints commands with valuesNo tracing around secrets
Encoded valuesbase64 or URL-encoded secrets are not maskedRegister derived values as masked too
ArtifactsA .npmrc or config file uploaded with the build outputDelete secret files; upload explicit paths only
CachesCredentials written into a cached folderKeep secret files out of cached paths
Pull request codeUnreviewed code runs where secrets existNo secrets for untrusted runs (question 250)
Script injectionUntrusted text pasted into a commandPass through environment variables
Third-party actionsAn action reads every environment variablePin by SHA, limit which steps get secrets
Debug loggingPlatform debug modes print moreRestrict who can enable it

Masking derived values in GitHub Actions:

yaml
- run: |
    BASIC=$(printf '%s:%s' "$USER" "$PASS" | base64 -w0)
    echo "::add-mask::$BASIC"            # (1)
    curl -H "Authorization: Basic $BASIC" https://api.example.com
  env:
    USER: ${{ secrets.API_USER }}
    PASS: ${{ secrets.API_PASS }}
  1. The base64 version of the credentials is a new string that GitHub does not know is secret. ::add-mask:: tells it to hide this value too in all later log lines.

Uploading only what you mean to:

yaml
- uses: actions/upload-artifact@v4
  with:
    name: dist
    path: dist/                          # never the whole workspace

Assume the log is public. Many CI logs can be read by everyone in the organisation, and public repositories' logs by everyone in the world. If a value would be harmful in a public log, it must never reach the log at all.

271. How do you apply least privilege to CI runners, deployment identities, and production environments? ​

The short answer

Give each piece only what its job needs, only when it needs it. Runners hold no standing credentials, and the pipeline token defaults to read-only. Each service and environment gets its own deploy identity, scoped to exactly its own resources, such as one registry repository and one Kubernetes namespace, using short-lived credentials. Production gets its own account, no permanent human write access, just-in-time elevation for emergencies, and ideally a pull-based GitOps tool so CI never holds cluster credentials at all.

PieceLeast-privilege setup
CI runnerNo stored cloud keys; ephemeral; separate pools per trust level
Pipeline tokenpermissions: contents: read by default; extra rights per job only
Build identityCan push to its own registry repository, nothing else
Deploy identityOne per service and environment; can update one namespace
Production access for peopleNone standing; time-limited elevation with approval
Production clusterGitOps pulls from Git; CI cannot reach the cluster API

Scoped deploy identities, for example in Kubernetes:

yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: checkout-deployer
  namespace: shop-prod
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["checkout-api"]                # (1)
    verbs: ["get", "patch"]                        # (2)
  1. Only the checkout-api Deployment, in only the shop-prod namespace.
  2. Only reading it and patching it, enough to update the image. It cannot create new workloads, read secrets or delete anything (question 416).

People and production. Day-to-day, engineers have read access. For an emergency, they request just-in-time elevated access, such as Azure's Privileged Identity Management or an approval-based role in AWS, which expires after an hour and is logged. This is often called break-glass access.

The strongest pattern: pull instead of push. With GitOps, a tool such as Argo CD runs inside the cluster and pulls changes from Git (question 507). CI only needs permission to update a Git repository. If CI is compromised, the attacker still cannot reach the cluster directly, and every change is a reviewable commit.

272. How would you harden a self-hosted runner that processes code from external contributors? ​

The short answer

The safest answer is: don't. Use the provider's hosted runners for outside contributions. If you must use your own, make every runner single-use, destroyed after one job. Put it in an isolated network with no route to anything internal and filtered outbound traffic. Give it no secrets, no Docker socket and no privileged mode. Require a maintainer's approval before outsiders' workflows run. Restrict the runner group to that one repository, and monitor it.

Why this is so risky. A pull request from a stranger can change the workflow or the build scripts. Whatever it says runs on your runner. On a long-lived machine inside your network, that is a stranger running code inside your network, with time to install something that waits for the next, more privileged job.

If you really must, layer the defences:

  1. One job per runner. Each runner is a fresh virtual machine or pod, deleted after a single job (question 183). Nothing persists for an attacker to hide in.
  2. Network isolation. The runner's network reaches the internet, through a filter, and nothing internal: no databases, no clusters, no metadata service. Cloud instance metadata services, an address every cloud machine can query for information including temporary credentials, should be blocked for these runners.
  3. No secrets. Outside contributions never receive secrets (question 250).
  4. No privileged access. No Docker socket, no privileged containers, an unprivileged user (questions 140 and 141). Stronger isolation, such as gVisor, Kata Containers or Firecracker micro-VMs, helps further.
  5. Approval before running. Require a maintainer to approve workflow runs from first-time or outside contributors.
  6. A dedicated runner group that only this public repository can use, so a compromise cannot spread to private repositories' jobs.
  7. Limits and monitoring. CPU, memory and time limits stop crypto-mining abuse. Alerts on unusual outbound connections catch the rest.

273. What are software supply-chain attacks, and how can a pipeline reduce the risk of compromised dependencies or build steps? ​

The short answer

A supply-chain attack targets something you depend on, instead of you directly: a library, a build tool, a pipeline action or the build system itself, so that malicious code arrives inside your own trusted software. Pipelines reduce the risk in several ways. They pin every dependency and tool exactly, and verify what they download. They proxy and vet packages. They isolate and harden the build, and limit what each step can access. And they produce signed records of exactly how each artifact was built.

Real examples, each a different link in the chain:

YearIncidentWhat was compromised
2018event-streamA popular npm package, handed to a new maintainer who added code to steal cryptocurrency wallets
2020SolarWindsThe vendor's build system, which inserted a backdoor into signed software updates
2021CodecovA script many CI pipelines downloaded and ran, altered to send their secrets to the attacker
2024xz utils (CVE-2024-3094)A compression library, where a contributor spent years gaining trust before hiding a backdoor in its build
2025tj-actions/changed-filesA GitHub Action, changed to print the secrets of the pipelines that used it

How a pipeline defends itself:

  1. Pin everything exactly. Lock files for libraries, digests for base images, full commit SHAs for actions. A moved tag or a new version cannot slip in silently (question 277).
  2. Verify downloads. Lock-file integrity hashes, checksums for downloaded tools, signatures where publishers provide them.
  3. Proxy and vet packages. Pull dependencies through your own repository, which can block known-malicious packages and hold brand-new versions for a few days before allowing them, since many malicious releases are caught within that time.
  4. Scan dependencies for known vulnerabilities and malicious packages (question 214).
  5. Harden the build. Ephemeral, isolated runners, builds that cannot reach the internet except through the proxy, and no secrets in steps that run third-party code.
  6. Limit each step. Minimal token permissions, secrets only for the steps that need them (question 271).
  7. Review pipeline changes with two people, because the pipeline itself is part of the supply chain.
  8. Record and sign provenance, so consumers can verify how and from what each artifact was built. The SLSA framework, Supply-chain Levels for Software Artifacts, describes increasing levels of this protection (question 275).

274. What is a software bill of materials (SBOM), and how could it help during vulnerability response? ​

The short answer

An SBOM is a machine-readable list of every component inside a piece of software, such as every library and operating system package with its exact version, in a standard format like SPDX or CycloneDX. When a new vulnerability is announced, you search your stored SBOMs for the affected package and version. Within minutes, you know exactly which images and services contain it. So you can prioritise and fix, instead of guessing or rescanning everything.

What an SBOM entry looks like, simplified CycloneDX:

json
{
  "type": "library",
  "name": "log4j-core",
  "version": "2.14.1",
  "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1"
}

The purl, a package URL, is a standard way to name a package across ecosystems, so tools can match it precisely.

Generating them: BuildKit can attach an SBOM to each image at build time (question 146), and tools such as Syft or Trivy can generate one for any image:

bash
syft myacr.azurecr.io/shop/checkout-api:3f9c2ab -o cyclonedx-json > sbom.json

A vulnerability response, with and without SBOMs. In December 2021, a critical flaw in the Java logging library Log4j, called Log4Shell, was announced. Every company asked: "where do we use Log4j 2 below version 2.15?"

  • Without SBOMs: teams searched code, asked around, and rescanned every image they could find. For many companies this took days, and some places were missed.
  • With SBOMs stored centrally, for example in a tool such as Dependency-Track: one query lists every image and every service containing log4j-core in an affected version.

The response then follows a clear path:

  1. Find every affected artifact, from the SBOMs.
  2. Map artifacts to running services, using deployment records (question 196).
  3. Prioritise by exposure: internet-facing services first.
  4. Fix: update the dependency, rebuild, redeploy.
  5. Verify: the new SBOMs no longer contain the affected version.

An SBOM is only useful if it is generated for every build, stored, and searchable. A pile of unindexed files does not help on the day it matters.

275. What are artifact signing, provenance attestations, and verification policies? ​

The short answer

Signing proves an artifact came from a specific identity and has not been changed since. An attestation is a signed statement about an artifact, and a provenance attestation is the statement of how it was built: from which source commit, by which workflow, on which builder, with which inputs. A verification policy is the rule that consumers enforce before using an artifact, for example: "only run images signed by our release workflow, with provenance showing they were built from the main branch of our own repository".

Signatures answer: who published this, and is it intact? (question 146)

Attestations answer more specific questions. Each is a signed document, usually in the in-toto format, that points at the artifact's digest and states something about it:

AttestationStates
ProvenanceHow it was built: source, workflow, builder, parameters
SBOMWhat it contains
Vulnerability scanWhat a scanner found, and when
Check resultsWhich pipeline checks passed

Verification policies turn those statements into rules. Examples:

  1. The image must be signed by https://github.com/shop/*/.github/workflows/release.yml@refs/heads/main.
  2. Its provenance must name a source repository in the shop organisation.
  3. It must have a vulnerability scan attestation less than seven days old with no critical findings.

Policies are enforced at deploy time, by an admission controller in the cluster (question 147), or in the pipeline before promotion.

SLSA, Supply-chain Levels for Software Artifacts, is a framework that grades how trustworthy provenance is. In simple terms, its build levels go from:

  • Level 1: provenance exists, describing how the artifact was built.
  • Level 2: the provenance is signed by a hosted build platform, so it is harder to fake.
  • Level 3: the build platform is hardened, so even the people running a build cannot forge or tamper with its provenance.

Hosted CI systems with built-in provenance generation, such as GitHub's artifact attestations, make the middle levels easy to reach.

276. How do you ensure that the artifact deployed to production was built by an approved workflow from an approved source revision? ​

The short answer

Have the approved workflow generate a signed provenance attestation for every artifact, recording the source repository, the exact commit and the workflow that built it, signed with the workflow's own short-lived identity. Then, at deployment, verify the attestation against a policy: the signer must be the release workflow, running on the protected main branch, of an approved repository. Deploy by digest so the verified artifact is exactly the deployed one, and reject anything that fails.

1. Generate provenance in the release workflow. In GitHub Actions:

yaml
permissions:
  id-token: write          # identity for signing
  attestations: write      # store the attestation
  packages: write
steps:
  - id: build
    uses: docker/build-push-action@v6
    with: { push: true, tags: "myacr.azurecr.io/shop/checkout-api:${{ github.sha }}" }
  - uses: actions/attest-build-provenance@v2
    with:
      subject-name: myacr.azurecr.io/shop/checkout-api
      subject-digest: ${{ steps.build.outputs.digest }}
      push-to-registry: true

The attestation records the repository, the commit SHA, the workflow file and the run, and it is signed using the workflow's OIDC identity, so it cannot be produced by any other workflow.

2. Verify before deploying.

bash
gh attestation verify oci://myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8... \
  --owner shop \
  --signer-workflow shop/checkout-api/.github/workflows/release.yml

This passes only if a valid attestation exists for that digest, signed by that exact workflow in the shop organisation.

3. Enforce it in the cluster, so the check cannot be skipped by deploying some other way, using an admission policy that verifies signatures and attestations (question 147).

4. Make "approved revision" mean something. The workflow only runs on the protected main branch, where every commit was reviewed and passed required checks (question 192). So "built by the release workflow from main" implies "built from reviewed code".

277. How should dependency pinning, lockfiles, image digests, and signature verification be used together? ​

The short answer

Each one pins down a different layer, and together they make the whole build exact and tamper-evident. Lock files fix the exact version and hash of every library. Image digests fix the exact base images and the exact images you deploy. Commit SHAs fix the exact pipeline actions and tools. Signatures and attestations prove where your own artifacts, and ideally your inputs, came from. Automation keeps the pins fresh, so exactness never turns into running old, vulnerable versions.

what you import: librarieslock files with integrity hashes: package-lock.json, go.sumwhat you build on: base imagesFROM node:20-slim@sha256:9a1f...what builds it: pipeline steps and toolsuses: actions/checkout@11bd719... (full commit SHA)what you ship: your imagedeployed by digest, signed, with provenance, verified on admissionupdate botrefreshes everypin through areviewed PR
Every layer pinned. Each layer is fixed to exact bytes, so nothing changes without a visible pull request. The bot on the right keeps all of them up to date, which is what stops pinning from turning into neglect.

What each layer protects against:

  1. Lock files with hashes: a library version silently changing, or a tampered download. npm ci refuses any package whose hash differs from the lock file.
  2. Base image digests: a base image tag moved to different, possibly malicious, contents (question 86).
  3. Action SHAs: a third-party action's tag being moved to compromised code, as happened with tj-actions/changed-files in 2025. A full commit SHA cannot be moved.
  4. Deploying by digest: the image you checked being swapped for another under the same tag.
  5. Signatures and attestations: an image that did not come from your pipeline at all (question 276). Verifying the signatures of your inputs too, such as signed base images, extends the same protection upstream.

Why automation is part of the answer. Pinned versions never update by themselves. Without a bot, pins become months-old software with known vulnerabilities. With Renovate or Dependabot, every pin is refreshed regularly, each refresh is a pull request that runs the full pipeline, and a human or an auto-merge rule decides (question 166).

278. How do you decide which security findings must block a release and which can be accepted temporarily? ​

The short answer

By risk, using a policy written down in advance. Block when a finding is serious, actually exploitable in your situation, and fixable now. Serious means critical severity, or known to be exploited in the wild. Exploitable means it is in a component the service really uses, on a service that is exposed. Fixable means a fix is available. Accept temporarily, with an owner and an expiry date, when there is no fix yet, the vulnerable code cannot be reached, or other controls already block the attack.

Question 137 covers the same decision for image vulnerabilities. In the pipeline, apply the same questions to every kind of finding:

QuestionPushes towards blockingPushes towards accepting
How severe?Critical or highLow or medium
Exploited in the wild?Listed in CISA's KEV catalogueNo known exploitation
Reachable?Code path used by the servicePackage present but never loaded
Exposed?Internet-facing, handles sensitive dataInternal, isolated
Fix available?Yes, a version bumpNo fix yet
Other protection?NoneA firewall rule or a disabled feature blocks it

Kinds of findings and typical rules:

  • A leaked secret in the change: always block. Rotate it, even after the change is fixed (question 267).
  • A static analysis finding of injection in new code: block until fixed or shown to be a false alarm.
  • A dependency vulnerability: apply the table.
  • A misconfiguration, such as a container running as root or a storage bucket open to the public: block for production.

Accepting is a decision, not a silence. Every accepted finding is recorded with its reason, its owner and its expiry (question 279), and the pipeline itself enforces the expiry.

279. How do you document security exceptions, owners, expiry dates, and compensating controls? ​

The short answer

Record each exception in a structured, versioned form, ideally as code next to the policy it overrides, so the pipeline can enforce it. Each record names the exact finding and the exact artifact or service, the reason, the risk assessment, the compensating controls that reduce the risk, the owner, the approver and an expiry date. When the expiry passes, the exception stops working automatically, and the finding blocks again until it is fixed or re-approved.

What every exception must contain:

FieldExample
FindingCVE-2026-12345 in libxml2
ScopeImage shop/catalog-api, versions up to 2.8.x
ReasonNo fixed version published yet
Risk assessmentThe vulnerable function parses XML; catalog-api never parses XML from users
Compensating controlUpload endpoint rejects XML; the WAF blocks XML request bodies
Ownercatalog team, Asha
ApproverSecurity, Ravi
Expiry2026-11-30

A compensating control is something else that reduces the risk while the real fix is not possible, such as a firewall rule, a disabled feature, network isolation or extra monitoring.

Keep it as code, so the pipeline enforces it. Many tools accept exception files with expiry dates. For example, a Trivy ignore file in YAML form:

yaml
# .trivyignore.yaml
vulnerabilities:
  - id: CVE-2026-12345
    paths: ["usr/lib/x86_64-linux-gnu/libxml2.so.2"]
    statement: "XML is never parsed from user input; WAF blocks XML bodies. Owner: catalog-team. Approved: security/ravi."
    expired_at: 2026-11-30

After expired_at, the scanner reports the finding again and the gate fails. Kyverno has a PolicyException object that does the same for cluster policies, and VEX documents can record "not affected" decisions with justifications (question 137).

Process around it:

  1. Exceptions are added through pull requests, approved by security, never by the requester alone.
  2. A dashboard lists all active exceptions and those expiring soon.
  3. Expiry dates are short, typically 30 to 90 days, so exceptions are reviewed rather than forgotten.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet