Appearance
Pipeline Security and the Supply Chain
267. How do you store, rotate, scope, and revoke secrets used by pipelines?
The short answer
Store them in the CI platform's secret store or an external vault, never in code. Scope each one as narrowly as possible: to one environment, one repository and one job. Prefer secrets that create themselves on demand and expire on their own, such as OIDC tokens or a vault's dynamic credentials. Rotate the rest on a schedule, using an overlap so nothing breaks. Revoke immediately when a secret leaks or a person leaves. Keep an inventory with an owner and an expiry date for every secret.
Store. The platform's own store, such as GitHub environment secrets or GitLab protected variables, or a vault such as HashiCorp Vault, Azure Key Vault or AWS Secrets Manager, which the pipeline reads at run time.
Scope. The smallest reach that works:
- Environment over repository, repository over organisation (question 246).
- Bound only into the steps that use it (question 239).
- With the smallest permissions on the other side, such as a registry token that can push to one repository only.
Prefer secrets that need no rotation. The best secret is one that does not exist until it is needed and expires minutes later:
- OIDC federation to clouds (question 269).
- Dynamic secrets from a vault, which creates, for example, a database user with a one-hour lifetime for each job.
Rotate the rest, without breaking things. A safe rotation overlaps old and new:
- Create the new secret, while the old one still works.
- Update every consumer to use the new one.
- Confirm nothing still uses the old one, from access logs.
- Revoke the old one.
Revoke immediately when:
- A secret appears in a log, an image, a commit or a chat message.
- A person with access leaves, or a laptop is lost.
- A third-party tool that held the secret reports a breach.
Rotating after a leak is not optional. Bots scan public code for leaked keys within minutes, so assume a leaked secret has been seen.
Keep an inventory: every secret, what it is for, who owns it, where it is used, and when it expires. Unowned secrets are the ones that never get rotated.
268. Why should long-lived cloud access keys be avoided when short-lived identity federation is available?
The short answer
A long-lived key works for anyone who has it, from anywhere, until someone remembers to revoke it. Keys stored in CI get copied, logged, leaked through compromised tools, and kept by people who left. With identity federation, the pipeline proves who it is each time and gets credentials that expire within an hour, are limited to one repository, branch or environment, and are never stored. There is nothing worth stealing, and nothing to rotate.
What goes wrong with long-lived keys:
- They leak. Into logs, into images, into Git history, onto laptops. A compromised build tool or action can read them from the environment. In March 2025, a widely used GitHub Action,
tj-actions/changed-files, was compromised and printed many repositories' secrets into their build logs. - They last. A leaked key stays valid until someone revokes it, which may be never.
- They are anonymous. Logs show the key was used, not which pipeline run or person used it.
- They are often too powerful. One key ends up used by many pipelines, so it gets the union of everyone's permissions.
- They need rotating, which is manual work that is usually skipped.
What federation changes:
| Long-lived key | Federated, short-lived token | |
|---|---|---|
| Stored anywhere | Yes, as a CI secret | No |
| Lifetime | Until revoked | Minutes to an hour |
| Usable by | Anyone holding it | Only the trusted repository, branch or environment |
| Traceable | To the key | To the exact workflow run |
| Rotation | Manual | Not needed |
All major clouds support it: AWS with an OIDC identity provider and an IAM role, Azure with federated credentials on an app registration or a managed identity, and Google Cloud with workload identity federation. Question 269 explains how it works step by step.
269. How does OpenID Connect (OIDC) let a CI system obtain short-lived cloud credentials?
The short answer
The CI system issues each job a signed identity token that says exactly who the job is: which repository, which branch or environment, which workflow. The job hands that token to the cloud's token service. The cloud checks the signature using the CI provider's public keys. It checks the token's claims against a trust rule you configured, such as "only the main branch of shop/checkout-api". If everything matches, it returns credentials that expire within an hour.
Step by step, with GitHub Actions and AWS:
- The job asks for a token. The workflow needs
permissions: id-token: write. The job asks GitHub's token service for an OIDC token. - GitHub returns a signed token. It is a JSON Web Token, a small signed piece of JSON. Its claims, the facts it states, include:
iss, the issuer:https://token.actions.githubusercontent.comaud, the audience: who the token is meant for, such assts.amazonaws.comsub, the subject: for examplerepo:shop/checkout-api:ref:refs/heads/main, orrepo:shop/checkout-api:environment:production
- The job sends the token to AWS, asking to assume a role.
- AWS checks it. It downloads GitHub's public signing keys from the issuer's address, verifies the signature, checks the expiry, and compares the claims with the role's trust policy.
- AWS returns temporary credentials, valid for up to an hour.
- The job uses them, and they expire by themselves.
The trust policy is where security lives:
json
{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:shop/checkout-api:environment:production"
}
}
}Only jobs from shop/checkout-api that run in its production environment can use this role. A pull request, another branch or another repository has a different sub, and is refused. A dangerous mistake is a loose condition, such as a wildcard that matches every repository in the organisation, or worse, every repository on GitHub.
On Azure, the same idea is a federated identity credential on an app registration or a user-assigned managed identity, with the issuer and the exact subject. GitLab, Azure Pipelines and Jenkins with plugins can all issue OIDC tokens in the same way.
270. How do you prevent secrets from being exposed through logs, shell tracing, artifacts, caches, or pull-request code?
The short answer
Block each route on purpose. Never print secrets, and never turn on shell tracing in steps that handle them, because log masking only hides exact matches. Keep secret files out of anything uploaded as an artifact or saved in a cache. Never give secrets to jobs that run unreviewed pull request code. Pass untrusted text through environment variables, not pasted into scripts. Give secrets only to the steps that need them, and be careful which third-party actions run where secrets exist.
| Route | How secrets leak | Prevention |
|---|---|---|
| Logs | echo $TOKEN, error messages, verbose tool output | Do not print; masking as a last line only |
| Shell tracing | set -x prints commands with values | No tracing around secrets |
| Encoded values | base64 or URL-encoded secrets are not masked | Register derived values as masked too |
| Artifacts | A .npmrc or config file uploaded with the build output | Delete secret files; upload explicit paths only |
| Caches | Credentials written into a cached folder | Keep secret files out of cached paths |
| Pull request code | Unreviewed code runs where secrets exist | No secrets for untrusted runs (question 250) |
| Script injection | Untrusted text pasted into a command | Pass through environment variables |
| Third-party actions | An action reads every environment variable | Pin by SHA, limit which steps get secrets |
| Debug logging | Platform debug modes print more | Restrict who can enable it |
Masking derived values in GitHub Actions:
yaml
- run: |
BASIC=$(printf '%s:%s' "$USER" "$PASS" | base64 -w0)
echo "::add-mask::$BASIC" # (1)
curl -H "Authorization: Basic $BASIC" https://api.example.com
env:
USER: ${{ secrets.API_USER }}
PASS: ${{ secrets.API_PASS }}- The base64 version of the credentials is a new string that GitHub does not know is secret.
::add-mask::tells it to hide this value too in all later log lines.
Uploading only what you mean to:
yaml
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist/ # never the whole workspaceAssume the log is public. Many CI logs can be read by everyone in the organisation, and public repositories' logs by everyone in the world. If a value would be harmful in a public log, it must never reach the log at all.
271. How do you apply least privilege to CI runners, deployment identities, and production environments?
The short answer
Give each piece only what its job needs, only when it needs it. Runners hold no standing credentials, and the pipeline token defaults to read-only. Each service and environment gets its own deploy identity, scoped to exactly its own resources, such as one registry repository and one Kubernetes namespace, using short-lived credentials. Production gets its own account, no permanent human write access, just-in-time elevation for emergencies, and ideally a pull-based GitOps tool so CI never holds cluster credentials at all.
| Piece | Least-privilege setup |
|---|---|
| CI runner | No stored cloud keys; ephemeral; separate pools per trust level |
| Pipeline token | permissions: contents: read by default; extra rights per job only |
| Build identity | Can push to its own registry repository, nothing else |
| Deploy identity | One per service and environment; can update one namespace |
| Production access for people | None standing; time-limited elevation with approval |
| Production cluster | GitOps pulls from Git; CI cannot reach the cluster API |
Scoped deploy identities, for example in Kubernetes:
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: checkout-deployer
namespace: shop-prod
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
resourceNames: ["checkout-api"] # (1)
verbs: ["get", "patch"] # (2)- Only the
checkout-apiDeployment, in only theshop-prodnamespace. - Only reading it and patching it, enough to update the image. It cannot create new workloads, read secrets or delete anything (question 416).
People and production. Day-to-day, engineers have read access. For an emergency, they request just-in-time elevated access, such as Azure's Privileged Identity Management or an approval-based role in AWS, which expires after an hour and is logged. This is often called break-glass access.
The strongest pattern: pull instead of push. With GitOps, a tool such as Argo CD runs inside the cluster and pulls changes from Git (question 507). CI only needs permission to update a Git repository. If CI is compromised, the attacker still cannot reach the cluster directly, and every change is a reviewable commit.
272. How would you harden a self-hosted runner that processes code from external contributors?
The short answer
The safest answer is: don't. Use the provider's hosted runners for outside contributions. If you must use your own, make every runner single-use, destroyed after one job. Put it in an isolated network with no route to anything internal and filtered outbound traffic. Give it no secrets, no Docker socket and no privileged mode. Require a maintainer's approval before outsiders' workflows run. Restrict the runner group to that one repository, and monitor it.
Why this is so risky. A pull request from a stranger can change the workflow or the build scripts. Whatever it says runs on your runner. On a long-lived machine inside your network, that is a stranger running code inside your network, with time to install something that waits for the next, more privileged job.
If you really must, layer the defences:
- One job per runner. Each runner is a fresh virtual machine or pod, deleted after a single job (question 183). Nothing persists for an attacker to hide in.
- Network isolation. The runner's network reaches the internet, through a filter, and nothing internal: no databases, no clusters, no metadata service. Cloud instance metadata services, an address every cloud machine can query for information including temporary credentials, should be blocked for these runners.
- No secrets. Outside contributions never receive secrets (question 250).
- No privileged access. No Docker socket, no privileged containers, an unprivileged user (questions 140 and 141). Stronger isolation, such as gVisor, Kata Containers or Firecracker micro-VMs, helps further.
- Approval before running. Require a maintainer to approve workflow runs from first-time or outside contributors.
- A dedicated runner group that only this public repository can use, so a compromise cannot spread to private repositories' jobs.
- Limits and monitoring. CPU, memory and time limits stop crypto-mining abuse. Alerts on unusual outbound connections catch the rest.
273. What are software supply-chain attacks, and how can a pipeline reduce the risk of compromised dependencies or build steps?
The short answer
A supply-chain attack targets something you depend on, instead of you directly: a library, a build tool, a pipeline action or the build system itself, so that malicious code arrives inside your own trusted software. Pipelines reduce the risk in several ways. They pin every dependency and tool exactly, and verify what they download. They proxy and vet packages. They isolate and harden the build, and limit what each step can access. And they produce signed records of exactly how each artifact was built.
Real examples, each a different link in the chain:
| Year | Incident | What was compromised |
|---|---|---|
| 2018 | event-stream | A popular npm package, handed to a new maintainer who added code to steal cryptocurrency wallets |
| 2020 | SolarWinds | The vendor's build system, which inserted a backdoor into signed software updates |
| 2021 | Codecov | A script many CI pipelines downloaded and ran, altered to send their secrets to the attacker |
| 2024 | xz utils (CVE-2024-3094) | A compression library, where a contributor spent years gaining trust before hiding a backdoor in its build |
| 2025 | tj-actions/changed-files | A GitHub Action, changed to print the secrets of the pipelines that used it |
How a pipeline defends itself:
- Pin everything exactly. Lock files for libraries, digests for base images, full commit SHAs for actions. A moved tag or a new version cannot slip in silently (question 277).
- Verify downloads. Lock-file integrity hashes, checksums for downloaded tools, signatures where publishers provide them.
- Proxy and vet packages. Pull dependencies through your own repository, which can block known-malicious packages and hold brand-new versions for a few days before allowing them, since many malicious releases are caught within that time.
- Scan dependencies for known vulnerabilities and malicious packages (question 214).
- Harden the build. Ephemeral, isolated runners, builds that cannot reach the internet except through the proxy, and no secrets in steps that run third-party code.
- Limit each step. Minimal token permissions, secrets only for the steps that need them (question 271).
- Review pipeline changes with two people, because the pipeline itself is part of the supply chain.
- Record and sign provenance, so consumers can verify how and from what each artifact was built. The SLSA framework, Supply-chain Levels for Software Artifacts, describes increasing levels of this protection (question 275).
274. What is a software bill of materials (SBOM), and how could it help during vulnerability response?
The short answer
An SBOM is a machine-readable list of every component inside a piece of software, such as every library and operating system package with its exact version, in a standard format like SPDX or CycloneDX. When a new vulnerability is announced, you search your stored SBOMs for the affected package and version. Within minutes, you know exactly which images and services contain it. So you can prioritise and fix, instead of guessing or rescanning everything.
What an SBOM entry looks like, simplified CycloneDX:
json
{
"type": "library",
"name": "log4j-core",
"version": "2.14.1",
"purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1"
}The purl, a package URL, is a standard way to name a package across ecosystems, so tools can match it precisely.
Generating them: BuildKit can attach an SBOM to each image at build time (question 146), and tools such as Syft or Trivy can generate one for any image:
bash
syft myacr.azurecr.io/shop/checkout-api:3f9c2ab -o cyclonedx-json > sbom.jsonA vulnerability response, with and without SBOMs. In December 2021, a critical flaw in the Java logging library Log4j, called Log4Shell, was announced. Every company asked: "where do we use Log4j 2 below version 2.15?"
- Without SBOMs: teams searched code, asked around, and rescanned every image they could find. For many companies this took days, and some places were missed.
- With SBOMs stored centrally, for example in a tool such as Dependency-Track: one query lists every image and every service containing
log4j-corein an affected version.
The response then follows a clear path:
- Find every affected artifact, from the SBOMs.
- Map artifacts to running services, using deployment records (question 196).
- Prioritise by exposure: internet-facing services first.
- Fix: update the dependency, rebuild, redeploy.
- Verify: the new SBOMs no longer contain the affected version.
An SBOM is only useful if it is generated for every build, stored, and searchable. A pile of unindexed files does not help on the day it matters.
275. What are artifact signing, provenance attestations, and verification policies?
The short answer
Signing proves an artifact came from a specific identity and has not been changed since. An attestation is a signed statement about an artifact, and a provenance attestation is the statement of how it was built: from which source commit, by which workflow, on which builder, with which inputs. A verification policy is the rule that consumers enforce before using an artifact, for example: "only run images signed by our release workflow, with provenance showing they were built from the main branch of our own repository".
Signatures answer: who published this, and is it intact? (question 146)
Attestations answer more specific questions. Each is a signed document, usually in the in-toto format, that points at the artifact's digest and states something about it:
| Attestation | States |
|---|---|
| Provenance | How it was built: source, workflow, builder, parameters |
| SBOM | What it contains |
| Vulnerability scan | What a scanner found, and when |
| Check results | Which pipeline checks passed |
Verification policies turn those statements into rules. Examples:
- The image must be signed by
https://github.com/shop/*/.github/workflows/release.yml@refs/heads/main. - Its provenance must name a source repository in the
shoporganisation. - It must have a vulnerability scan attestation less than seven days old with no critical findings.
Policies are enforced at deploy time, by an admission controller in the cluster (question 147), or in the pipeline before promotion.
SLSA, Supply-chain Levels for Software Artifacts, is a framework that grades how trustworthy provenance is. In simple terms, its build levels go from:
- Level 1: provenance exists, describing how the artifact was built.
- Level 2: the provenance is signed by a hosted build platform, so it is harder to fake.
- Level 3: the build platform is hardened, so even the people running a build cannot forge or tamper with its provenance.
Hosted CI systems with built-in provenance generation, such as GitHub's artifact attestations, make the middle levels easy to reach.
276. How do you ensure that the artifact deployed to production was built by an approved workflow from an approved source revision?
The short answer
Have the approved workflow generate a signed provenance attestation for every artifact, recording the source repository, the exact commit and the workflow that built it, signed with the workflow's own short-lived identity. Then, at deployment, verify the attestation against a policy: the signer must be the release workflow, running on the protected main branch, of an approved repository. Deploy by digest so the verified artifact is exactly the deployed one, and reject anything that fails.
1. Generate provenance in the release workflow. In GitHub Actions:
yaml
permissions:
id-token: write # identity for signing
attestations: write # store the attestation
packages: write
steps:
- id: build
uses: docker/build-push-action@v6
with: { push: true, tags: "myacr.azurecr.io/shop/checkout-api:${{ github.sha }}" }
- uses: actions/attest-build-provenance@v2
with:
subject-name: myacr.azurecr.io/shop/checkout-api
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: trueThe attestation records the repository, the commit SHA, the workflow file and the run, and it is signed using the workflow's OIDC identity, so it cannot be produced by any other workflow.
2. Verify before deploying.
bash
gh attestation verify oci://myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8... \
--owner shop \
--signer-workflow shop/checkout-api/.github/workflows/release.ymlThis passes only if a valid attestation exists for that digest, signed by that exact workflow in the shop organisation.
3. Enforce it in the cluster, so the check cannot be skipped by deploying some other way, using an admission policy that verifies signatures and attestations (question 147).
4. Make "approved revision" mean something. The workflow only runs on the protected main branch, where every commit was reviewed and passed required checks (question 192). So "built by the release workflow from main" implies "built from reviewed code".
277. How should dependency pinning, lockfiles, image digests, and signature verification be used together?
The short answer
Each one pins down a different layer, and together they make the whole build exact and tamper-evident. Lock files fix the exact version and hash of every library. Image digests fix the exact base images and the exact images you deploy. Commit SHAs fix the exact pipeline actions and tools. Signatures and attestations prove where your own artifacts, and ideally your inputs, came from. Automation keeps the pins fresh, so exactness never turns into running old, vulnerable versions.
What each layer protects against:
- Lock files with hashes: a library version silently changing, or a tampered download.
npm cirefuses any package whose hash differs from the lock file. - Base image digests: a base image tag moved to different, possibly malicious, contents (question 86).
- Action SHAs: a third-party action's tag being moved to compromised code, as happened with
tj-actions/changed-filesin 2025. A full commit SHA cannot be moved. - Deploying by digest: the image you checked being swapped for another under the same tag.
- Signatures and attestations: an image that did not come from your pipeline at all (question 276). Verifying the signatures of your inputs too, such as signed base images, extends the same protection upstream.
Why automation is part of the answer. Pinned versions never update by themselves. Without a bot, pins become months-old software with known vulnerabilities. With Renovate or Dependabot, every pin is refreshed regularly, each refresh is a pull request that runs the full pipeline, and a human or an auto-merge rule decides (question 166).
278. How do you decide which security findings must block a release and which can be accepted temporarily?
The short answer
By risk, using a policy written down in advance. Block when a finding is serious, actually exploitable in your situation, and fixable now. Serious means critical severity, or known to be exploited in the wild. Exploitable means it is in a component the service really uses, on a service that is exposed. Fixable means a fix is available. Accept temporarily, with an owner and an expiry date, when there is no fix yet, the vulnerable code cannot be reached, or other controls already block the attack.
Question 137 covers the same decision for image vulnerabilities. In the pipeline, apply the same questions to every kind of finding:
| Question | Pushes towards blocking | Pushes towards accepting |
|---|---|---|
| How severe? | Critical or high | Low or medium |
| Exploited in the wild? | Listed in CISA's KEV catalogue | No known exploitation |
| Reachable? | Code path used by the service | Package present but never loaded |
| Exposed? | Internet-facing, handles sensitive data | Internal, isolated |
| Fix available? | Yes, a version bump | No fix yet |
| Other protection? | None | A firewall rule or a disabled feature blocks it |
Kinds of findings and typical rules:
- A leaked secret in the change: always block. Rotate it, even after the change is fixed (question 267).
- A static analysis finding of injection in new code: block until fixed or shown to be a false alarm.
- A dependency vulnerability: apply the table.
- A misconfiguration, such as a container running as root or a storage bucket open to the public: block for production.
Accepting is a decision, not a silence. Every accepted finding is recorded with its reason, its owner and its expiry (question 279), and the pipeline itself enforces the expiry.
279. How do you document security exceptions, owners, expiry dates, and compensating controls?
The short answer
Record each exception in a structured, versioned form, ideally as code next to the policy it overrides, so the pipeline can enforce it. Each record names the exact finding and the exact artifact or service, the reason, the risk assessment, the compensating controls that reduce the risk, the owner, the approver and an expiry date. When the expiry passes, the exception stops working automatically, and the finding blocks again until it is fixed or re-approved.
What every exception must contain:
| Field | Example |
|---|---|
| Finding | CVE-2026-12345 in libxml2 |
| Scope | Image shop/catalog-api, versions up to 2.8.x |
| Reason | No fixed version published yet |
| Risk assessment | The vulnerable function parses XML; catalog-api never parses XML from users |
| Compensating control | Upload endpoint rejects XML; the WAF blocks XML request bodies |
| Owner | catalog team, Asha |
| Approver | Security, Ravi |
| Expiry | 2026-11-30 |
A compensating control is something else that reduces the risk while the real fix is not possible, such as a firewall rule, a disabled feature, network isolation or extra monitoring.
Keep it as code, so the pipeline enforces it. Many tools accept exception files with expiry dates. For example, a Trivy ignore file in YAML form:
yaml
# .trivyignore.yaml
vulnerabilities:
- id: CVE-2026-12345
paths: ["usr/lib/x86_64-linux-gnu/libxml2.so.2"]
statement: "XML is never parsed from user input; WAF blocks XML bodies. Owner: catalog-team. Approved: security/ravi."
expired_at: 2026-11-30After expired_at, the scanner reports the finding again and the gate fails. Kyverno has a PolicyException object that does the same for cluster policies, and VEX documents can record "not affected" decisions with justifications (question 137).
Process around it:
- Exceptions are added through pull requests, approved by security, never by the requester alone.
- A dashboard lists all active exceptions and those expiring soon.
- Expiry dates are short, typically 30 to 90 days, so exceptions are reviewed rather than forgotten.