Skip to content

Interview PrepCI/CD

Jenkins ​

233. What is a Jenkinsfile, and how does it define a Jenkins pipeline? ​

The short answer

A Jenkinsfile is a text file, usually at the root of the repository, that describes a Jenkins pipeline in a Groovy-based language. Jenkins reads it from the repository for each branch and pull request, and runs the stages and steps it lists. Because it lives in Git, the pipeline is reviewed, versioned and changed together with the code it builds.

Groovy is a programming language that runs on the Java virtual machine. Jenkins uses it as the base of its pipeline language. You mostly use a small, structured part of it.

A typical Jenkinsfile for the shop's API:

groovy
pipeline {
    agent { label 'linux && docker' }                        // (1)
    options {
        timeout(time: 30, unit: 'MINUTES')                   // (2)
        disableConcurrentBuilds()
    }
    environment {
        REGISTRY = 'myacr.azurecr.io'                        // (3)
    }
    stages {
        stage('Install') {
            steps { sh 'npm ci' }                            // (4)
        }
        stage('Checks') {
            parallel {                                       // (5)
                stage('Lint')  { steps { sh 'npm run lint' } }
                stage('Unit')  { steps { sh 'npm run unit' } }
            }
        }
        stage('Image') {
            when { branch 'main' }                           // (6)
            steps {
                sh 'docker build -t $REGISTRY/shop/checkout-api:$GIT_COMMIT .'
            }
        }
    }
    post {
        failure { echo "Build failed: ${env.BUILD_URL}" }    // (7)
        always  { cleanWs() }
    }
}
  1. Run on any agent that carries both the linux and docker labels (question 236).
  2. Stop the whole pipeline if it runs longer than 30 minutes, and never run two builds of this branch at once.
  3. A variable available to every step.
  4. A step that runs a shell command.
  5. Two stages that run at the same time.
  6. Only build the image on the main branch.
  7. Actions that run after the stages, depending on the result. cleanWs() deletes the workspace, the folder where the build ran, so the next build starts clean.

How Jenkins finds it. A Multibranch Pipeline job, or an Organization Folder for a whole GitHub organisation, scans the repository, finds every branch and pull request that contains a Jenkinsfile, and creates a pipeline for each. New branches appear automatically.

234. What is the difference between Declarative Pipeline and Scripted Pipeline in Jenkins? ​

The short answer

Declarative Pipeline is a structured format with fixed sections such as agent, stages, steps and post, which Jenkins checks before running and which is easy to read. Scripted Pipeline is plain Groovy code inside a node block, where you can write any logic you like, with loops and conditions, but it is harder to read and to keep consistent. Declarative is the default choice today, with small script blocks where real logic is needed.

groovy
pipeline {
    agent { label 'linux' }
    stages {
        stage('Build') {
            steps { sh 'npm ci && npm run build' }
        }
    }
    post {
        always { junit 'reports/*.xml' }
    }
}
groovy
node('linux') {
    try {
        stage('Build') {
            sh 'npm ci && npm run build'
        }
    } finally {
        junit 'reports/*.xml'
    }
}
DeclarativeScripted
StructureFixed sectionsFree-form Groovy
Checked before runningYes, syntax and structureMostly not
ReadabilityHighDepends on the author
FlexibilityLimited, use script {} for logicUnlimited
Restart from a stageSupportedNot supported
Built-in post, when, optionsYesWrite it yourself

Mixing them. Declarative pipelines can contain a script { } block inside steps, where full Groovy is allowed. That gives you structure by default and flexibility where you need it:

groovy
steps {
    script {
        def services = ['checkout-api', 'catalog-api']
        for (s in services) { sh "make build SERVICE=${s}" }
    }
}

When Scripted still makes sense: very dynamic pipelines that build their stages from data, and older pipelines nobody wants to rewrite. For anything new, Declarative is easier for the next person to understand.

235. What are Jenkins stages, steps, agents, post actions, and environment blocks? ​

The short answer

A stage is a named phase of the pipeline, such as "Build" or "Deploy", shown as a column in the Jenkins interface. A step is one action inside a stage, such as running a shell command. An agent says where the pipeline or a stage runs. Post actions run after the pipeline or a stage, depending on how it ended, for example on failure or always. An environment block sets variables, including credentials, for the steps.

groovy
pipeline {
    agent none                                               // (1)
    environment {
        NODE_ENV = 'production'                              // (2)
        NPM_TOKEN = credentials('npm-token')                 // (3)
    }
    stages {
        stage('Build') {                                     // (4)
            agent { docker { image 'node:20-slim' } }        // (5)
            steps {                                          // (6)
                sh 'npm ci'
                sh 'npm run build'
            }
            post {
                failure { echo 'Build stage failed' }        // (7)
            }
        }
    }
    post {
        success  { echo 'Pipeline succeeded' }               // (8)
        failure  { echo 'Pipeline failed' }
        always   { echo 'Runs no matter what' }
    }
}
  1. agent none at the top means "no default". Each stage picks its own agent.
  2. An environment variable visible to every step.
  3. credentials() reads a stored credential and exposes it as an environment variable. Jenkins hides its value in the logs (question 239).
  4. A stage: a named phase.
  5. This stage's agent: run inside a node:20-slim container. The tools come from the image, not from the agent machine, which avoids "it works on agent 3 but not agent 7" problems (question 242).
  6. Steps: the actual commands, run one after another.
  7. A stage-level post action, only for this stage.
  8. Pipeline-level post actions. Common conditions are success, failure, unstable, changed, aborted and always.

Two useful extras: options { } sets pipeline-wide behaviour such as timeouts and retries, and when { } decides whether a stage runs, for example only on a certain branch or when certain files changed.

236. What is the difference between a Jenkins controller and an agent? ​

The short answer

The controller is the central Jenkins server: it serves the web interface, stores the configuration, the jobs, the build history and the credentials, runs the plugins, and decides which build runs where. Agents are the machines that actually run the builds. The controller hands work to agents by matching labels, and the agents stream logs and results back. For security and stability, builds should run on agents, never on the controller itself.

controllerweb interface, job configbuild history, credentialsplugins, scheduling0 executors: runs no buildsagent: linux, dockera VM, 4 executorsagent: windowsfor .NET Framework buildsagents: Kubernetes podscreated per build, then deletedmatched by label
One brain, many hands. The controller decides and records. Agents do the work. Kubernetes pod agents are created for one build and removed afterwards.

The controller:

  • Hosts the web interface and the API.
  • Stores job configuration, build history and logs in its home folder, JENKINS_HOME.
  • Holds the credentials store, encrypted with its own keys.
  • Runs the plugins, which is where most of Jenkins' features come from.
  • Keeps the build queue and assigns builds to agents.

Agents:

  • Connect to the controller, either the controller connects out over SSH, or the agent connects in, called an inbound agent.
  • Each has a number of executors, slots for builds running at the same time.
  • Carry labels, such as linux, docker, gpu or windows, which pipelines use to ask for the right kind of machine.

Why no builds on the controller. A build runs whatever code is in the repository. On the controller, that code could read the controller's files, including every stored credential and the keys that decrypt them. Setting the controller's own executors to zero keeps build code away from all of that. It also keeps a heavy build from slowing the web interface for everyone.

237. How do you provision Jenkins agents safely and scale them for parallel builds? ​

The short answer

Prefer short-lived cloud agents created per build, such as Kubernetes pods from the Kubernetes plugin, built from pinned, patched images, so every build starts clean and capacity grows with the queue. Keep untrusted builds, such as pull requests from forks, on separate agent pools with no access to secrets or production. Give agents as few credentials as possible, never run builds on the controller, and set resource limits so one build cannot starve the others.

Scaling with the Kubernetes plugin. Jenkins creates a pod for each build, from a pod template, and deletes it when the build ends:

groovy
pipeline {
    agent {
        kubernetes {
            yaml '''
              apiVersion: v1
              kind: Pod
              spec:
                containers:
                  - name: node
                    image: node:20-slim
                    command: ["sleep"]
                    args: ["infinity"]
                    resources:
                      requests: { cpu: "1", memory: 1Gi }
                      limits:   { memory: 2Gi }
            '''
        }
    }
    stages {
        stage('Build') {
            steps { container('node') { sh 'npm ci && npm run build' } }
        }
    }
}

The pod has a node container that just waits, sleep infinity, and the build's steps run inside it with container('node'). A second container, such as a rootless BuildKit container for image builds (question 469), can be added to the same pod. The cluster's capacity, and its autoscaler, decide how many builds run at once.

Making it safe:

  1. Short-lived agents. Nothing survives between builds: no files, no credentials, no attacker (question 183).
  2. Separate pools by trust. Builds of reviewed code from main can use agents with deploy access. Pull requests, especially from outside contributors, get a separate pool with none.
  3. Pinned images, rebuilt regularly with security fixes.
  4. Minimal credentials. Bind credentials only in the stages that need them (question 239), and use short-lived cloud tokens where possible.
  5. Limits. CPU and memory limits per build pod, and a timeout per pipeline.
  6. Controller protection. Zero executors on the controller, and Jenkins' agent-to-controller security left on, which stops agents from asking the controller to read its files.

Static agents still have a place, for special hardware or licensed software, but they need regular cleaning and patching, and should be treated as long-lived servers.

238. What are Jenkins Shared Libraries, and when are they preferable to copying pipeline code between repositories? ​

The short answer

A Shared Library is a separate Git repository of reusable pipeline code that any Jenkinsfile can load with one line, such as @Library('shop-pipeline@v3') _. It can define new pipeline steps, such as buildNodeService(), and helper classes. It is preferable whenever many repositories need the same pipeline logic, because a fix or improvement is made once and versioned, instead of being copied and slowly drifting apart in dozens of places.

The layout of a shared library repository:

text
shop-pipeline/
  vars/
    buildNodeService.groovy      a new step anyone can call
  src/com/shop/
    Registry.groovy              helper classes
  resources/
    sonar.properties             files the library can load

A step defined in the library:

groovy
// vars/buildNodeService.groovy
def call(Map cfg) {
    pipeline {
        agent { label 'linux && docker' }
        stages {
            stage('Install') { steps { sh 'npm ci' } }
            stage('Checks')  { steps { sh 'npm run lint && npm run unit' } }
            stage('Image') {
                when { branch 'main' }
                steps { sh "docker build -t myacr.azurecr.io/shop/${cfg.name}:\$GIT_COMMIT ." }
            }
        }
    }
}

A whole Jenkinsfile in a service repository:

groovy
@Library('shop-pipeline@v3') _
buildNodeService(name: 'checkout-api')

The @v3 pins the library to a tag, so a library change cannot surprise a service until it chooses to upgrade. The trailing _ is required syntax when the annotation has nothing else to attach to.

Why not copy and paste?

  1. Drift. Thirty copies become thirty slightly different pipelines within months.
  2. Slow fixes. A security fix to the pipeline needs thirty pull requests instead of one release of the library.
  3. No standards. Every team solves the same problems differently.

The trade-off: a shared library is shared risk. A bug in it can break every team's builds at once. So version it with tags, test changes on a few pilot repositories first, and keep a changelog (question 263).

239. How do you store and use credentials in Jenkins without printing them in logs? ​

The short answer

Store them in Jenkins' credentials store, which encrypts them, or in an external vault through a plugin, and scope them to the folder that needs them. Bind them only for the steps that use them, with withCredentials or the credentials() helper, so they exist as environment variables for those steps only. Jenkins masks known secret values in the console log, but masking is not protection: never print secrets, never turn on shell tracing around them, and never put them inside Groovy double-quoted strings.

Binding a credential to a few steps only:

groovy
stage('Publish') {
    steps {
        withCredentials([string(credentialsId: 'npm-token', variable: 'NPM_TOKEN')]) {   // (1)
            sh 'echo "//npm.pkg.github.com/:_authToken=${NPM_TOKEN}" > .npmrc'          // (2)
            sh 'npm publish && rm -f .npmrc'
        }
    }
}
  1. The secret npm-token is exposed as the environment variable NPM_TOKEN only inside this block.
  2. Single quotes. The Groovy string is passed to the shell unchanged, and the shell expands ${NPM_TOKEN}. The secret never becomes part of the Groovy string.

The double-quote trap:

groovy
sh "curl -H 'Authorization: Bearer ${NPM_TOKEN}' https://api.example.com"   // unsafe

With double quotes, Groovy replaces ${NPM_TOKEN} before the shell runs, so the secret becomes part of the command text itself. The command text can then appear in places masking does not cover, such as the list of running processes on the agent. Jenkins prints a warning when it sees this: "A secret was passed to "sh" using Groovy String interpolation, which is insecure."

Other leaks to avoid:

  1. set -x in a shell step prints every command with its values.
  2. Transformed secrets. Masking hides the exact string. A secret encoded in base64, or split into parts, is not recognised and appears in plain text.
  3. Files left behind, such as the .npmrc above, which is why the example deletes it. Short-lived agents help here too.

Storage options:

  • Jenkins' own store, encrypted with keys kept on the controller. Scope credentials to folders, so one team's jobs cannot use another team's secrets.
  • An external vault, such as HashiCorp Vault or Azure Key Vault, through plugins, so secrets are not stored in Jenkins at all.
  • Best of all for cloud access: short-lived tokens through OIDC instead of stored keys (question 268).

240. How do you manage plugin updates, backups, and recovery for a production Jenkins installation? ​

The short answer

Treat Jenkins as production infrastructure. Run the long-term-support release, pin the plugin list in a file, and test core and plugin upgrades on a staging Jenkins before applying them. Keep the configuration in Git using Configuration as Code, and pipelines in Jenkinsfiles, so a new controller can be rebuilt from scratch. Back up the Jenkins home folder, especially its secrets, regularly and off the machine, and practise restoring it.

1. Plugins are the biggest risk. Most of Jenkins' features, and most of its security advisories, come from plugins. Some are poorly maintained, and one update can break others.

  • Keep a pinned list, such as a plugins.txt file, used to build the controller's image.
  • Upgrade on a schedule, and quickly when a security advisory affects you.
  • Test upgrades on a staging Jenkins with the same plugins and a sample of real pipelines.
  • Remove plugins nobody uses.

2. Use the LTS line. The long-term support release of Jenkins changes more slowly and gets backported fixes.

3. Configuration in Git. Jenkins Configuration as Code, usually called JCasC, describes the controller's configuration in YAML: security settings, agents, clouds, credential references and tool installations. Jobs come from Jenkinsfiles and Multibranch or Organization Folder jobs. Together, a new controller can be built from Git alone.

yaml
# jenkins.yaml, read at start-up
jenkins:
  numExecutors: 0
  securityRealm:
    local:
      allowsSignup: false
unclassified:
  location:
    url: https://jenkins.shop.internal/

4. Backups. JENKINS_HOME holds everything that is not in Git:

Back upWhy
config.xml and other configuration filesSettings not managed by JCasC
jobs/Build history, if you need it
credentials.xmlStored credentials, encrypted
secrets/, including master.keyWithout these, the stored credentials cannot be decrypted
workspace/Usually not, it is disposable

Store backups off the machine, and protect them as carefully as the credentials they contain.

5. Recovery drills. Restore into a fresh controller regularly, and time it. A backup nobody has restored is only a hope (question 306).

241. What would you check if a Jenkins job remains queued and never starts? ​

The short answer

Look at why Jenkins says it is waiting: hover over the queued item, or check the queue's explanation. Usually no online agent has the requested label, all matching executors are busy, or the cloud plugin is failing to create new agents. Other causes are a lock or throttle waiting for another build, "disable concurrent builds" waiting for the previous run, Jenkins being in quiet-down mode before a restart, and agents taken offline automatically because their disk is full.

Read Jenkins' own explanation first. Every queued item shows a reason, such as:

MessageWhat to check
"There are no nodes with the label 'gpu'"A typo in the label, or no agent with it
"Waiting for next available executor on 'linux'"All matching agents are busy or offline
"All nodes of label 'k8s' are offline"Agents disconnected, or the cloud cannot create pods
"Build #41 is already in progress"disableConcurrentBuilds() is waiting for the previous build
"Waiting for resource 'staging-db'"A lock step waiting for another build to release it
"Jenkins is about to shut down"Quiet-down mode before a restart

Then check the agents, on the Nodes page:

  1. Offline agents. Jenkins takes agents offline automatically when their free disk space or temporary space falls below a threshold. The node page shows why. Clearing old workspaces usually brings them back.
  2. Disconnected agents. Network problems, an expired SSH key, or an inbound agent that stopped.
  3. Executors set to zero on the agents you expected to use.

If agents come from a cloud, such as the Kubernetes plugin:

  1. Look at the controller's log for provisioning errors.
  2. In the cluster, look at the agent pods' events: an image that cannot be pulled, a namespace quota that is full, a service account that lacks permission, or a node pool that cannot scale (question 408).
  3. Check the plugin's limit on how many agents it may create at once.

Prevent it next time: alert when the queue time goes above a few minutes, which is one of the most useful CI health signals (question 266).

242. How would you diagnose a Jenkins pipeline that works on one agent but fails on another? ​

The short answer

Something about the two machines differs, so compare them. Check tool versions and the PATH, the operating system and the processor type. Check installed packages and leftover files in the workspace. Check the user and permissions, and network access to registries and proxies. Check stored keys and known hosts, disk space, and the locale or time zone. The lasting fix is to stop depending on the agent at all: run the build inside a pinned container image, clean the workspace, and use labels to describe what each agent really provides.

Step 1: compare the two machines. Add a short diagnostic step and run it on both:

groovy
stage('Diagnose') {
    steps {
        sh '''
          uname -a; id; echo "PATH=$PATH"
          node --version || true; java -version 2>&1 || true; docker version || true
          df -h .; locale; date
        '''
    }
}

The usual differences:

  1. Tool versions. Node.js 18 on one agent, 20 on the other. Or a tool installed on one only.
  2. Operating system or processor type. An ARM agent or a different Linux distribution.
  3. A dirty workspace. The working agent has a leftover file from an earlier build that the build quietly needs. The failing agent starts clean, and tells the truth.
  4. User and permissions. A different user, or files owned by another user from a previous build.
  5. Network. One agent can reach the private registry, the other sits behind a proxy or firewall.
  6. Keys. One agent's SSH settings already trust a Git server's key, the other's do not, so git clone asks for confirmation and hangs.
  7. Disk space, and locale or time zone, which change how dates and sorting behave.

Step 2: fix it so it cannot recur.

  • Run steps in a container: agent { docker { image 'node:20.18-slim' } }, or a Kubernetes pod template. The tools then come from the image, identical everywhere.
  • Clean the workspace: cleanWs() in post { always }, or use short-lived agents.
  • Honest labels: a label such as docker should mean "Docker is installed and working", and nothing else.
  • Pin everything the build relies on, as in question 225.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet