Appearance
Deploying, Config and Secrets
20. How should a production Express application be deployed?
The short answer
Build once, in CI, into an immutable artifact, usually a Docker image tagged with the commit, after linting, type checking and a security scan. Promote that same image through staging to production. Run several stateless instances across zones, behind a load balancer or reverse proxy that terminates TLS. Inject configuration and secrets from the environment and a secrets manager. Use rolling, blue-green or canary deploys with readiness checks, graceful shutdown and automatic rollback. Run database migrations as a separate, backward-compatible step. Ship logs, metrics and traces from day one.
The path from commit to production:
- CI installs with
npm ci, lints, type-checks, builds, scans dependencies, and builds the image. - The image is pushed to a registry with an immutable tag, such as the git SHA.
- Staging deploys that exact image, runs migrations and smoke checks.
- Production deploys the same image with a rolling or canary strategy.
- Readiness checks gate traffic to each new instance; old instances shut down gracefully.
- Monitoring compares error rate and latency; a bad release rolls back automatically.
| Do | Avoid |
|---|---|
| One image, promoted through environments | Rebuilding per environment |
NODE_ENV=production, npm ci --omit=dev | npm install on the server |
| Run as a non-root user | Root inside the container |
| Config from env and a secrets manager | .env files baked into the image |
| Expand-and-contract migrations | Migrations that break the running version (safe migrations) |
| Several instances, two zones | One server you SSH into to git pull |
21. How do you implement graceful shutdown during deployments?
The short answer
When the platform sends SIGTERM, the app should stop accepting new work, finish the requests already in progress, then close its resources, database pools, Redis, queue consumers, and exit, all within a time limit. In Kubernetes, first fail the readiness check so the load balancer stops sending traffic, wait a few seconds for that to take effect, then call server.close(). Set a hard timeout so a stuck request cannot block shutdown forever, and make the platform's grace period longer than that timeout. Without this, every deploy drops in-flight requests as errors.
ts
let shuttingDown = false
app.get('/health/ready', (req, res) => res.status(shuttingDown ? 503 : 200).end()) // (1)
process.on('SIGTERM', async () => {
shuttingDown = true
logger.info('SIGTERM received, draining')
await sleep(5_000) // (2)
server.close(async () => { // (3)
await Promise.allSettled([pool.end(), redis.quit(), worker.close()]) // (4)
process.exit(0)
})
server.closeIdleConnections() // (5)
setTimeout(() => { logger.error('forced exit'); process.exit(1) }, 20_000).unref() // (6)
})- Readiness reports not ready as soon as shutdown begins.
- Wait for the load balancer to notice and stop sending new requests.
- Stop accepting connections; the callback runs when in-flight requests finish.
- Close the database, cache and queue consumer cleanly, so no work is lost half-way.
- Close idle keep-alive connections now, instead of waiting for them to time out.
- A hard limit of 20 seconds. Kubernetes'
terminationGracePeriodSecondsmust be longer, for example 30.
The details of signals and the Node APIs are in graceful shutdown in Node.
22. What should readiness and liveness endpoints report?
The short answer
Liveness answers "is this process stuck and in need of a restart?" It should be cheap and local: if the event loop can respond, it is alive. Never check the database in liveness, or a database outage makes the platform restart every healthy instance in a loop. Readiness answers "should this instance receive traffic right now?" It may check what the instance needs to serve requests, such as a database connection and finished startup, and must return 503 during startup and shutdown. A startup probe covers slow boots, so liveness does not kill an instance still warming up.
| Probe | Question | Should check | Should not check | On failure |
|---|---|---|---|---|
| Liveness | Is it stuck? | Process responds | Database, other services | Container restarted |
| Readiness | Can it take traffic? | Startup done, DB reachable, not shutting down | Every optional dependency | Removed from load balancing |
| Startup | Has it finished booting? | Init complete | Liveness waits |
ts
let ready = false
app.get('/health/live', (req, res) => res.status(200).send('ok')) // (1)
app.get('/health/ready', async (req, res) => {
if (!ready || shuttingDown) return res.status(503).end() // (2)
try {
await pool.query('SELECT 1') // (3)
res.status(200).send('ready')
} catch {
res.status(503).send('database unavailable')
}
})
await runStartupTasks()
ready = true- Liveness: answering at all proves the event loop works.
- Not ready while starting or draining.
- A tiny check of the one dependency every request needs. If the recommendations service is down, the instance should still serve, so it is not checked here.
Why a bad liveness check causes outages: a slow database makes liveness fail everywhere; the platform restarts every instance; the restarts add load; the outage gets worse. More in probes in Kubernetes.
23. How do you manage environment-specific configuration?
The short answer
Keep code identical across environments and change only configuration, read from environment variables, as the twelve-factor app approach recommends. Validate all of it at startup against a schema, so a missing or malformed value stops the process with a clear error before it takes traffic. Keep non-secret config in version control, per environment, such as Kubernetes ConfigMaps or Helm values, and secrets in a secrets manager. Load it once into a typed, frozen object, and never scatter process.env reads through the code.
ts
import { z } from 'zod'
const Config = z.object({
NODE_ENV: z.enum(['development', 'staging', 'production']),
PORT: z.coerce.number().default(3000), // (1)
DATABASE_URL: z.string().url(),
REDIS_URL: z.string().url(),
LOG_LEVEL: z.enum(['debug', 'info', 'warn', 'error']).default('info'),
FEATURE_NEW_CHECKOUT: z.coerce.boolean().default(false), // (2)
})
export const config = Object.freeze(Config.parse(process.env)) // (3)- Environment values are strings;
coerceturns"3000"into a number. - Feature flags as config, so staging can turn a feature on before production. Note that
z.coerce.boolean()treats any non-empty string, even"false", astrue; usez.enum(['true', 'false']).transform((v) => v === 'true')if you need strict parsing. - Parse once, fail fast, freeze. Import
configeverywhere instead of readingprocess.env.
| Kind of setting | Where it lives |
|---|---|
| Ports, log level, feature flags, URLs of other services | Env vars from ConfigMaps or Helm values, in git |
| Passwords, API keys, signing keys | Secrets manager, injected at runtime |
| Per-developer overrides | .env.local, git-ignored |
24. How do you store and rotate application secrets in production?
The short answer
Store secrets in a secrets manager or vault, never in code, images or git. Give each service its own secrets and access through its workload identity, so no "master password" is needed to fetch them. Inject them at runtime as env vars or files, or fetch them at startup. Rotate on a schedule and immediately after any suspected leak, using an overlap: create the new secret, make consumers accept or use it, then revoke the old one. Prefer short-lived, automatically issued credentials, such as dynamic database users or cloud tokens, over long-lived ones.
The full answers are elsewhere in this guide:
- Protecting secrets and env vars, including leaks to git and images: protecting secrets.
- How a vault is used, and the "secret zero" problem: secret management.
- Rotation without downtime, with the two-key overlap: key rotation.
Picking up a rotated secret without a restart:
| Approach | How the app sees the new value |
|---|---|
| Env var | Only on restart; roll the deployment after rotating |
| Mounted file, from a CSI driver or sidecar | File changes; app re-reads it or watches it |
| SDK fetch with a cache | Re-fetch when the cached value is older than a few minutes, or on an auth failure |
25. How do you rotate signing keys and certificates without downtime?
The short answer
Always run two keys at once during the change. For JWT signing keys: publish the new public key in your JWKS beside the old one, wait for verifiers to refresh their cache, switch signing to the new key, wait until the longest-lived old token has expired, then remove the old key; tokens name their key with kid, so both verify meanwhile. For TLS certificates: automate renewal well before expiry, install the new certificate with its full chain, reload the server gracefully rather than restarting, verify from outside, and alert on anything close to expiring.
Both procedures, with code, are covered in key rotation and certificate rotation. For SAML signing certificates, where every customer's SP must load the new certificate, see SAML certificate rotation.
| Credential | Overlap by | Remove old when |
|---|---|---|
| JWT signing key | Two keys in the JWKS, chosen by kid | Longest token lifetime has passed |
| TLS certificate | New certificate loaded before the old expires | Old one is no longer served |
| Session-cookie secret | Accept an array of secrets, sign with the newest | Longest session lifetime has passed |
| Database password | Two users, or a second password | All instances use the new one |
26. How should application secrets be managed in a cloud environment?
The short answer
Use the cloud's managed secret store, AWS Secrets Manager, Azure Key Vault or Google Secret Manager, and give each workload a cloud identity, such as an IAM role, a managed identity or workload identity, with permission to read only its own secrets. Then no credentials are stored anywhere to bootstrap access. Encrypt secrets with keys in the cloud KMS, turn on audit logging of every read, enable automatic rotation where supported, such as for managed database passwords, and never put secrets in plain environment variables of task definitions, function settings or infrastructure state without encryption.
| Need | AWS | Azure | Google Cloud |
|---|---|---|---|
| Store secrets | Secrets Manager, Parameter Store | Key Vault secrets | Secret Manager |
| Workload identity | IAM roles for tasks and pods | Managed identities | Service accounts, workload identity |
| Encryption keys | KMS | Key Vault keys | Cloud KMS |
| Into Kubernetes | Secrets Store CSI driver, External Secrets Operator | Same | Same |
More on how cloud identities remove stored secrets: managed identities, and the DevOps view in cloud secrets.
27. How do you configure health checks and rollback strategies during deployment?
The short answer
Health checks decide when a new instance may receive traffic, so the deploy waits for readiness before moving on, and stops if new instances never become ready. Choose a strategy: rolling, replacing instances a few at a time; blue-green, starting a full new set and switching traffic at once; or canary, sending a small share of traffic to the new version first. Watch error rate and latency during the rollout, and roll back automatically if they worsen, by redeploying the previous image tag. Keep database changes backward compatible so rolling back the code is always safe.
A Kubernetes rolling update that cannot break the service:
yaml
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate: { maxUnavailable: 0, maxSurge: 2 } # (1)
template:
spec:
terminationGracePeriodSeconds: 30
containers:
- name: orders-api
image: registry.example/orders-api:3f9a1c2 # (2)
readinessProbe: { httpGet: { path: /health/ready, port: 3000 }, periodSeconds: 5, failureThreshold: 2 } # (3)
livenessProbe: { httpGet: { path: /health/live, port: 3000 }, periodSeconds: 10, failureThreshold: 3 }- Never remove an old pod before a new one is ready, and create at most two extra at a time.
- An immutable tag, the commit SHA, so rolling back means pointing at the previous tag.
- Traffic only after readiness passes. If new pods never become ready, the rollout stalls instead of replacing healthy pods.
Rolling back: kubectl rollout undo deployment/orders-api, or redeploy the previous tag from CI. With Argo Rollouts or Flagger, a canary is promoted or aborted automatically based on metrics.
| Strategy | Extra capacity | Rollback speed | Risk |
|---|---|---|---|
| Rolling | A little | Minutes | Old and new run side by side |
| Blue-green | Double, briefly | Seconds: switch back | Cost; database shared by both |
| Canary | A little | Seconds: send traffic back | Needs good metrics |
Strategy details: release strategies.