Skip to content

Interview PrepObservability

Observability and SRE ​

623. What is the difference between monitoring and observability? ​

The short answer

Monitoring watches for problems you already know to look for: you decide in advance which numbers matter, such as error rate or disk space, and alert when they cross a line. Observability is how well you can understand problems you did not predict, by asking new questions of the data your system produces, such as "why are only card payments from Android slow since 14:10?". Monitoring tells you something is wrong. Observability helps you find out why.

An example. The checkout error rate alert fires. That is monitoring doing its job. Now you need to know which requests fail, for which customers, at which step, and since which release. If your logs, metrics and traces carry enough detail, such as the route, the payment method, the app version and a trace ID, you can slice the data until the answer appears. That is observability.

MonitoringObservability
Answers"Is it broken?""Why is it broken?"
QuestionsDecided in advanceAsked during the investigation
Built fromDashboards and alerts on known signalsRich, connected logs, metrics and traces
HandlesKnown failure modesNew, surprising ones

They are not rivals. Observability is built on good telemetry, the data a system emits about itself. Monitoring is one use of that data. You need both: alerts to wake you up, and rich data to find the cause quickly (question 638).

What makes a system observable:

  1. Structured events with useful fields, not free text only (question 633).
  2. Shared IDs, such as a trace ID in every log line, so you can jump between signals (question 634).
  3. Consistent names for services, routes and versions across all data.
  4. High-detail fields where they help, such as customer or order ID in logs and traces, though not in metrics labels (question 627).

624. What are logs, metrics, and traces, and when do you use each? ​

The short answer

Metrics are numbers measured over time, such as requests per second or memory used: cheap to store, fast to query, and ideal for dashboards and alerts. Logs are records of individual events, with details, such as "payment 8812 failed: card declined": best for understanding exactly what happened. Traces follow one request across all the services it touches, timing each step, which shows where time was spent or where it failed. Alerts usually start from metrics, and investigations move into traces and logs.

metricsnumbers over timetracescheckout 840 mscartpaymentone request, step by steplogs14:10:02 INFO order 881214:10:03 ERROR card declined, trace=4bf9what exactly happened① the metric alerts ② the trace finds the slow step ③ its logs explain why
From "something is wrong" to "this is why". ① A metric shows the problem and fires the alert. ② A trace of a failing request shows which step failed. ③ The logs with the same trace ID give the exact error.
MetricsLogsTraces
ShapeA number, with labels, over timeA record per eventA tree of timed steps per request
Examplehttp_requests_total{status="500"}{"level":"error","msg":"card declined"}checkout → cart → payment
Cost per unitVery lowMedium to highMedium; usually sampled
Best forAlerts, trends, capacityExact details of an eventWhere time goes, which hop fails
Weak atIndividual eventsTrends across millions of eventsLong-term trends

Two more signals you will hear about:

  • Events, such as Kubernetes events or deployment markers, show when something changed, which often explains a graph (question 424).
  • Profiles show which functions in the code use CPU or memory, for performance work.

Connect them. Put the same trace ID in logs and traces, and link from dashboards to traces. Then one click takes you from a red graph to the failing request and its error message.

625. How does Prometheus work? ​

The short answer

Prometheus is an open-source monitoring system that pulls metrics. Every application or exporter publishes its current numbers as text on an HTTP endpoint, usually /metrics. Prometheus discovers these targets, for example from the Kubernetes API, scrapes them every few seconds, and stores each value with its labels and timestamp in its own time-series database. You query it with PromQL, rules turn queries into alerts, and Alertmanager groups those alerts and sends them to people.

checkout pod /metricsnode exporter /metricskube-state-metricsKubernetes APIPrometheus① discover targets② scrape every 30 s③ store time series④ evaluate alert rulesGrafanadashboardsAlertmanager⑤ group and notifyPromQL
Prometheus pulls. ① It finds targets through the Kubernetes API. ② It scrapes each target's metrics endpoint on a timer. ③ It stores the values. ④ It evaluates alert rules. ⑤ Firing alerts go to Alertmanager, which notifies people. Grafana queries it for dashboards.

What a scrape returns, plain text from the checkout service:

text
# HELP http_requests_total Requests handled.
# TYPE http_requests_total counter
http_requests_total{route="/checkout",status="200"} 18233
http_requests_total{route="/checkout",status="500"} 41
# HELP process_resident_memory_bytes Memory in use.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.83e+08

Each line is one time series: a metric name, a set of labels in braces, and the current value. Prometheus stores a new sample of each series at every scrape.

Why pull instead of push?

  1. Prometheus knows what should exist. A target that stops answering shows up immediately as down, through the built-in up metric.
  2. Apps stay simple. They only keep current numbers in memory and answer when asked.
  3. No overload from senders. Prometheus controls the rate.

Short batch jobs that finish before a scrape can push to a Pushgateway, which Prometheus then scrapes.

In Kubernetes, the Prometheus Operator adds resources such as ServiceMonitor, which say "scrape every Service with label app=checkout on port metrics". Teams add their own monitoring without editing Prometheus's configuration.

Its limits. One Prometheus server stores data on its local disk, typically for days or weeks. For long-term storage, many clusters, and high availability, teams use remote write to systems such as Thanos or Grafana Mimir, or a managed Prometheus service from their cloud.

626. How do you write basic PromQL queries such as rate, sum, and percentiles? ​

The short answer

Counters only ever go up, so you almost never graph them directly: rate() turns a counter into a per-second speed over a time window, such as rate(http_requests_total[5m]). sum by (label) adds series together, grouped by the labels you keep. For latency percentiles, histograms store counts in buckets, and histogram_quantile(0.99, ...) estimates the 99th percentile from the bucket rates. Error rate is failed requests divided by all requests.

Requests per second for checkout:

text
sum(rate(http_requests_total{job="checkout"}[5m]))

Read from the inside out:

  1. http_requests_total{job="checkout"} picks every request-counter series from the checkout service, one per route and status.
  2. [5m] takes the last five minutes of samples for each.
  3. rate(...) turns each into an average per-second increase. It also handles counter resets, when a pod restarts and its counter starts again from zero.
  4. sum(...) adds all series into one number.

Error rate, as a fraction:

text
  sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
/
  sum(rate(http_requests_total{job="checkout"}[5m]))

status=~"5.." is a regular expression matching any status from 500 to 599. A result of 0.004 means 0.4% of requests fail.

Requests per second by route:

text
sum by (route) (rate(http_requests_total{job="checkout"}[5m]))

by (route) keeps the route label and adds up everything else, giving one line per route.

The 99th percentile of latency:

text
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))
)

A histogram stores counts in buckets, each with a label le, "less than or equal", such as "requests faster than 0.25 s". The query takes the rate of each bucket, adds them across pods while keeping le, then estimates the value below which 99% of requests fell. A result of 0.82 means 99% of requests finished within 820 milliseconds (question 627).

Common mistakes:

MistakeProblem
Graphing a raw counterA line that only climbs; resets look like crashes
sum before rateCounter resets in one pod corrupt the result
Averaging percentiles across podsMathematically wrong; aggregate buckets, then take the quantile
A window shorter than two scrapesGaps and noisy graphs

627. What are the metric types, and what is cardinality? ​

The short answer

Prometheus has four types. A counter only goes up, such as requests served. A gauge goes up and down, such as memory in use. A histogram counts observations into buckets, such as request durations, so percentiles can be estimated across many pods. A summary calculates percentiles inside each process, which cannot be combined across pods. Cardinality is the number of unique label combinations, each of which is a separate series, so a label with many possible values, such as user ID, can create millions of series and overload the system.

In code, for the checkout service in TypeScript with prom-client:

ts
import client from 'prom-client'

export const ordersPlaced = new client.Counter({                      // (1)
  name: 'orders_placed_total',
  help: 'Orders successfully placed',
  labelNames: ['payment_method'],
})

export const cartsInProgress = new client.Gauge({                     // (2)
  name: 'carts_in_progress',
  help: 'Carts currently being checked out',
})

export const requestDuration = new client.Histogram({                 // (3)
  name: 'http_request_duration_seconds',
  help: 'Time to handle a request',
  labelNames: ['route', 'status'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5],
})

// in a request handler:
const stop = requestDuration.startTimer({ route: '/checkout' })
// ... handle the request ...
ordersPlaced.inc({ payment_method: 'card' })                          // (4)
stop({ status: '200' })                                               // (5)
  1. A counter, ending in _total by convention. It only increases. Queries use rate() on it.
  2. A gauge, which goes up and down, set to the current value.
  3. A histogram with chosen bucket edges in seconds. Pick edges around your targets, such as 0.25 s if the goal is "under 250 ms".
  4. Count one order paid by card.
  5. Stop the timer, which records the duration into the right bucket, with the status label.

The service exposes all of this on /metrics for Prometheus to scrape (question 625).

Cardinality, with numbers. Every unique combination of label values is a separate time series:

text
routes (20) × status codes (8) × pods (30) = 4,800 series

That is fine. Now add a user_id label with 2 million users:

text
4,800 × 2,000,000 = 9,600,000,000 series

That is not fine. Memory explodes, queries crawl, and managed services bill per series.

Rules for labels:

Good labelsBad labels
Route template: /orders/:idThe real path: /orders/8812
Status code, methodUser ID, order ID, email
Service, version, regionFull error messages
Payment methodTimestamps, random IDs

Details such as user and order IDs belong in logs and traces, which are designed for them (question 624).

628. What makes a good alert, and how do you avoid alert fatigue? ​

The short answer

A good alert means a person must act now, on something users feel. Page on symptoms, such as errors and slow responses breaking your SLO, not on causes like high CPU that may be harmless. Every page should be urgent, actionable, and link to a runbook, a short guide with first steps. Send less urgent problems to a ticket queue, keep the rest on dashboards, group related alerts, and regularly delete alerts nobody acts on. Too many alerts teach people to ignore all of them.

Alert fatigue is what happens when people get so many alerts, most of them unimportant, that they stop reacting, and miss the one that matters.

Symptom versus cause:

AlertTypePage someone?
Checkout error rate above SLO target for 5 minutesSymptom: users fail to payYes
99th percentile latency above 2 s for 10 minutesSymptom: users waitYes
Node CPU at 85%Cause, maybe harmlessNo: dashboard
One pod restartedCause; Kubernetes handles itNo
Disk will be full in 4 hours at current rateCause, but certain harm soonYes, or a ticket if days away
Certificate expires in 14 daysFuture harmTicket

What a good alert carries:

  1. A clear title: "Checkout error rate 3.2%, SLO 0.1%".
  2. Impact: who is affected and how.
  3. A runbook link: what to check first.
  4. Links to the dashboard and recent deployments.
  5. Severity and routing: who gets it, and how loudly.

An alert rule in Prometheus:

yaml
groups:
  - name: checkout
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="checkout"}[5m])) > 0.02
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Checkout errors above 2% for 5 minutes"
          runbook: "https://runbooks.shop.example/checkout-errors"

for: 5m means the condition must stay true for five minutes before the alert fires, which ignores brief blips. The severity label lets Alertmanager route pages to on-call and warnings to a chat channel.

Keep the alert set healthy:

  • Review every page in a weekly or monthly meeting: did someone need to act? If not, change or delete the alert.
  • Group related alerts, so one database failure is one notification, not fifty.
  • Silence during planned work, with an end time.
  • Prefer SLO burn-rate alerts for user-facing services, which fire in proportion to how fast users are being hurt (question 631).

629. What are the golden signals, and the RED and USE methods? ​

The short answer

They are checklists of what to measure. The four golden signals, from Google's SRE book, are latency, traffic, errors and saturation. RED, for services, is rate of requests, errors, and duration. USE, for resources such as CPUs, disks and queues, is utilisation, saturation and errors. Use RED or the golden signals for every service users call, and USE for the machines and resources underneath. Measure latency with percentiles, such as the 99th, not averages.

MethodForMeasures
Golden signalsAny user-facing systemLatency, traffic, errors, saturation
REDRequest-driven servicesRate, errors, duration
USEResources: CPU, memory, disk, network, queuesUtilisation, saturation, errors

What each word means:

  • Latency, or duration: how long requests take. Measure successful and failed requests separately, because a fast error can make latency look better.
  • Traffic, or rate: how much demand, such as requests per second.
  • Errors: how many requests fail, including "successful" responses with wrong content.
  • Utilisation: how busy a resource is, such as 70% CPU.
  • Saturation: how much work is waiting, such as a queue that keeps growing, or CPU throttling (question 392).

Why percentiles, not averages. Imagine 100 requests: 98 take 100 ms and 2 take 5 seconds. The average is about 200 ms, which looks fine. The 99th percentile is 5 seconds, which shows that one user in fifty is waiting five seconds. Those users are often the busiest customers with the biggest carts.

PercentileMeaning
p50, the medianHalf of requests are faster than this
p9595 in 100 are faster
p9999 in 100 are faster: the "tail"

USE in practice. A slow service with normal RED numbers in its own code but a database at 100% CPU and a growing connection queue points straight at the database. USE on the resources underneath finds the bottleneck.

630. What are SLIs, SLOs, and SLAs? ​

The short answer

A service level indicator, SLI, is a measurement of what users experience, such as the share of checkout requests that succeed in under 500 ms. A service level objective, SLO, is the target for that measurement over a period, such as 99.9% over 30 days. A service level agreement, SLA, is a promise to customers, usually with money back if it is broken, and is set looser than the internal SLO. SLIs measure, SLOs guide engineering decisions, and SLAs are contracts.

SLISLOSLA
IsA measurementAn internal targetAn external promise
ExampleGood requests ÷ all requests99.9% good over 30 days99.5% monthly, or service credits
Who uses itEngineersEngineers and product ownersCustomers, legal, sales
When missedReliability work takes priorityRefunds or penalties

A good SLI is a ratio of good events to all events:

text
SLI = requests that succeeded within 500 ms ÷ all valid requests

Clear rules decide what counts. For example, requests rejected because the user typed a wrong card number are not service failures, so they are left out of "valid".

Choosing SLIs for the shop:

User journeySLISLO
Pay for an orderSuccessful checkouts within 1 s99.9% over 30 days
Browse productsProduct pages served within 300 ms99.5%
Order confirmation emailSent within 5 minutes99%

How to set the target. Start from what users notice and what the business needs, check what you achieve today, and set a target slightly below what you can sustain. A target of 100% is wrong: nothing achieves it, and chasing it stops all change.

Why the SLA is looser. If the SLO is 99.9% and the SLA is 99.5%, you see trouble, and fix it, long before customers are owed money.

The same ideas apply to internal platforms, such as a CI system (question 266).

631. What is an error budget, and how do burn-rate alerts work? ​

The short answer

The error budget is the amount of failure the SLO allows: with a 99.9% target over 30 days, 0.1% of requests may fail, about 43 minutes of full outage. While budget remains, the team can ship changes and take risks. When it runs out, reliability work comes first. The burn rate is how fast the budget is being spent: a burn rate of 1 uses it up exactly at the end of the period. Alerts fire on high burn rates, measured over two windows, so fast disasters page quickly and slow leaks raise tickets.

The budget, worked out:

text
SLO 99.9% over 30 days
Budget = 100% − 99.9% = 0.1%

In time:      30 days × 24 h × 60 min × 0.001 = 43.2 minutes
In requests:  10,000,000 requests × 0.001 = 10,000 failed requests allowed
100%0%day 0day 30budget leftburn rate 1: lasts exactly 30 daysbad releaseburn rate 14: gone in ~2 days, page now
Speed of spending, not just the amount. At burn rate 1, the budget lasts exactly the whole period. After the bad release, the steep red line burns 14 times faster, which would empty it in about two days, so it pages immediately.

Burn rate is the actual error rate divided by the allowed error rate:

text
Error rate 1.4%, allowed 0.1%   →   burn rate = 1.4 ÷ 0.1 = 14
At burn rate 14, a 30-day budget lasts 30 ÷ 14 ≈ 2.1 days

Multi-window burn-rate alerts, the approach from Google's SRE workbook:

Burn rateLong windowShort windowActionMeaning
14.41 hour5 minutesPage2% of the month's budget gone in an hour
66 hours30 minutesPage5% gone in six hours
13 days6 hoursTicketSlowly spending more than planned

The alert fires only when both windows exceed the rate. The long window shows the problem is real, not a blip. The short window shows it is still happening, so the alert stops soon after the problem is fixed.

The error budget policy is the agreement that makes budgets useful. While budget remains, the team ships at full speed. When it is spent, releases slow down or pause, except fixes, and the team works on reliability until the budget recovers. It turns "should we ship or stabilise?" from an argument into a number (question 294).

632. What is OpenTelemetry? ​

The short answer

OpenTelemetry is an open standard, with libraries and tools, for producing and sending telemetry: traces, metrics and logs. You instrument your code once with its APIs and SDKs, often automatically for common frameworks, and send data in its standard protocol, OTLP, to a collector. The collector can process the data and send it to any backend, such as Prometheus, Jaeger, Grafana or a commercial service. It stops you being locked into one vendor's agent and gives every service the same shape of data.

Its parts:

PartDoes
APIThe calls your code uses: start a span, record a metric
SDKThe implementation per language: sampling, batching, exporting
Auto-instrumentationAdds traces for common libraries, such as HTTP, databases and queues, without code changes
OTLPThe standard protocol for sending telemetry
CollectorA service that receives, processes and forwards telemetry

Turning it on in a Node.js service, the checkout service in TypeScript:

ts
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'

const sdk = new NodeSDK({
  serviceName: 'checkout',                                                    // (1)
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector.observability:4318/v1/traces',               // (2)
  }),
  instrumentations: [getNodeAutoInstrumentations()],                         // (3)
})

sdk.start()                                                                   // (4)
  1. The service name appears on every span, so traces show which service did what.
  2. Send to the collector over OTLP on HTTP, port 4318, inside the cluster.
  3. Auto-instrumentation for HTTP servers and clients, Express, database drivers and more.
  4. Start before the app's own code loads, usually by running this file first, so every library is instrumented.

The collector in the middle:

  1. Receivers accept data, such as OTLP from apps.
  2. Processors batch it, add attributes such as the Kubernetes pod name, drop noisy data, or sample traces.
  3. Exporters send it to one or more backends.

Changing backend means changing the collector's configuration, not every service.

Where it came from. OpenTelemetry was formed in 2019 by merging two earlier projects, OpenTracing and OpenCensus. It is a Cloud Native Computing Foundation project, and one of its most active.

633. How does logging work in Kubernetes, and how should applications log? ​

The short answer

Containers should write logs to standard output and standard error. The container runtime saves them to files on the node, where kubectl logs reads them. Those files rotate and disappear with the pod. So a log agent reads them on every node, usually a DaemonSet such as Fluent Bit or the OpenTelemetry Collector. It adds pod and namespace details, and ships the logs to a central store, such as Loki, Elasticsearch or the cloud's log service. Applications should log structured JSON, with levels, a trace ID, and no secrets.

The path of a log line:

  1. The checkout app writes a line to standard output.
  2. The container runtime writes it to a file under /var/log/pods/ on the node.
  3. The log agent on that node reads the file, adds labels such as namespace, pod and container, and sends it on.
  4. The central store indexes it, and you search it there, long after the pod is gone.

Structured logs, as JSON, one object per line:

json
{"time":"2026-10-27T14:10:03.211Z","level":"error","service":"checkout","msg":"payment declined","orderId":"8812","paymentMethod":"card","traceId":"4bf92f3577b34da6a3ce929d0e0e4736"}

Each field can be searched and filtered, such as "all errors for order 8812", or counted, such as "declines by payment method". A free-text line like Payment for 8812 failed!! can only be grepped.

Good logging habits:

DoAvoid
JSON, one event per lineMulti-line text that agents split wrongly
A level: debug, info, warn, errorEverything at info
The trace ID in every lineLogs that cannot be linked to a request
IDs that help: order, requestPasswords, tokens, card numbers, personal data
Log at the boundaries: requests, calls, errorsLogging inside tight loops

Logs cost money. Many teams' largest observability bill is log storage. Keep debug logs off in production, drop noisy health-check lines at the agent, keep short retention for verbose logs, and use metrics for things you only count.

Loki is a log store from Grafana Labs that indexes only labels, such as namespace and app, not the full text, which makes it much cheaper to run, at the cost of slower full-text search.

The different kinds of logs in a cluster, and where each lives, are compared in question 424.

634. How does distributed tracing work? ​

The short answer

When a request enters the system, it gets a trace ID. Every service it passes through records a span: a named, timed step that carries the trace ID, its own span ID, and its parent's span ID. Each service passes the trace ID on to the next one in a request header, usually traceparent. A tracing backend collects all spans with the same trace ID and draws them as a timeline, showing exactly which step was slow or failed. Because tracing every request is expensive, systems keep only a sample.

trace 4bf9…, 840 ms0840 msgatewaycheckoutcart60 mspayment620 ms, the slow stepcard provider580 ms waitingeach bar is one span; indentation shows which span called which
A trace as a timeline. Each bar is a span: one step of the request, with its start and length. Nested bars show calls. Here, most of the 840 ms is spent in payment, waiting on the card provider.

How the ID travels. The checkout service calls payment with an HTTP header, defined by the W3C Trace Context standard:

text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Split by dashes:

  1. 00: the version of the format.
  2. 4bf92f…4736: the trace ID, the same for every span of this request.
  3. 00f067aa0ba902b7: the parent span ID, the caller's span.
  4. 01: flags; 01 means "this trace is sampled, record it".

Auto-instrumentation adds and reads this header for you. Messages on queues carry it in message attributes, so traces continue through asynchronous steps (question 632).

Adding your own span around an important step:

ts
import { trace, SpanStatusCode } from '@opentelemetry/api'

const tracer = trace.getTracer('checkout')

export async function chargeCard(order: Order) {
  return tracer.startActiveSpan('charge-card', async (span) => {    // (1)
    span.setAttribute('order.id', order.id)                          // (2)
    try {
      return await payments.charge(order)
    } catch (err) {
      span.recordException(err as Error)                             // (3)
      span.setStatus({ code: SpanStatusCode.ERROR })
      throw err
    } finally {
      span.end()                                                     // (4)
    }
  })
}
  1. Start a span named charge-card, as a child of whatever span is active, such as the incoming request.
  2. Add an attribute, so you can find traces for one order. High-detail values are fine here, unlike in metric labels (question 627).
  3. Record the error on the span, so the trace view shows it in red with the message.
  4. End the span in every case, which records its duration.

Sampling:

KindDecidesGood atWeak at
Head-basedAt the start of the request, such as "keep 10%"Simple, cheapMay drop the rare slow or failed request
Tail-basedAfter the request completes, in the collectorKeeps all errors and slow tracesNeeds the collector to hold spans briefly

Backends include Jaeger, Grafana Tempo, Zipkin, and the clouds' own tools, such as AWS X-Ray, Azure Monitor Application Insights and Google Cloud Trace.

635. What is the difference between metrics-server, Prometheus, kube-state-metrics, and node exporter? ​

The short answer

metrics-server collects current CPU and memory use of pods and nodes from each kubelet and serves it to the Kubernetes API, only for autoscaling and kubectl top: it keeps no history. Prometheus is the full monitoring system that scrapes and stores many metrics over time. kube-state-metrics turns the state of Kubernetes objects, such as desired versus available replicas or pod phases, into metrics. Node exporter publishes the machine's own metrics, such as disk space and network traffic. Prometheus usually scrapes the last two, plus the kubelet's container metrics.

ComponentProvidesAnswersKeeps history
metrics-serverCurrent CPU and memory per pod and node"What is this pod using now?"; feeds HPANo
kubelet with cAdvisorDetailed container CPU, memory, network, throttling"Is this container throttled?"No, scraped by Prometheus
kube-state-metricsObject state: replicas, pod phases, restarts, conditions"Does the Deployment have all its replicas?"No, scraped by Prometheus
node exporterMachine metrics: disk, filesystem, network, load"Is the node's disk filling?"No, scraped by Prometheus
PrometheusScrapes all of the above and your appsAnything, over timeYes

Why kube-state-metrics matters. Resource metrics tell you a pod uses 300 MB. They do not tell you that a Deployment wants 5 replicas but only 3 are available, or that a pod has restarted 40 times. kube-state-metrics publishes exactly that, such as:

text
kube_deployment_status_replicas_available{deployment="checkout"} 3
kube_deployment_spec_replicas{deployment="checkout"} 5
kube_pod_container_status_restarts_total{pod="checkout-7d9f-x2",container="app"} 40

An alert on "available less than desired for 10 minutes" catches a stuck rollout (question 425).

A common bundle. The kube-prometheus-stack Helm chart installs Prometheus, Alertmanager, Grafana, kube-state-metrics and node exporter, with ready-made dashboards and alerts, which is how many teams start (question 503). Managed clusters also offer managed Prometheus services that collect the same data without running Prometheus yourself.

636. What makes a useful dashboard? ​

The short answer

A useful dashboard answers one question for one audience quickly, such as "is checkout healthy right now?" for the on-call engineer. Put the user-facing signals at the top, traffic, errors, latency percentiles and SLO status, then the things that explain them, such as dependencies and saturation, below. Use consistent time ranges, mark deployments on the graphs, keep each panel simple, and link to traces and logs. Fewer good dashboards beat hundreds that nobody trusts.

A layout for a service dashboard, top to bottom:

  1. SLO status: budget remaining and current burn rate (question 631).
  2. RED: requests per second, error rate, latency p50, p95 and p99 (question 629).
  3. Recent changes: deployment markers, config changes.
  4. Dependencies: error rate and latency of calls to the database, payment provider, cache.
  5. Saturation: CPU throttling, memory against limits, pod count, queue depth.

Habits that make dashboards trustworthy:

DoAvoid
Titles that say what is shown and its unit"Metric 7", unlabelled axes
Percentiles for latencyAverages only
Error rate as a percentageRaw error counts with no total
Deployment annotations on every graphGuessing when the release went out
A link to the runbook and tracesDead-end panels
Dashboards in code, reviewed in GitHand-edited dashboards nobody can rebuild

Different audiences, different dashboards. The on-call engineer needs the service's health and its causes. A business owner needs orders per minute and conversion. A platform team needs cluster capacity and node health. One dashboard for everyone serves nobody.

637. How should a production incident be run? ​

The short answer

Declare the incident early and give it a severity. Assign roles: an incident commander who coordinates and decides, people who investigate and fix, and a communications lead who updates stakeholders and customers on a regular schedule. Focus first on restoring service, often by rolling back or failing over, not on finding the root cause. Keep a timeline in one shared channel. Afterwards, write a blameless review that explains what happened and why, and track the follow-up actions to completion.

Roles:

RoleDoesDoes not
Incident commanderCoordinates, sets priorities, makes calls, keeps timeDebug the problem personally
Operations, subject expertsInvestigate and apply fixesTalk to customers
Communications leadStatus page, stakeholder updates every 30 minutesSpeculate about causes
ScribeRecords the timeline and decisions

In a small team, one person may hold two roles, but the incident commander should still not be the person typing the fix.

Severity levels, an example:

LevelMeaningExample
SEV1Most users cannot use a core functionNobody can check out
SEV2Major degradation or a large group affectedCard payments fail on Android
SEV3Minor, with a workaroundOrder history loads slowly

During the incident:

  1. Stop the bleeding first. Roll back the last deploy, turn off a feature flag, fail over, or scale up. Diagnosis can continue afterwards (question 293).
  2. One channel, one timeline. Every action and finding is written down with the time.
  3. Change one thing at a time, and say it before doing it, so the effect is clear.
  4. Communicate on a schedule, even if the update is "still investigating".
  5. Declare it over when service is back and stable, not when the root cause is known.

After: the blameless review. Blameless means the review asks how the system and process allowed the mistake, not who made it. People who fear blame hide information, and hidden information means the same incident happens again. The review covers:

  1. The timeline, from first signal to recovery.
  2. The impact: users, duration, money.
  3. The causes: usually several, not one root cause.
  4. What went well, and what made it worse.
  5. Actions, each with an owner and a date, such as "add an alert on payment-provider latency".

638. What are MTTD, MTTR, and MTBF, and how do you reduce recovery time? ​

The short answer

MTTD, mean time to detect, is how long problems last before anyone notices. MTTR, mean time to restore, is how long from the start of a failure until service is back. MTBF, mean time between failures, is how long things typically run before failing. You cannot prevent every failure, so improving detection and recovery usually gives the biggest gains: alert on user symptoms, make rollback one click, practise failovers, keep runbooks current, and make telemetry good enough to find the cause fast.

failuredetectedacknowledgedcause foundrestoredtime to detecttime to restore: what users feel
Where recovery time goes. Users feel everything from the failure to the restore. Detection, response, diagnosis and the fix each take a slice, and each can be shortened.
MetricMeasuresImprove with
MTTD, detectFailure to first alertSymptom alerts, SLO burn rates, synthetic checks
MTTA, acknowledgeAlert to a person respondingClear on-call rotation, working paging
MTTR, restoreFailure to service restoredRollback, failover, runbooks, good telemetry
MTBF, between failuresTypical time between failuresSafer releases, redundancy, fixing review actions

"R" is ambiguous. MTTR can mean restore, repair, recover or resolve. Say which one you mean. For users, restore is the one that matters.

Ways to cut time to restore:

  1. Detect early: alert on user symptoms, plus synthetic checks, scripts that act like a user, such as placing a test order every minute from outside, so problems are seen even with no real traffic.
  2. Roll back fast: one command or click, practised, and safe because database changes are backwards compatible (question 287).
  3. Feature flags: turn off a broken feature without deploying (question 285).
  4. Runbooks: the first five steps for each alert, written before they are needed.
  5. Good telemetry: deployment markers on graphs, traces and correlated logs, so the cause is found in minutes (question 623).
  6. Practise: game days and failover drills, so the response is muscle memory (question 588).

Averages hide the story. One six-hour outage and twenty five-minute blips have very different lessons. Look at each incident, not only the mean, and track this alongside the DORA recovery measure (question 294).

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet