Appearance
Observability and SRE
623. What is the difference between monitoring and observability?
The short answer
Monitoring watches for problems you already know to look for: you decide in advance which numbers matter, such as error rate or disk space, and alert when they cross a line. Observability is how well you can understand problems you did not predict, by asking new questions of the data your system produces, such as "why are only card payments from Android slow since 14:10?". Monitoring tells you something is wrong. Observability helps you find out why.
An example. The checkout error rate alert fires. That is monitoring doing its job. Now you need to know which requests fail, for which customers, at which step, and since which release. If your logs, metrics and traces carry enough detail, such as the route, the payment method, the app version and a trace ID, you can slice the data until the answer appears. That is observability.
| Monitoring | Observability | |
|---|---|---|
| Answers | "Is it broken?" | "Why is it broken?" |
| Questions | Decided in advance | Asked during the investigation |
| Built from | Dashboards and alerts on known signals | Rich, connected logs, metrics and traces |
| Handles | Known failure modes | New, surprising ones |
They are not rivals. Observability is built on good telemetry, the data a system emits about itself. Monitoring is one use of that data. You need both: alerts to wake you up, and rich data to find the cause quickly (question 638).
What makes a system observable:
- Structured events with useful fields, not free text only (question 633).
- Shared IDs, such as a trace ID in every log line, so you can jump between signals (question 634).
- Consistent names for services, routes and versions across all data.
- High-detail fields where they help, such as customer or order ID in logs and traces, though not in metrics labels (question 627).
624. What are logs, metrics, and traces, and when do you use each?
The short answer
Metrics are numbers measured over time, such as requests per second or memory used: cheap to store, fast to query, and ideal for dashboards and alerts. Logs are records of individual events, with details, such as "payment 8812 failed: card declined": best for understanding exactly what happened. Traces follow one request across all the services it touches, timing each step, which shows where time was spent or where it failed. Alerts usually start from metrics, and investigations move into traces and logs.
| Metrics | Logs | Traces | |
|---|---|---|---|
| Shape | A number, with labels, over time | A record per event | A tree of timed steps per request |
| Example | http_requests_total{status="500"} | {"level":"error","msg":"card declined"} | checkout → cart → payment |
| Cost per unit | Very low | Medium to high | Medium; usually sampled |
| Best for | Alerts, trends, capacity | Exact details of an event | Where time goes, which hop fails |
| Weak at | Individual events | Trends across millions of events | Long-term trends |
Two more signals you will hear about:
- Events, such as Kubernetes events or deployment markers, show when something changed, which often explains a graph (question 424).
- Profiles show which functions in the code use CPU or memory, for performance work.
Connect them. Put the same trace ID in logs and traces, and link from dashboards to traces. Then one click takes you from a red graph to the failing request and its error message.
625. How does Prometheus work?
The short answer
Prometheus is an open-source monitoring system that pulls metrics. Every application or exporter publishes its current numbers as text on an HTTP endpoint, usually /metrics. Prometheus discovers these targets, for example from the Kubernetes API, scrapes them every few seconds, and stores each value with its labels and timestamp in its own time-series database. You query it with PromQL, rules turn queries into alerts, and Alertmanager groups those alerts and sends them to people.
What a scrape returns, plain text from the checkout service:
text
# HELP http_requests_total Requests handled.
# TYPE http_requests_total counter
http_requests_total{route="/checkout",status="200"} 18233
http_requests_total{route="/checkout",status="500"} 41
# HELP process_resident_memory_bytes Memory in use.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.83e+08Each line is one time series: a metric name, a set of labels in braces, and the current value. Prometheus stores a new sample of each series at every scrape.
Why pull instead of push?
- Prometheus knows what should exist. A target that stops answering shows up immediately as down, through the built-in
upmetric. - Apps stay simple. They only keep current numbers in memory and answer when asked.
- No overload from senders. Prometheus controls the rate.
Short batch jobs that finish before a scrape can push to a Pushgateway, which Prometheus then scrapes.
In Kubernetes, the Prometheus Operator adds resources such as ServiceMonitor, which say "scrape every Service with label app=checkout on port metrics". Teams add their own monitoring without editing Prometheus's configuration.
Its limits. One Prometheus server stores data on its local disk, typically for days or weeks. For long-term storage, many clusters, and high availability, teams use remote write to systems such as Thanos or Grafana Mimir, or a managed Prometheus service from their cloud.
626. How do you write basic PromQL queries such as rate, sum, and percentiles?
The short answer
Counters only ever go up, so you almost never graph them directly: rate() turns a counter into a per-second speed over a time window, such as rate(http_requests_total[5m]). sum by (label) adds series together, grouped by the labels you keep. For latency percentiles, histograms store counts in buckets, and histogram_quantile(0.99, ...) estimates the 99th percentile from the bucket rates. Error rate is failed requests divided by all requests.
Requests per second for checkout:
text
sum(rate(http_requests_total{job="checkout"}[5m]))Read from the inside out:
http_requests_total{job="checkout"}picks every request-counter series from the checkout service, one per route and status.[5m]takes the last five minutes of samples for each.rate(...)turns each into an average per-second increase. It also handles counter resets, when a pod restarts and its counter starts again from zero.sum(...)adds all series into one number.
Error rate, as a fraction:
text
sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))status=~"5.." is a regular expression matching any status from 500 to 599. A result of 0.004 means 0.4% of requests fail.
Requests per second by route:
text
sum by (route) (rate(http_requests_total{job="checkout"}[5m]))by (route) keeps the route label and adds up everything else, giving one line per route.
The 99th percentile of latency:
text
histogram_quantile(
0.99,
sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))
)A histogram stores counts in buckets, each with a label le, "less than or equal", such as "requests faster than 0.25 s". The query takes the rate of each bucket, adds them across pods while keeping le, then estimates the value below which 99% of requests fell. A result of 0.82 means 99% of requests finished within 820 milliseconds (question 627).
Common mistakes:
| Mistake | Problem |
|---|---|
| Graphing a raw counter | A line that only climbs; resets look like crashes |
sum before rate | Counter resets in one pod corrupt the result |
| Averaging percentiles across pods | Mathematically wrong; aggregate buckets, then take the quantile |
| A window shorter than two scrapes | Gaps and noisy graphs |
627. What are the metric types, and what is cardinality?
The short answer
Prometheus has four types. A counter only goes up, such as requests served. A gauge goes up and down, such as memory in use. A histogram counts observations into buckets, such as request durations, so percentiles can be estimated across many pods. A summary calculates percentiles inside each process, which cannot be combined across pods. Cardinality is the number of unique label combinations, each of which is a separate series, so a label with many possible values, such as user ID, can create millions of series and overload the system.
In code, for the checkout service in TypeScript with prom-client:
ts
import client from 'prom-client'
export const ordersPlaced = new client.Counter({ // (1)
name: 'orders_placed_total',
help: 'Orders successfully placed',
labelNames: ['payment_method'],
})
export const cartsInProgress = new client.Gauge({ // (2)
name: 'carts_in_progress',
help: 'Carts currently being checked out',
})
export const requestDuration = new client.Histogram({ // (3)
name: 'http_request_duration_seconds',
help: 'Time to handle a request',
labelNames: ['route', 'status'],
buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5],
})
// in a request handler:
const stop = requestDuration.startTimer({ route: '/checkout' })
// ... handle the request ...
ordersPlaced.inc({ payment_method: 'card' }) // (4)
stop({ status: '200' }) // (5)- A counter, ending in
_totalby convention. It only increases. Queries userate()on it. - A gauge, which goes up and down, set to the current value.
- A histogram with chosen bucket edges in seconds. Pick edges around your targets, such as 0.25 s if the goal is "under 250 ms".
- Count one order paid by card.
- Stop the timer, which records the duration into the right bucket, with the status label.
The service exposes all of this on /metrics for Prometheus to scrape (question 625).
Cardinality, with numbers. Every unique combination of label values is a separate time series:
text
routes (20) × status codes (8) × pods (30) = 4,800 seriesThat is fine. Now add a user_id label with 2 million users:
text
4,800 × 2,000,000 = 9,600,000,000 seriesThat is not fine. Memory explodes, queries crawl, and managed services bill per series.
Rules for labels:
| Good labels | Bad labels |
|---|---|
Route template: /orders/:id | The real path: /orders/8812 |
| Status code, method | User ID, order ID, email |
| Service, version, region | Full error messages |
| Payment method | Timestamps, random IDs |
Details such as user and order IDs belong in logs and traces, which are designed for them (question 624).
628. What makes a good alert, and how do you avoid alert fatigue?
The short answer
A good alert means a person must act now, on something users feel. Page on symptoms, such as errors and slow responses breaking your SLO, not on causes like high CPU that may be harmless. Every page should be urgent, actionable, and link to a runbook, a short guide with first steps. Send less urgent problems to a ticket queue, keep the rest on dashboards, group related alerts, and regularly delete alerts nobody acts on. Too many alerts teach people to ignore all of them.
Alert fatigue is what happens when people get so many alerts, most of them unimportant, that they stop reacting, and miss the one that matters.
Symptom versus cause:
| Alert | Type | Page someone? |
|---|---|---|
| Checkout error rate above SLO target for 5 minutes | Symptom: users fail to pay | Yes |
| 99th percentile latency above 2 s for 10 minutes | Symptom: users wait | Yes |
| Node CPU at 85% | Cause, maybe harmless | No: dashboard |
| One pod restarted | Cause; Kubernetes handles it | No |
| Disk will be full in 4 hours at current rate | Cause, but certain harm soon | Yes, or a ticket if days away |
| Certificate expires in 14 days | Future harm | Ticket |
What a good alert carries:
- A clear title: "Checkout error rate 3.2%, SLO 0.1%".
- Impact: who is affected and how.
- A runbook link: what to check first.
- Links to the dashboard and recent deployments.
- Severity and routing: who gets it, and how loudly.
An alert rule in Prometheus:
yaml
groups:
- name: checkout
rules:
- alert: CheckoutHighErrorRate
expr: |
sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="checkout"}[5m])) > 0.02
for: 5m
labels:
severity: page
annotations:
summary: "Checkout errors above 2% for 5 minutes"
runbook: "https://runbooks.shop.example/checkout-errors"for: 5m means the condition must stay true for five minutes before the alert fires, which ignores brief blips. The severity label lets Alertmanager route pages to on-call and warnings to a chat channel.
Keep the alert set healthy:
- Review every page in a weekly or monthly meeting: did someone need to act? If not, change or delete the alert.
- Group related alerts, so one database failure is one notification, not fifty.
- Silence during planned work, with an end time.
- Prefer SLO burn-rate alerts for user-facing services, which fire in proportion to how fast users are being hurt (question 631).
629. What are the golden signals, and the RED and USE methods?
The short answer
They are checklists of what to measure. The four golden signals, from Google's SRE book, are latency, traffic, errors and saturation. RED, for services, is rate of requests, errors, and duration. USE, for resources such as CPUs, disks and queues, is utilisation, saturation and errors. Use RED or the golden signals for every service users call, and USE for the machines and resources underneath. Measure latency with percentiles, such as the 99th, not averages.
| Method | For | Measures |
|---|---|---|
| Golden signals | Any user-facing system | Latency, traffic, errors, saturation |
| RED | Request-driven services | Rate, errors, duration |
| USE | Resources: CPU, memory, disk, network, queues | Utilisation, saturation, errors |
What each word means:
- Latency, or duration: how long requests take. Measure successful and failed requests separately, because a fast error can make latency look better.
- Traffic, or rate: how much demand, such as requests per second.
- Errors: how many requests fail, including "successful" responses with wrong content.
- Utilisation: how busy a resource is, such as 70% CPU.
- Saturation: how much work is waiting, such as a queue that keeps growing, or CPU throttling (question 392).
Why percentiles, not averages. Imagine 100 requests: 98 take 100 ms and 2 take 5 seconds. The average is about 200 ms, which looks fine. The 99th percentile is 5 seconds, which shows that one user in fifty is waiting five seconds. Those users are often the busiest customers with the biggest carts.
| Percentile | Meaning |
|---|---|
| p50, the median | Half of requests are faster than this |
| p95 | 95 in 100 are faster |
| p99 | 99 in 100 are faster: the "tail" |
USE in practice. A slow service with normal RED numbers in its own code but a database at 100% CPU and a growing connection queue points straight at the database. USE on the resources underneath finds the bottleneck.
630. What are SLIs, SLOs, and SLAs?
The short answer
A service level indicator, SLI, is a measurement of what users experience, such as the share of checkout requests that succeed in under 500 ms. A service level objective, SLO, is the target for that measurement over a period, such as 99.9% over 30 days. A service level agreement, SLA, is a promise to customers, usually with money back if it is broken, and is set looser than the internal SLO. SLIs measure, SLOs guide engineering decisions, and SLAs are contracts.
| SLI | SLO | SLA | |
|---|---|---|---|
| Is | A measurement | An internal target | An external promise |
| Example | Good requests ÷ all requests | 99.9% good over 30 days | 99.5% monthly, or service credits |
| Who uses it | Engineers | Engineers and product owners | Customers, legal, sales |
| When missed | Reliability work takes priority | Refunds or penalties |
A good SLI is a ratio of good events to all events:
text
SLI = requests that succeeded within 500 ms ÷ all valid requestsClear rules decide what counts. For example, requests rejected because the user typed a wrong card number are not service failures, so they are left out of "valid".
Choosing SLIs for the shop:
| User journey | SLI | SLO |
|---|---|---|
| Pay for an order | Successful checkouts within 1 s | 99.9% over 30 days |
| Browse products | Product pages served within 300 ms | 99.5% |
| Order confirmation email | Sent within 5 minutes | 99% |
How to set the target. Start from what users notice and what the business needs, check what you achieve today, and set a target slightly below what you can sustain. A target of 100% is wrong: nothing achieves it, and chasing it stops all change.
Why the SLA is looser. If the SLO is 99.9% and the SLA is 99.5%, you see trouble, and fix it, long before customers are owed money.
The same ideas apply to internal platforms, such as a CI system (question 266).
631. What is an error budget, and how do burn-rate alerts work?
The short answer
The error budget is the amount of failure the SLO allows: with a 99.9% target over 30 days, 0.1% of requests may fail, about 43 minutes of full outage. While budget remains, the team can ship changes and take risks. When it runs out, reliability work comes first. The burn rate is how fast the budget is being spent: a burn rate of 1 uses it up exactly at the end of the period. Alerts fire on high burn rates, measured over two windows, so fast disasters page quickly and slow leaks raise tickets.
The budget, worked out:
text
SLO 99.9% over 30 days
Budget = 100% − 99.9% = 0.1%
In time: 30 days × 24 h × 60 min × 0.001 = 43.2 minutes
In requests: 10,000,000 requests × 0.001 = 10,000 failed requests allowedBurn rate is the actual error rate divided by the allowed error rate:
text
Error rate 1.4%, allowed 0.1% → burn rate = 1.4 ÷ 0.1 = 14
At burn rate 14, a 30-day budget lasts 30 ÷ 14 ≈ 2.1 daysMulti-window burn-rate alerts, the approach from Google's SRE workbook:
| Burn rate | Long window | Short window | Action | Meaning |
|---|---|---|---|---|
| 14.4 | 1 hour | 5 minutes | Page | 2% of the month's budget gone in an hour |
| 6 | 6 hours | 30 minutes | Page | 5% gone in six hours |
| 1 | 3 days | 6 hours | Ticket | Slowly spending more than planned |
The alert fires only when both windows exceed the rate. The long window shows the problem is real, not a blip. The short window shows it is still happening, so the alert stops soon after the problem is fixed.
The error budget policy is the agreement that makes budgets useful. While budget remains, the team ships at full speed. When it is spent, releases slow down or pause, except fixes, and the team works on reliability until the budget recovers. It turns "should we ship or stabilise?" from an argument into a number (question 294).
632. What is OpenTelemetry?
The short answer
OpenTelemetry is an open standard, with libraries and tools, for producing and sending telemetry: traces, metrics and logs. You instrument your code once with its APIs and SDKs, often automatically for common frameworks, and send data in its standard protocol, OTLP, to a collector. The collector can process the data and send it to any backend, such as Prometheus, Jaeger, Grafana or a commercial service. It stops you being locked into one vendor's agent and gives every service the same shape of data.
Its parts:
| Part | Does |
|---|---|
| API | The calls your code uses: start a span, record a metric |
| SDK | The implementation per language: sampling, batching, exporting |
| Auto-instrumentation | Adds traces for common libraries, such as HTTP, databases and queues, without code changes |
| OTLP | The standard protocol for sending telemetry |
| Collector | A service that receives, processes and forwards telemetry |
Turning it on in a Node.js service, the checkout service in TypeScript:
ts
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
const sdk = new NodeSDK({
serviceName: 'checkout', // (1)
traceExporter: new OTLPTraceExporter({
url: 'http://otel-collector.observability:4318/v1/traces', // (2)
}),
instrumentations: [getNodeAutoInstrumentations()], // (3)
})
sdk.start() // (4)- The service name appears on every span, so traces show which service did what.
- Send to the collector over OTLP on HTTP, port 4318, inside the cluster.
- Auto-instrumentation for HTTP servers and clients, Express, database drivers and more.
- Start before the app's own code loads, usually by running this file first, so every library is instrumented.
The collector in the middle:
- Receivers accept data, such as OTLP from apps.
- Processors batch it, add attributes such as the Kubernetes pod name, drop noisy data, or sample traces.
- Exporters send it to one or more backends.
Changing backend means changing the collector's configuration, not every service.
Where it came from. OpenTelemetry was formed in 2019 by merging two earlier projects, OpenTracing and OpenCensus. It is a Cloud Native Computing Foundation project, and one of its most active.
633. How does logging work in Kubernetes, and how should applications log?
The short answer
Containers should write logs to standard output and standard error. The container runtime saves them to files on the node, where kubectl logs reads them. Those files rotate and disappear with the pod. So a log agent reads them on every node, usually a DaemonSet such as Fluent Bit or the OpenTelemetry Collector. It adds pod and namespace details, and ships the logs to a central store, such as Loki, Elasticsearch or the cloud's log service. Applications should log structured JSON, with levels, a trace ID, and no secrets.
The path of a log line:
- The checkout app writes a line to standard output.
- The container runtime writes it to a file under
/var/log/pods/on the node. - The log agent on that node reads the file, adds labels such as namespace, pod and container, and sends it on.
- The central store indexes it, and you search it there, long after the pod is gone.
Structured logs, as JSON, one object per line:
json
{"time":"2026-10-27T14:10:03.211Z","level":"error","service":"checkout","msg":"payment declined","orderId":"8812","paymentMethod":"card","traceId":"4bf92f3577b34da6a3ce929d0e0e4736"}Each field can be searched and filtered, such as "all errors for order 8812", or counted, such as "declines by payment method". A free-text line like Payment for 8812 failed!! can only be grepped.
Good logging habits:
| Do | Avoid |
|---|---|
| JSON, one event per line | Multi-line text that agents split wrongly |
| A level: debug, info, warn, error | Everything at info |
| The trace ID in every line | Logs that cannot be linked to a request |
| IDs that help: order, request | Passwords, tokens, card numbers, personal data |
| Log at the boundaries: requests, calls, errors | Logging inside tight loops |
Logs cost money. Many teams' largest observability bill is log storage. Keep debug logs off in production, drop noisy health-check lines at the agent, keep short retention for verbose logs, and use metrics for things you only count.
Loki is a log store from Grafana Labs that indexes only labels, such as namespace and app, not the full text, which makes it much cheaper to run, at the cost of slower full-text search.
The different kinds of logs in a cluster, and where each lives, are compared in question 424.
634. How does distributed tracing work?
The short answer
When a request enters the system, it gets a trace ID. Every service it passes through records a span: a named, timed step that carries the trace ID, its own span ID, and its parent's span ID. Each service passes the trace ID on to the next one in a request header, usually traceparent. A tracing backend collects all spans with the same trace ID and draws them as a timeline, showing exactly which step was slow or failed. Because tracing every request is expensive, systems keep only a sample.
How the ID travels. The checkout service calls payment with an HTTP header, defined by the W3C Trace Context standard:
text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01Split by dashes:
00: the version of the format.4bf92f…4736: the trace ID, the same for every span of this request.00f067aa0ba902b7: the parent span ID, the caller's span.01: flags;01means "this trace is sampled, record it".
Auto-instrumentation adds and reads this header for you. Messages on queues carry it in message attributes, so traces continue through asynchronous steps (question 632).
Adding your own span around an important step:
ts
import { trace, SpanStatusCode } from '@opentelemetry/api'
const tracer = trace.getTracer('checkout')
export async function chargeCard(order: Order) {
return tracer.startActiveSpan('charge-card', async (span) => { // (1)
span.setAttribute('order.id', order.id) // (2)
try {
return await payments.charge(order)
} catch (err) {
span.recordException(err as Error) // (3)
span.setStatus({ code: SpanStatusCode.ERROR })
throw err
} finally {
span.end() // (4)
}
})
}- Start a span named
charge-card, as a child of whatever span is active, such as the incoming request. - Add an attribute, so you can find traces for one order. High-detail values are fine here, unlike in metric labels (question 627).
- Record the error on the span, so the trace view shows it in red with the message.
- End the span in every case, which records its duration.
Sampling:
| Kind | Decides | Good at | Weak at |
|---|---|---|---|
| Head-based | At the start of the request, such as "keep 10%" | Simple, cheap | May drop the rare slow or failed request |
| Tail-based | After the request completes, in the collector | Keeps all errors and slow traces | Needs the collector to hold spans briefly |
Backends include Jaeger, Grafana Tempo, Zipkin, and the clouds' own tools, such as AWS X-Ray, Azure Monitor Application Insights and Google Cloud Trace.
635. What is the difference between metrics-server, Prometheus, kube-state-metrics, and node exporter?
The short answer
metrics-server collects current CPU and memory use of pods and nodes from each kubelet and serves it to the Kubernetes API, only for autoscaling and kubectl top: it keeps no history. Prometheus is the full monitoring system that scrapes and stores many metrics over time. kube-state-metrics turns the state of Kubernetes objects, such as desired versus available replicas or pod phases, into metrics. Node exporter publishes the machine's own metrics, such as disk space and network traffic. Prometheus usually scrapes the last two, plus the kubelet's container metrics.
| Component | Provides | Answers | Keeps history |
|---|---|---|---|
| metrics-server | Current CPU and memory per pod and node | "What is this pod using now?"; feeds HPA | No |
| kubelet with cAdvisor | Detailed container CPU, memory, network, throttling | "Is this container throttled?" | No, scraped by Prometheus |
| kube-state-metrics | Object state: replicas, pod phases, restarts, conditions | "Does the Deployment have all its replicas?" | No, scraped by Prometheus |
| node exporter | Machine metrics: disk, filesystem, network, load | "Is the node's disk filling?" | No, scraped by Prometheus |
| Prometheus | Scrapes all of the above and your apps | Anything, over time | Yes |
Why kube-state-metrics matters. Resource metrics tell you a pod uses 300 MB. They do not tell you that a Deployment wants 5 replicas but only 3 are available, or that a pod has restarted 40 times. kube-state-metrics publishes exactly that, such as:
text
kube_deployment_status_replicas_available{deployment="checkout"} 3
kube_deployment_spec_replicas{deployment="checkout"} 5
kube_pod_container_status_restarts_total{pod="checkout-7d9f-x2",container="app"} 40An alert on "available less than desired for 10 minutes" catches a stuck rollout (question 425).
A common bundle. The kube-prometheus-stack Helm chart installs Prometheus, Alertmanager, Grafana, kube-state-metrics and node exporter, with ready-made dashboards and alerts, which is how many teams start (question 503). Managed clusters also offer managed Prometheus services that collect the same data without running Prometheus yourself.
636. What makes a useful dashboard?
The short answer
A useful dashboard answers one question for one audience quickly, such as "is checkout healthy right now?" for the on-call engineer. Put the user-facing signals at the top, traffic, errors, latency percentiles and SLO status, then the things that explain them, such as dependencies and saturation, below. Use consistent time ranges, mark deployments on the graphs, keep each panel simple, and link to traces and logs. Fewer good dashboards beat hundreds that nobody trusts.
A layout for a service dashboard, top to bottom:
- SLO status: budget remaining and current burn rate (question 631).
- RED: requests per second, error rate, latency p50, p95 and p99 (question 629).
- Recent changes: deployment markers, config changes.
- Dependencies: error rate and latency of calls to the database, payment provider, cache.
- Saturation: CPU throttling, memory against limits, pod count, queue depth.
Habits that make dashboards trustworthy:
| Do | Avoid |
|---|---|
| Titles that say what is shown and its unit | "Metric 7", unlabelled axes |
| Percentiles for latency | Averages only |
| Error rate as a percentage | Raw error counts with no total |
| Deployment annotations on every graph | Guessing when the release went out |
| A link to the runbook and traces | Dead-end panels |
| Dashboards in code, reviewed in Git | Hand-edited dashboards nobody can rebuild |
Different audiences, different dashboards. The on-call engineer needs the service's health and its causes. A business owner needs orders per minute and conversion. A platform team needs cluster capacity and node health. One dashboard for everyone serves nobody.
637. How should a production incident be run?
The short answer
Declare the incident early and give it a severity. Assign roles: an incident commander who coordinates and decides, people who investigate and fix, and a communications lead who updates stakeholders and customers on a regular schedule. Focus first on restoring service, often by rolling back or failing over, not on finding the root cause. Keep a timeline in one shared channel. Afterwards, write a blameless review that explains what happened and why, and track the follow-up actions to completion.
Roles:
| Role | Does | Does not |
|---|---|---|
| Incident commander | Coordinates, sets priorities, makes calls, keeps time | Debug the problem personally |
| Operations, subject experts | Investigate and apply fixes | Talk to customers |
| Communications lead | Status page, stakeholder updates every 30 minutes | Speculate about causes |
| Scribe | Records the timeline and decisions |
In a small team, one person may hold two roles, but the incident commander should still not be the person typing the fix.
Severity levels, an example:
| Level | Meaning | Example |
|---|---|---|
| SEV1 | Most users cannot use a core function | Nobody can check out |
| SEV2 | Major degradation or a large group affected | Card payments fail on Android |
| SEV3 | Minor, with a workaround | Order history loads slowly |
During the incident:
- Stop the bleeding first. Roll back the last deploy, turn off a feature flag, fail over, or scale up. Diagnosis can continue afterwards (question 293).
- One channel, one timeline. Every action and finding is written down with the time.
- Change one thing at a time, and say it before doing it, so the effect is clear.
- Communicate on a schedule, even if the update is "still investigating".
- Declare it over when service is back and stable, not when the root cause is known.
After: the blameless review. Blameless means the review asks how the system and process allowed the mistake, not who made it. People who fear blame hide information, and hidden information means the same incident happens again. The review covers:
- The timeline, from first signal to recovery.
- The impact: users, duration, money.
- The causes: usually several, not one root cause.
- What went well, and what made it worse.
- Actions, each with an owner and a date, such as "add an alert on payment-provider latency".
638. What are MTTD, MTTR, and MTBF, and how do you reduce recovery time?
The short answer
MTTD, mean time to detect, is how long problems last before anyone notices. MTTR, mean time to restore, is how long from the start of a failure until service is back. MTBF, mean time between failures, is how long things typically run before failing. You cannot prevent every failure, so improving detection and recovery usually gives the biggest gains: alert on user symptoms, make rollback one click, practise failovers, keep runbooks current, and make telemetry good enough to find the cause fast.
| Metric | Measures | Improve with |
|---|---|---|
| MTTD, detect | Failure to first alert | Symptom alerts, SLO burn rates, synthetic checks |
| MTTA, acknowledge | Alert to a person responding | Clear on-call rotation, working paging |
| MTTR, restore | Failure to service restored | Rollback, failover, runbooks, good telemetry |
| MTBF, between failures | Typical time between failures | Safer releases, redundancy, fixing review actions |
"R" is ambiguous. MTTR can mean restore, repair, recover or resolve. Say which one you mean. For users, restore is the one that matters.
Ways to cut time to restore:
- Detect early: alert on user symptoms, plus synthetic checks, scripts that act like a user, such as placing a test order every minute from outside, so problems are seen even with no real traffic.
- Roll back fast: one command or click, practised, and safe because database changes are backwards compatible (question 287).
- Feature flags: turn off a broken feature without deploying (question 285).
- Runbooks: the first five steps for each alert, written before they are needed.
- Good telemetry: deployment markers on graphs, traces and correlated logs, so the cause is found in minutes (question 623).
- Practise: game days and failover drills, so the response is muscle memory (question 588).
Averages hide the story. One six-hour outage and twenty five-minute blips have very different lessons. Look at each incident, not only the mean, and track this alongside the DORA recovery measure (question 294).