Appearance
Observability
37. What is observability, and how do logs, metrics, and traces differ?
The short answer
Observability is how well you can understand what a system is doing, and why, from the data it emits, including problems nobody predicted. Three kinds of data work together. Metrics are numbers over time, such as request rate, error rate and p99 latency: cheap, good for dashboards and alerts, answering "is something wrong?". Traces follow one request across every service and show where its time went: "where is it wrong?". Logs are detailed event records with context: "what exactly happened?". You alert on metrics, find the slow hop with traces, and read the logs for that trace.
| Metrics | Traces | Logs | |
|---|---|---|---|
| Is | Numbers aggregated over time | One request's path, as timed spans | Individual event records |
| Example | http_requests_total{status="500"} | Checkout 840 ms: payments span 700 ms | {"level":"error","msg":"card declined","orderId":"ord_88"} |
| Cost | Low | Medium; usually sampled | High at volume |
| Answers | Is something wrong? How much? | Where is the time or error? | What exactly happened? |
| Typical tools | Prometheus, Grafana | Jaeger, Tempo, Honeycomb | Loki, Elasticsearch, CloudWatch |
How they connect in an incident:
- A metric alert fires: checkout p99 latency is 3 seconds.
- A trace of a slow checkout shows the payments call taking 2.7 seconds.
- The logs for that trace ID show the payments client retrying three times after timeouts.
Without the shared trace ID, step 3 means searching millions of log lines by time and guessing (question 42).
38. What is distributed tracing, and why is it useful?
The short answer
Distributed tracing records the journey of one request through every service it touches. The whole journey is a trace with one trace ID. Each unit of work, such as an HTTP handler, a database query or a call to another service, is a span with a start time, duration, attributes and a parent span. The trace ID and the current span ID travel between services in a header, traceparent, so every service's spans join the same tree. You see at once which hop is slow or failing, which is nearly impossible from separate logs once a request crosses five services.
The traceparent header, from the W3C Trace Context standard, carries the version, the trace ID, the parent span ID and flags:
text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01Sampling. Tracing every request is expensive at scale. Teams keep, say, 10% of normal traces plus all traces with errors or high latency, which needs tail-based sampling, deciding after the trace finishes. More in how tracing works.
39. What is OpenTelemetry, and how can it instrument a Node.js service?
The short answer
OpenTelemetry, OTel, is the open, vendor-neutral standard for producing traces, metrics and logs: APIs and SDKs for each language, auto-instrumentation for common libraries, the OTLP protocol for sending data, and the Collector, a service that receives, processes and forwards telemetry to whichever backend you use. In Node, you load the SDK before your app starts, with auto-instrumentations for HTTP, Express, pg, Redis and others, so spans for incoming requests, outgoing calls and queries are created without code changes. You add custom spans and attributes for business steps.
js
// instrumentation.mjs, loaded before the app
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter(), // (1)
instrumentations: [getNodeAutoInstrumentations()], // (2)
})
sdk.start()
process.on('SIGTERM', () => sdk.shutdown()) // (3)bash
OTEL_SERVICE_NAME=orders-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
node --import ./instrumentation.mjs dist/server.js # (4)- Send spans with OTLP over HTTP to a Collector. The endpoint comes from an environment variable.
- Auto-instrument HTTP, Express,
pg,ioredis,fetchand many more. - Flush pending spans on shutdown, so the last requests are not lost.
--importloads instrumentation first, before Express orpgare imported, which is what lets it patch them. The service name labels every span.
A custom span for a business step:
ts
import { trace } from '@opentelemetry/api'
const tracer = trace.getTracer('orders')
export async function placeOrder(input: OrderInput) {
return tracer.startActiveSpan('placeOrder', async (span) => {
span.setAttribute('order.items', input.items.length)
try { return await doPlaceOrder(input) }
catch (err) { span.recordException(err as Error); throw err }
finally { span.end() }
})
}More: what OpenTelemetry is.
40. How would you design centralized logging for a microservices system?
The short answer
Every service writes structured JSON logs to stdout, with a shared set of fields: time, level, service, environment, version, trace ID and span ID, request ID, user or tenant ID where allowed, and a message. The platform collects them, with a node agent such as Fluent Bit, Vector or the OpenTelemetry Collector, and ships them to one central store, such as Loki, Elasticsearch or a cloud service, where they are searchable together. Redact secrets and personal data at the source, set retention by importance, sample noisy debug logs, and alert on log-derived signals only when a metric cannot do it.
ts
import pino from 'pino'
import { trace } from '@opentelemetry/api'
export const logger = pino({
level: config.LOG_LEVEL,
base: { service: 'orders-api', env: config.NODE_ENV, version: config.GIT_SHA }, // (1)
redact: ['req.headers.authorization', 'req.headers.cookie', '*.password', '*.cardNumber'], // (2)
mixin() {
const ctx = trace.getActiveSpan()?.spanContext()
return ctx ? { traceId: ctx.traceId, spanId: ctx.spanId } : {} // (3)
},
})
logger.info({ orderId: 'ord_8812', totalPence: 8550 }, 'order placed')- The same base fields on every line, so logs from all services can be filtered the same way.
- Redact secrets before they leave the process.
- Attach the current trace ID, so any log line can be joined to its trace.
One line of output:
json
{"level":30,"time":1793095331512,"service":"orders-api","env":"production","version":"3f9a1c2","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"00f067aa0ba902b7","orderId":"ord_8812","totalPence":8550,"msg":"order placed"}| Rule | Why |
|---|---|
| JSON, one event per line | Machines parse it; no multi-line stack trace mess |
| stdout, not files | The platform collects it; no disk filling up |
| Shared field names | Search across services with one query |
| Trace ID on every line | Jump from a trace to its logs |
| Levels used consistently | error means someone may need to act |
| Redaction at the source | Logs are often less protected than databases |
41. How would you design monitoring and alerting for a backend system?
The short answer
Monitor from the user's point of view first: for each important service and endpoint, track the golden signals, latency, traffic, errors and saturation, or RED for request-driven services: rate, errors, duration. Define SLOs, such as "99.9% of checkout requests succeed within 500 ms over 30 days", and alert on symptoms that users feel, ideally with burn-rate alerts on the error budget, not on every high CPU spike. Every alert must be actionable, link to a dashboard and a runbook, and go to an owner. Use dashboards and lower-priority tickets for causes like disk and memory.
| Signal | Metric | Example alert |
|---|---|---|
| Rate | Requests per second per route | Traffic drops 80% suddenly: something upstream broke |
| Errors | 5xx share per route | Error budget burning 14 times faster than allowed for an hour |
| Duration | p95 and p99 latency per route | p99 over 1 second for 10 minutes |
| Saturation | CPU, memory, event loop lag, pool waits, queue depth | Queue depth growing for 15 minutes |
What makes a good alert:
- It means users are, or soon will be, affected.
- Someone can do something about it now.
- It names the service, the symptom and a runbook link.
- It fires rarely enough to be trusted. Pages that are usually ignorable train people to ignore pages.
An SLO in numbers. A 99.9% monthly success target allows 0.1% failures, the error budget. With 10 million requests a month, that is 10,000 failed requests. A burn-rate alert fires when failures are using that budget so quickly that it would be gone in days, rather than when a single request fails. More: SLIs, SLOs and SLAs, error budgets and burn rates and good alerts.
42. What is a correlation ID or trace ID, and how should it propagate between services?
The short answer
A correlation ID is a unique ID attached to one request, or one business flow, and carried through every service, queue message and log line it touches, so you can find all of them later with one search. Today the standard is the OpenTelemetry trace ID, carried in the W3C traceparent header over HTTP and in message headers for queues. Create it at the edge if the request has none, accept it from trusted callers, add it to every log line, return it to the client, such as in an X-Request-Id response header, and pass it on every outgoing call and published message.
4bf92f returns every log line and span from all four services.ts
import { randomUUID } from 'node:crypto'
import { trace, context, propagation } from '@opentelemetry/api'
app.use((req, res, next) => {
const traceId = trace.getActiveSpan()?.spanContext().traceId ?? randomUUID() // (1)
res.set('X-Request-Id', traceId) // (2)
req.log = logger.child({ traceId }) // (3)
next()
})
export async function publish(topic: string, payload: object) {
const headers: Record<string, string> = {}
propagation.inject(context.active(), headers) // (4)
await broker.publish(topic, payload, { headers })
}- Use the trace ID that OpenTelemetry already created for this request, or a new random one if tracing is off.
- Give it back to the client, so a user or support agent can quote it in a bug report.
- A child logger stamps the ID on every log line for this request.
- Copy
traceparentinto the message headers. The consumer extracts it withpropagation.extract, so its spans and logs continue the same trace.
Rules: never trust an incoming ID from the public internet for anything but correlation, generate one if missing, and put it on audit records too (audit logging).