Skip to content

Interview PrepObservability · Core

Observability ​

37. What is observability, and how do logs, metrics, and traces differ? ​

The short answer

Observability is how well you can understand what a system is doing, and why, from the data it emits, including problems nobody predicted. Three kinds of data work together. Metrics are numbers over time, such as request rate, error rate and p99 latency: cheap, good for dashboards and alerts, answering "is something wrong?". Traces follow one request across every service and show where its time went: "where is it wrong?". Logs are detailed event records with context: "what exactly happened?". You alert on metrics, find the slow hop with traces, and read the logs for that trace.

MetricsTracesLogs
IsNumbers aggregated over timeOne request's path, as timed spansIndividual event records
Examplehttp_requests_total{status="500"}Checkout 840 ms: payments span 700 ms{"level":"error","msg":"card declined","orderId":"ord_88"}
CostLowMedium; usually sampledHigh at volume
AnswersIs something wrong? How much?Where is the time or error?What exactly happened?
Typical toolsPrometheus, GrafanaJaeger, Tempo, HoneycombLoki, Elasticsearch, CloudWatch

How they connect in an incident:

  1. A metric alert fires: checkout p99 latency is 3 seconds.
  2. A trace of a slow checkout shows the payments call taking 2.7 seconds.
  3. The logs for that trace ID show the payments client retrying three times after timeouts.

Without the shared trace ID, step 3 means searching millions of log lines by time and guessing (question 42).

38. What is distributed tracing, and why is it useful? ​

The short answer

Distributed tracing records the journey of one request through every service it touches. The whole journey is a trace with one trace ID. Each unit of work, such as an HTTP handler, a database query or a call to another service, is a span with a start time, duration, attributes and a parent span. The trace ID and the current span ID travel between services in a header, traceparent, so every service's spans join the same tree. You see at once which hop is slow or failing, which is nearly impossible from separate logs once a request crosses five services.

trace 4bf92f35… · POST /checkout · 840 msgateway: POST /checkout · 840 msorders: placeOrder · 820 msdb INSERT · 40 mspayments: charge · 700 msHTTP POST api.stripe.com · 650 mspublish · 15 msbar length is time; nesting shows which call made which
The slow hop stands out. Each bar is a span, placed by when it started and sized by how long it took. Nested bars are calls made by their parent. Here almost all of the 840 ms is the call to the payment provider.

The traceparent header, from the W3C Trace Context standard, carries the version, the trace ID, the parent span ID and flags:

text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Sampling. Tracing every request is expensive at scale. Teams keep, say, 10% of normal traces plus all traces with errors or high latency, which needs tail-based sampling, deciding after the trace finishes. More in how tracing works.

39. What is OpenTelemetry, and how can it instrument a Node.js service? ​

The short answer

OpenTelemetry, OTel, is the open, vendor-neutral standard for producing traces, metrics and logs: APIs and SDKs for each language, auto-instrumentation for common libraries, the OTLP protocol for sending data, and the Collector, a service that receives, processes and forwards telemetry to whichever backend you use. In Node, you load the SDK before your app starts, with auto-instrumentations for HTTP, Express, pg, Redis and others, so spans for incoming requests, outgoing calls and queries are created without code changes. You add custom spans and attributes for business steps.

js
// instrumentation.mjs, loaded before the app
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),                      // (1)
  instrumentations: [getNodeAutoInstrumentations()],           // (2)
})
sdk.start()
process.on('SIGTERM', () => sdk.shutdown())                    // (3)
bash
OTEL_SERVICE_NAME=orders-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
node --import ./instrumentation.mjs dist/server.js             # (4)
  1. Send spans with OTLP over HTTP to a Collector. The endpoint comes from an environment variable.
  2. Auto-instrument HTTP, Express, pg, ioredis, fetch and many more.
  3. Flush pending spans on shutdown, so the last requests are not lost.
  4. --import loads instrumentation first, before Express or pg are imported, which is what lets it patch them. The service name labels every span.

A custom span for a business step:

ts
import { trace } from '@opentelemetry/api'
const tracer = trace.getTracer('orders')

export async function placeOrder(input: OrderInput) {
  return tracer.startActiveSpan('placeOrder', async (span) => {
    span.setAttribute('order.items', input.items.length)
    try { return await doPlaceOrder(input) }
    catch (err) { span.recordException(err as Error); throw err }
    finally { span.end() }
  })
}

More: what OpenTelemetry is.

40. How would you design centralized logging for a microservices system? ​

The short answer

Every service writes structured JSON logs to stdout, with a shared set of fields: time, level, service, environment, version, trace ID and span ID, request ID, user or tenant ID where allowed, and a message. The platform collects them, with a node agent such as Fluent Bit, Vector or the OpenTelemetry Collector, and ships them to one central store, such as Loki, Elasticsearch or a cloud service, where they are searchable together. Redact secrets and personal data at the source, set retention by importance, sample noisy debug logs, and alert on log-derived signals only when a metric cannot do it.

ts
import pino from 'pino'
import { trace } from '@opentelemetry/api'

export const logger = pino({
  level: config.LOG_LEVEL,
  base: { service: 'orders-api', env: config.NODE_ENV, version: config.GIT_SHA },   // (1)
  redact: ['req.headers.authorization', 'req.headers.cookie', '*.password', '*.cardNumber'],   // (2)
  mixin() {
    const ctx = trace.getActiveSpan()?.spanContext()
    return ctx ? { traceId: ctx.traceId, spanId: ctx.spanId } : {}              // (3)
  },
})

logger.info({ orderId: 'ord_8812', totalPence: 8550 }, 'order placed')
  1. The same base fields on every line, so logs from all services can be filtered the same way.
  2. Redact secrets before they leave the process.
  3. Attach the current trace ID, so any log line can be joined to its trace.

One line of output:

json
{"level":30,"time":1793095331512,"service":"orders-api","env":"production","version":"3f9a1c2","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"00f067aa0ba902b7","orderId":"ord_8812","totalPence":8550,"msg":"order placed"}
RuleWhy
JSON, one event per lineMachines parse it; no multi-line stack trace mess
stdout, not filesThe platform collects it; no disk filling up
Shared field namesSearch across services with one query
Trace ID on every lineJump from a trace to its logs
Levels used consistentlyerror means someone may need to act
Redaction at the sourceLogs are often less protected than databases

41. How would you design monitoring and alerting for a backend system? ​

The short answer

Monitor from the user's point of view first: for each important service and endpoint, track the golden signals, latency, traffic, errors and saturation, or RED for request-driven services: rate, errors, duration. Define SLOs, such as "99.9% of checkout requests succeed within 500 ms over 30 days", and alert on symptoms that users feel, ideally with burn-rate alerts on the error budget, not on every high CPU spike. Every alert must be actionable, link to a dashboard and a runbook, and go to an owner. Use dashboards and lower-priority tickets for causes like disk and memory.

SignalMetricExample alert
RateRequests per second per routeTraffic drops 80% suddenly: something upstream broke
Errors5xx share per routeError budget burning 14 times faster than allowed for an hour
Durationp95 and p99 latency per routep99 over 1 second for 10 minutes
SaturationCPU, memory, event loop lag, pool waits, queue depthQueue depth growing for 15 minutes

What makes a good alert:

  1. It means users are, or soon will be, affected.
  2. Someone can do something about it now.
  3. It names the service, the symptom and a runbook link.
  4. It fires rarely enough to be trusted. Pages that are usually ignorable train people to ignore pages.

An SLO in numbers. A 99.9% monthly success target allows 0.1% failures, the error budget. With 10 million requests a month, that is 10,000 failed requests. A burn-rate alert fires when failures are using that budget so quickly that it would be gone in days, rather than when a single request fails. More: SLIs, SLOs and SLAs, error budgets and burn rates and good alerts.

42. What is a correlation ID or trace ID, and how should it propagate between services? ​

The short answer

A correlation ID is a unique ID attached to one request, or one business flow, and carried through every service, queue message and log line it touches, so you can find all of them later with one search. Today the standard is the OpenTelemetry trace ID, carried in the W3C traceparent header over HTTP and in message headers for queues. Create it at the edge if the request has none, accept it from trusted callers, add it to every log line, return it to the client, such as in an X-Request-Id response header, and pass it on every outgoing call and published message.

gatewaycreates 4bf92forderslogs 4bf92fpaymentslogs 4bf92fqueue messageheader: 4bf92femail workerlogs 4bf92fheader
One ID follows the request everywhere. It crosses HTTP calls in a header and queue hops in message headers, so a single search for 4bf92f returns every log line and span from all four services.
ts
import { randomUUID } from 'node:crypto'
import { trace, context, propagation } from '@opentelemetry/api'

app.use((req, res, next) => {
  const traceId = trace.getActiveSpan()?.spanContext().traceId ?? randomUUID()  // (1)
  res.set('X-Request-Id', traceId)                                              // (2)
  req.log = logger.child({ traceId })                                           // (3)
  next()
})

export async function publish(topic: string, payload: object) {
  const headers: Record<string, string> = {}
  propagation.inject(context.active(), headers)                                 // (4)
  await broker.publish(topic, payload, { headers })
}
  1. Use the trace ID that OpenTelemetry already created for this request, or a new random one if tracing is off.
  2. Give it back to the client, so a user or support agent can quote it in a bug report.
  3. A child logger stamps the ID on every log line for this request.
  4. Copy traceparent into the message headers. The consumer extracts it with propagation.extract, so its spans and logs continue the same trace.

Rules: never trust an incoming ID from the public internet for anything but correlation, generate one if missing, and put it on audit records too (audit logging).

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Production & Operations
Performance
Scaling
Edge
Deploying
Delivery
Observability
Other topics
Nothing picked yet