Distributed Tracing with OpenTelemetry: One Request Across Six Services

I could tell you our checkout was slow. I could not tell you where. Every service logged fine, every dashboard was green, and the number that mattered — p95 latency — was somewhere between the API gateway and the payment provider.

Logs answer "what happened on this machine." Metrics answer "how bad, and how often." Neither answers "where did the time go." That needs the request to carry its own history, which is what a trace is.

The three words you need

A trace is one request's journey, made of spans — one unit of work each: an HTTP call, a database query, a cache lookup. A span records a start time, an end time, attributes, and the span that called it.

The crucial part is context propagation. Something at the edge generates a trace ID and every hop passes it forward. In HTTP that's the W3C traceparent header:

code

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Send and read that header and you get a tree for free. Don't, and you get isolated spans that never join up — the most common way a tracing rollout produces pretty pictures that answer nothing.

Instrument without rewriting the service

OpenTelemetry separates the API, the SDK, and the exporter into three layers you can swap without touching application code.

Auto-instrumentation where it exists — HTTP clients, database drivers, common frameworks. Spans around the calls that matter with no line changed.
Manual spans for your own boundaries: "pricing calculation", "inventory check." Two or three per service, not twenty.
The exporter sends them somewhere. In practice: OTLP to a collector.

bash

export OTEL_SERVICE_NAME=checkout
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
node -r @opentelemetry/auto-instrumentations-node/register server.js

The service name and the endpoint are the two settings people forget, and without them spans arrive anonymous or don't arrive at all.

The collector is the whole trick

Running an SDK exporter straight to a vendor from every pod is the architecture people build first and regret second. The OpenTelemetry Collector sits in between: agent mode on each node receives, batches and compresses locally; gateway mode centrally fans in, enriches with Kubernetes attributes, samples, and fans out.

Once central you can change the backend without redeploying services, drop noisy attributes in one place, and sample with context rather than blindly.

What I sample, and why

Full-fidelity tracing on 100% of traffic is expensive and, for most systems, unnecessary. But naive head sampling — keep 1% of traces — throws away exactly the slow ones you care about, because at request time you don't know yet whether a trace will be slow.

Tail sampling in the collector is the answer: buffer spans for a trace, then decide on the finished result.

code

processors:
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code { statusCodes: [ERROR] }
      - name: keep-slow
        type: latency { latency: { p99: 1000ms } }
      - name: sample-the-rest
        type: probabilistic { probabilistic: { sampling_percentage: 5 } }

Errors: keep all. Slow traces: keep all. Everything else: a thin statistical sample. You store what you'd actually investigate and nothing else.

The costs nobody warns you about

Attribute cardinality. A user ID or a raw URL with query params on every span multiplies storage and often breaks the backend. Put IDs on the span that owns the decision, not on every child.
Span volume. One database call per row is a classic. Batch, or sample leaf spans aggressively.
Storage. Traces are the most expensive of the three signals — which is why the sampling policy is a design decision, not an afterthought.

What it will not fix

Tracing won't catch a memory leak and won't survive a protocol nobody instruments. One uninstrumented hop in the middle breaks the tree, and the tool gets blamed.

It also doesn't replace metrics-driven observability — metrics say checkout is slow, the trace says it's the inventory service, and logs say why. Where eBPF shines is traffic you didn't instrument: the sidecar-free case.

What I'd ship first

W3C propagation verified end to end, on one request path, before anything else.
Auto-instrumentation plus two or three manual spans per service.
A collector in agent + gateway mode, tail sampling keeping errors and slow traces.
Attributes: service, endpoint template, status. Not user IDs, not raw URLs.

Get one trace showing a full journey and the conversation changes — people stop guessing and start reading.

Summary

Tracing answers the question the other two signals can't: where the time went. Propagate context, instrument the boundaries, and put a collector in the middle so you sample with context instead of luck.