---
title: "Tracing"
url: "/docs/observe/tracing/"
kind: "feature"
summary: "Distributed tracing with OpenTelemetry: OTLP export, sampling controls, and W3C, B3 and Jaeger propagation."
---

> Arion site index: [/llms.txt](/llms.txt)

# Tracing

Distributed tracing with OpenTelemetry: OTLP export, sampling controls, and W3C, B3 and Jaeger propagation.

Tracing records each request as spans in a distributed trace, so you can follow a call through Arion into the upstream and across the rest of your system. Use it to find where latency lives; use [metrics](@/docs/observe/metrics.md) for aggregate rates.

## Before you begin

Tracing is a build feature: compile with `--features tracing`. You also need an OpenTelemetry collector (or any OTLP gRPC endpoint) to receive the spans.

## Enable it

Tracing is configured per `http_connection_manager`, with Envoy's `tracing` block and the OpenTelemetry provider:

```yaml,name=arion.yaml,arion=fragment
# inside the HttpConnectionManager typed_config
tracing:
  random_sampling:
    value: 100.0
  provider:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
      service_name: "arion-gateway"
      grpc_service:
        google_grpc:
          target_uri: "http://127.0.0.1:4317"
          stat_prefix: "arion"
```

`service_name` is how the gateway appears in your tracing backend; `target_uri` is the OTLP gRPC endpoint.

## What Arion emits

For each traced request Arion records a server span, enriched with OpenTelemetry semantic-convention attributes: `http.request.method`, `url.path`, `url.query`, `url.scheme`, `http.response.status_code`, protocol name and version, `user_agent.original`, plus `upstream.cluster.name` and `upstream.address` for the chosen endpoint. Responses with a 5xx status mark the span as an error.

## Variations

### Sampling

Three percentages (0–100) control how much is traced, matching Envoy's semantics: `client_sampling` for requests where the client already forced tracing, `random_sampling` for ordinary requests, and `overall_sampling` as a final ceiling over both. Requests arriving with a sampled parent context stay sampled, so traces are not broken mid-flight.

### Propagation formats

Arion reads the incoming trace context from, in order of precedence: W3C Trace Context (`traceparent`/`tracestate`), Jaeger (`uber-trace-id`), B3 single header (`b3`), and B3 multi-header (`x-b3-traceid`/`x-b3-spanid`/`x-b3-sampled`). Whatever format a request arrives with is the format re-injected toward the upstream, so mixed estates keep working. When Arion is the trace root, it generates a new trace ID and propagates W3C `traceparent`.

### Upstream client spans

Set `spawn_upstream_span: true` in the `tracing` block to emit a separate client span per upstream attempt, child of the server span. Retries then show up as individual spans with their own endpoint attributes.

## Verify

Send a request carrying a sampled context and watch your collector:

```bash
curl -s -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' \
  http://localhost:8080/
```

Expected: a span from service `arion-gateway` under trace `4bf92f3577b34da6a3ce929d0e0e4736`, with the request's method, path and status attributes. `%TRACE_ID%` in an [access log](@/docs/observe/access-logs.md) format prints the same ID, which is handy for correlating the two.

## Limits

- The only supported provider is `envoy.config.trace.v3.OpenTelemetryConfig`; Zipkin, Datadog and other Envoy tracers are rejected at load.
- Export is OTLP over gRPC via `google_grpc.target_uri`; the provider's `http_service`, `sampler` and `resource_detectors` fields are not supported.
- Tracing applies to HTTP traffic through the connection manager; the TCP proxy does not emit spans.
- A malformed incoming trace header is ignored: the request proceeds normally and is treated as a new trace root, not rejected.