Tracing

Tracing records each request as spans in a distributed trace, so you can follow a call through Arion into the upstream and across the rest of your system. Use it to find where latency lives; use metrics for aggregate rates.

Before you begin

Tracing is a build feature: compile with --features tracing. You also need an OpenTelemetry collector (or any OTLP gRPC endpoint) to receive the spans.

Enable it

Tracing is configured per http_connection_manager, with Envoy's tracing block and the OpenTelemetry provider:

# inside the HttpConnectionManager typed_config
tracing:
  random_sampling:
    value: 100.0
  provider:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
      service_name: "arion-gateway"
      grpc_service:
        google_grpc:
          target_uri: "http://127.0.0.1:4317"
          stat_prefix: "arion"

service_name is how the gateway appears in your tracing backend; target_uri is the OTLP gRPC endpoint.

What Arion emits

For each traced request Arion records a server span, enriched with OpenTelemetry semantic-convention attributes: http.request.method, url.path, url.query, url.scheme, http.response.status_code, protocol name and version, user_agent.original, plus upstream.cluster.name and upstream.address for the chosen endpoint. Responses with a 5xx status mark the span as an error.

Variations

Sampling

Three percentages (0–100) control how much is traced, matching Envoy's semantics: client_sampling for requests where the client already forced tracing, random_sampling for ordinary requests, and overall_sampling as a final ceiling over both. Requests arriving with a sampled parent context stay sampled, so traces are not broken mid-flight.

Propagation formats

Arion reads the incoming trace context from, in order of precedence: W3C Trace Context (traceparent/tracestate), Jaeger (uber-trace-id), B3 single header (b3), and B3 multi-header (x-b3-traceid/x-b3-spanid/x-b3-sampled). Whatever format a request arrives with is the format re-injected toward the upstream, so mixed estates keep working. When Arion is the trace root, it generates a new trace ID and propagates W3C traceparent.

Upstream client spans

Set spawn_upstream_span: true in the tracing block to emit a separate client span per upstream attempt, child of the server span. Retries then show up as individual spans with their own endpoint attributes.

Verify

Send a request carrying a sampled context and watch your collector:

curl -s -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' \
  http://localhost:8080/

Expected: a span from service arion-gateway under trace 4bf92f3577b34da6a3ce929d0e0e4736, with the request's method, path and status attributes. %TRACE_ID% in an access log format prints the same ID, which is handy for correlating the two.

Limits

The only supported provider is envoy.config.trace.v3.OpenTelemetryConfig; Zipkin, Datadog and other Envoy tracers are rejected at load.
Export is OTLP over gRPC via google_grpc.target_uri; the provider's http_service, sampler and resource_detectors fields are not supported.
Tracing applies to HTTP traffic through the connection manager; the TCP proxy does not emit spans.
A malformed incoming trace header is ignored: the request proceeds normally and is treated as a new trace root, not rejected.

Reference and guides