---
title: "Troubleshooting"
url: "/docs/operate/troubleshooting/"
kind: "feature"
summary: "Symptom-led fixes: startup refusals, certificate and SNI errors, listener and upstream failures, JWT 401s and missing endpoints."
---

> Arion site index: [/llms.txt](/llms.txt)

# Troubleshooting

Symptom-led fixes: startup refusals, certificate and SNI errors, listener and upstream failures, JWT 401s and missing endpoints.

This page maps symptoms to causes and fixes. Two tools solve most incidents: rerun with `RUST_LOG=debug` and read the error chain (each `caused by:` line narrows it down), and query the [admin interface](@/docs/observe/admin-interface.md) to see what the process actually loaded.

## Arion refuses to start on my Envoy config

This is deliberate. Arion validates every field and refuses configuration it cannot honor, instead of silently ignoring it — the error names the exact path:

```text
failed to deserialize ... listeners[0].filter_chains[0] ...
caused by:
    Unsupported field: use_original_dst
```

Remove or replace the named field. `Unsupported enum variant` means the field exists but that value does not (an unsupported filter's `@type`, for example). The [supported configuration reference](@/reference/envoy-compatibility/supported-configuration.md) lists what converts; anything outside it must come out of the file before Arion will run it.

## "secret doesn't contain server name"

Your TLS certificate has no DNS name in its Subject Alternative Name extension. Arion takes the server name from the SAN (not the CN) to match SNI, so a SAN-less certificate is unusable. Regenerate it with a SAN — certificates cannot be amended after signing:

```bash
openssl x509 -in server-cert.pem -text -noout | grep -A2 "Subject Alternative Name"
```

If that prints nothing, reissue the certificate with `subjectAltName = DNS:api.example.com` and make sure the listener's `filter_chain_match.server_names` matches one of the SAN entries.

## The private key or certificate is rejected

Three frequent causes: the `certificate_chain` block contains `BEGIN PUBLIC KEY` instead of `BEGIN CERTIFICATE`; certificate and key are swapped between the two fields; or the key is PKCS#1 (`BEGIN RSA PRIVATE KEY`) while Arion expects PKCS#8 (`BEGIN PRIVATE KEY`). Convert with:

```bash
openssl pkcs8 -topk8 -nocrypt -in pkcs1-key.pem -out pkcs8-key.pem
```

## TLS works, but the wrong certificate is served

The client's SNI selects the filter chain. Test what a given name gets: `openssl s_client -connect host:8443 -servername api.example.com`. If the default chain answers, no `filter_chain_match.server_names` entry matched the SNI — check for exact-name versus wildcard mismatches, and remember `%REQUESTED_SERVER_NAME%` in an access log shows what clients actually sent.

## Connection refused on my listener

The listener is not bound where you are connecting. Check `curl http://127.0.0.1:9901/listeners` for the live `name::address` list. Usual causes: the listener binds `127.0.0.1` while you connect from outside (bind `0.0.0.0` in containers), the container port is not published, or the process is not running at all — configuration Arion cannot honor stops it at startup, so read the log before anything else.

## Upstream 5xx / no healthy endpoints

A 503 produced by Arion itself (rather than the backend) usually means no usable endpoint: the access-log operators `%RESPONSE_CODE_DETAILS%` and `%RESPONSE_FLAGS_LONG%` show `no_healthy_upstream` / `NoHealthyUpstream` when [health checks](@/docs/traffic/health-checks-and-resilience.md) have marked everything down, and `UpstreamConnectionFailure` when connects fail. Check `curl http://127.0.0.1:9901/clusters` for per-endpoint health and failure counters. If endpoints show connect failures, verify they are reachable *from the gateway host* and that `connect_timeout` is realistic. 504s point at response timeouts instead — see the route timeout before blaming the backend.

## JWT-protected requests always get 401

The token is being rejected, and the reasons are almost always one of: the token's `iss` claim does not exactly match the provider's `issuer`; the token's `aud` is not in `audiences`; or the JWKS is not available — for `remote_jwks`, the cluster it points at must be reachable so keys can be fetched (a fetch failure leaves no keys to validate with; a `local_jwks` file removes that dependency). Decode the token's claims, compare them character-for-character with the [JWT configuration](@/docs/security/jwt.md), and rerun with `RUST_LOG=debug` to see which check fails.

## xDS is connected, but updates never apply

Your control plane may be receiving NACKs. Search the Arion log for `rejecting configs with nack response` — the message lists each rejected resource and why (typically an unsupported field, as at startup). Decoding or conversion failures leave that response unapplied; later application failures can leave some changes applied. Check the [admin interface](@/docs/observe/admin-interface.md) and verify traffic before retrying a corrected configuration. See [xDS control planes](@/docs/operate/xds-control-planes.md).

## An admin or metrics endpoint returns 404

`/config_dump`, `/stats` and `/stats/prometheus` exist only in binaries built with the `config-dump`, `metrics` and `prometheus` features respectively — the same applies to access logging (`access-log`), tracing (`tracing`) and WASM (`wasm`). `curl http://127.0.0.1:9901/help` lists exactly what your build contains; if the endpoint is missing there, rebuild with the feature rather than debugging configuration.