Health checks and resilience

Resilience in Arion is three mechanisms working together: active health checks take bad endpoints out of rotation, circuit breakers cap the work a cluster is allowed to absorb, and retry policies re-attempt requests that are safe to repeat. Configure all three on any cluster that fronts a service you do not fully trust.

Active health checks

A cluster takes one health check, which probes every endpoint on its own schedule and removes failing ones from load balancing:

clusters:
  - name: api
    type: STATIC
    lb_policy: ROUND_ROBIN
    health_checks:
      - timeout: 1s
        interval: 5s
        healthy_threshold: 3
        unhealthy_threshold: 2
        http_health_check:
          path: /health

timeout, interval, and the two thresholds are required. An endpoint must fail unhealthy_threshold checks to leave rotation and pass healthy_threshold checks to return. Jitter fields (initial_jitter, interval_jitter, interval_jitter_percent) spread probes out, and unhealthy_interval, unhealthy_edge_interval, and healthy_edge_interval re-check flapping endpoints on a different cadence.

The HTTP checker sends GET requests and treats status 200 as healthy unless expected_statuses lists other ranges; its host defaults to the cluster name, and its codec_client_type must match the cluster's upstream protocol. Two other checkers exist:

tcp_health_check: {}         # connect-only probe; optional send/receive payloads

grpc_health_check:
  service_name: api          # standard gRPC health protocol; cluster must speak HTTP/2

Circuit breakers

Circuit breakers refuse work beyond a threshold instead of queueing it, so one slow upstream cannot absorb the whole proxy. Limits are per cluster and per routing priority:

clusters:
  - name: api
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 512
          max_requests: 200
          max_retries: 3
          track_remaining: true

Defaults are 1024 connections, 1024 requests, and 3 concurrent retries per priority (DEFAULT and HIGH — a route's priority field selects which set applies). When a limit is hit, new work is rejected immediately; track_remaining: true exposes the remaining budget as metrics. max_retries is the backstop that keeps retry storms from amplifying an outage.

Retries

A retry_policy on a route (or a whole virtual host) retries failed requests against the same cluster, picking a fresh endpoint each attempt:

route:
  cluster: api
  timeout: 15s
  retry_policy:
    retry_on: connect-failure,reset,5xx
    num_retries: 2
    per_try_timeout: 2s
    retry_back_off:
      base_interval: 0.025s
      max_interval: 0.25s

retry_on is a comma-separated list of conditions: 5xx, gateway-error, reset, connect-failure, refused-stream, envoy-ratelimited, retriable-4xx, retriable-status-codes (with retriable_status_codes), retriable-headers (with retriable_headers), and http3-post-connect-failure. num_retries defaults to 1. Attempts back off exponentially from base_interval (default 25ms) up to max_interval (default 10x the base). retriable_request_headers restricts retries to requests that carry a matching header. Retry only what is idempotent: connect-failure and reset are safe almost everywhere, while 5xx on a POST can duplicate work.

Timeouts

Timeouts bound how long a bad request can hold resources. The route timeout (default 15s, 0s disables) caps the whole upstream exchange and contains every retry; per_try_timeout caps a single attempt. The cluster's connect_timeout bounds TCP establishment, and common_http_protocol_options.idle_timeout reaps idle upstream connections — see upstreams.

Verify

curl -s http://127.0.0.1:9901/clusters | grep health_flags

api::127.0.0.1:5678::health_flags::healthy
api::127.0.0.1:5679::health_flags::/failed_active_hc

Limits

Outlier detection (passive health checking) is not supported — use active checks.
One health check per cluster; the HTTP checker only sends GET.
max_pending_requests and retry_budget circuit-breaker fields are not supported.
Health-check event logging and reuse_connection are not supported.
Retries count against the cluster's max_retries circuit breaker.

Field-by-field detail lives in the cluster and route references.