---
title: "Health checks and resilience"
url: "/docs/traffic/health-checks-and-resilience/"
kind: "feature"
summary: "Detect bad endpoints with active health checks, stop cascading failure with circuit breakers, and retry what is safe."
---

> Arion site index: [/llms.txt](/llms.txt)

# Health checks and resilience

Detect bad endpoints with active health checks, stop cascading failure with circuit breakers, and retry what is safe.

Resilience in Arion is three mechanisms working together: active health checks take bad endpoints out of rotation, circuit breakers cap the work a cluster is allowed to absorb, and retry policies re-attempt requests that are safe to repeat. Configure all three on any cluster that fronts a service you do not fully trust.

## Active health checks

A cluster takes one health check, which probes every endpoint on its own schedule and removes failing ones from load balancing:

```yaml,arion=fragment
clusters:
  - name: api
    type: STATIC
    lb_policy: ROUND_ROBIN
    health_checks:
      - timeout: 1s
        interval: 5s
        healthy_threshold: 3
        unhealthy_threshold: 2
        http_health_check:
          path: /health
```

`timeout`, `interval`, and the two thresholds are required. An endpoint must fail `unhealthy_threshold` checks to leave rotation and pass `healthy_threshold` checks to return. Jitter fields (`initial_jitter`, `interval_jitter`, `interval_jitter_percent`) spread probes out, and `unhealthy_interval`, `unhealthy_edge_interval`, and `healthy_edge_interval` re-check flapping endpoints on a different cadence.

The HTTP checker sends `GET` requests and treats status 200 as healthy unless `expected_statuses` lists other ranges; its `host` defaults to the cluster name, and its `codec_client_type` must match the cluster's upstream protocol. Two other checkers exist:

```yaml,arion=fragment
tcp_health_check: {}         # connect-only probe; optional send/receive payloads
```

```yaml,arion=fragment
grpc_health_check:
  service_name: api          # standard gRPC health protocol; cluster must speak HTTP/2
```

## Circuit breakers

Circuit breakers refuse work beyond a threshold instead of queueing it, so one slow upstream cannot absorb the whole proxy. Limits are per cluster and per routing priority:

```yaml,arion=fragment
clusters:
  - name: api
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 512
          max_requests: 200
          max_retries: 3
          track_remaining: true
```

Defaults are 1024 connections, 1024 requests, and 3 concurrent retries per priority (`DEFAULT` and `HIGH` — a route's `priority` field selects which set applies). When a limit is hit, new work is rejected immediately; `track_remaining: true` exposes the remaining budget as [metrics](@/docs/observe/metrics.md). `max_retries` is the backstop that keeps retry storms from amplifying an outage.

## Retries

A `retry_policy` on a route (or a whole virtual host) retries failed requests against the same cluster, picking a fresh endpoint each attempt:

```yaml,arion=fragment
route:
  cluster: api
  timeout: 15s
  retry_policy:
    retry_on: connect-failure,reset,5xx
    num_retries: 2
    per_try_timeout: 2s
    retry_back_off:
      base_interval: 0.025s
      max_interval: 0.25s
```

`retry_on` is a comma-separated list of conditions: `5xx`, `gateway-error`, `reset`, `connect-failure`, `refused-stream`, `envoy-ratelimited`, `retriable-4xx`, `retriable-status-codes` (with `retriable_status_codes`), `retriable-headers` (with `retriable_headers`), and `http3-post-connect-failure`. `num_retries` defaults to 1. Attempts back off exponentially from `base_interval` (default 25ms) up to `max_interval` (default 10x the base). `retriable_request_headers` restricts retries to requests that carry a matching header. Retry only what is idempotent: `connect-failure` and `reset` are safe almost everywhere, while `5xx` on a POST can duplicate work.

## Timeouts

Timeouts bound how long a bad request can hold resources. The route `timeout` (default 15s, `0s` disables) caps the whole upstream exchange and contains every retry; `per_try_timeout` caps a single attempt. The cluster's `connect_timeout` bounds TCP establishment, and `common_http_protocol_options.idle_timeout` reaps idle upstream connections — see [upstreams](@/docs/traffic/upstreams-and-load-balancing.md).

## Verify

```bash
curl -s http://127.0.0.1:9901/clusters | grep health_flags
```

```text
api::127.0.0.1:5678::health_flags::healthy
api::127.0.0.1:5679::health_flags::/failed_active_hc
```

## Limits

- Outlier detection (passive health checking) is not supported — use active checks.
- One health check per cluster; the HTTP checker only sends `GET`.
- `max_pending_requests` and `retry_budget` circuit-breaker fields are not supported.
- Health-check event logging and `reuse_connection` are not supported.
- Retries count against the cluster's `max_retries` circuit breaker.

Field-by-field detail lives in the [cluster](@/reference/resources/clusters.md) and [route](@/reference/resources/routes.md) references.