Canary with weighted clusters

Outcome

This guide builds a canary release: one route that splits traffic by weight between a stable cluster (90%) and a canary cluster (10%), both health-checked, with failed requests retried. Use it to expose a new version to a controlled slice of real traffic before committing to it.

Prerequisites

An arion binary — see install.
Two deployments of your service: the current version and the candidate. The verify section fakes them with echo servers.
curl, and Docker for the fakes.

The architecture

A listener on 0.0.0.0:8080 runs the HTTP connection manager with a single catch-all route.
Instead of naming one cluster, the route names weighted_clusters: stable at weight 90 and canary at weight 10.
stable holds two endpoints balanced with LEAST_REQUEST; canary holds one. Both probe /healthz every five seconds, and the virtual host retries 5xx twice.

The complete config

# arion.yaml, composed from 8 features.
logging:
  log_level: info
envoy_bootstrap:
  static_resources:
    listeners:
      - name: http_8080
        address:
          socket_address: { address: 0.0.0.0, port_value: 8080 }
        filter_chains:
          - name: http_8080_chain
            filters:
              - name: http_gateway
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                  http_filters:
                    - name: envoy.filters.http.router
                      typed_config:
                        "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                  route_config:
                    name: http_8080_routes
                    virtual_hosts:
                      - name: backend
                        domains: ["*"]
                        retry_policy:
                          retry_on: 5xx
                          num_retries: 2
                        routes:
                          - match:
                              prefix: /
                            route:
                              weighted_clusters:
                                clusters:
                                  - name: stable
                                    weight: 90
                                  - name: canary
                                    weight: 10
    clusters:
      - name: stable
        connect_timeout: 5s
        type: STATIC
        lb_policy: LEAST_REQUEST
        load_assignment:
          cluster_name: stable
          endpoints:
            - lb_endpoints:
                - endpoint:
                    address:
                      socket_address: { address: 10.0.2.11, port_value: 8080 }
                - endpoint:
                    address:
                      socket_address: { address: 10.0.2.12, port_value: 8080 }
        health_checks:
          - timeout: 1s
            interval: 5s
            healthy_threshold: 3
            unhealthy_threshold: 2
            http_health_check:
              path: /healthz
      - name: canary
        connect_timeout: 5s
        type: STATIC
        load_assignment:
          cluster_name: canary
          endpoints:
            - lb_endpoints:
                - endpoint:
                    address:
                      socket_address: { address: 10.0.2.21, port_value: 8080 }
        health_checks:
          - timeout: 1s
            interval: 5s
            healthy_threshold: 3
            unhealthy_threshold: 2
            http_health_check:
              path: /healthz

Walkthrough

The weighted route

The only structural difference from a plain load balancer sits in the route action: route.weighted_clusters.clusters lists two entries with a name and a weight. Each request is assigned to a cluster with probability proportional to its weight over the sum — the weights here sum to 100, so they read directly as percentages, but 9 and 1 would behave identically. Everything else about routing is unchanged; see routing.

Retries

The virtual host's retry_policy (retry_on: 5xx, num_retries: 2) covers the weighted route like any other, so a canary that starts throwing 500s costs retries rather than immediately surfacing errors — useful cover, but watch for it masking a broken canary. Retry mechanics are described in health checks and resilience.

The stable cluster

stable is a STATIC cluster with two endpoints and lb_policy: LEAST_REQUEST: the 90% share is then spread across both machines by in-flight load. Its health check — GET /healthz every 5s, out after 2 failures, back after 3 passes — ejects a bad instance without touching the canary split.

The canary cluster

canary is the same shape with one endpoint and no lb_policy line, leaving the default round-robin (trivial with a single endpoint). It carries an identical health check. Weight decides how much traffic a cluster is assigned; the health check decides whether its endpoints can take it — a canary that is sick and unchecked would keep receiving its full share, so check the canary exactly like stable.

Run and verify

Fake the two versions with echo servers — both stable endpoints answer stable, the canary answers canary:

docker run -d --rm -p 5678:5678 hashicorp/http-echo -text=stable
docker run -d --rm -p 5679:5678 hashicorp/http-echo -text=stable
docker run -d --rm -p 5680:5678 hashicorp/http-echo -text=canary

For a local run, change the two stable endpoints to 127.0.0.1:5678 and 127.0.0.1:5679, and the canary endpoint to 127.0.0.1:5680. Start Arion:

arion --config arion.yaml

Send a hundred requests and count the answers:

for i in $(seq 1 100); do curl -s http://127.0.0.1:8080/; done | sort | uniq -c

  91 stable
   9 canary

The assignment is probabilistic, so expect roughly 90/10, not exactly — over a few hundred requests the ratio settles. To watch a shift live, change the weights to 50/50, restart Arion, and re-run the loop.

In production nobody curls for the split. Give the connection manager an access log whose template includes the %UPSTREAM_CLUSTER% operator — as in the L7 load balancer guide — and every log line names which side served the request, ready to be counted and compared. See access logs.

Production notes

Shift in steps — 10, 25, 50, 100 — and hold each step long enough to see real error rates and latencies, not just a smoke test. Weight changes are a two-line edit.
Weights must be at least 1 — Arion refuses a zero weight. To roll back fully, remove the canary entry from weighted_clusters (or revert to a plain cluster: stable action); the canary cluster itself can stay defined, warm and health-checked, for the next attempt.
For deterministic testing, put a header-matching route above the weighted one — routes match in order, so a request carrying your header (say x-canary: always) can be pinned to the canary cluster while everyone else rolls the dice. Header matchers are covered in routing.
Weight edits in a static file need a restart to apply. If canaries are routine, serve the route table from a control plane so weights shift without restarts — see xDS control planes.
Keep both clusters' health checks identical. A canary with a laxer check can hold traffic while sick, and that noise is indistinguishable from a bad release.

The rest of this configuration is the L7 load balancer pattern; route fields, including weighted clusters and header matchers, are listed in the route reference.