Introducing the Arion control plane

When we set out to build Arion, we made the decision to keep close compatability the Envoy configuration model. We made this choice because, whilst the Envoy configuration model has some ergonomic issues and had become a little bloated in places, it had extensive coverage. Coupled with the xDS discovery protocol, Envoy led the way in terms of dynamic (re)configuration in Cloud L7 networking. In choosing Envoy's native configuration model over xDS, we could also leverage the existing Envoy ecosystem of xDS control plane software.

Having worked with a bunch of the controllers in the Envoy xDS ecosystem over a number of years, however, we got frustrated and decided to build our own. Arion-ctl is our own control plane, still fully xDS compatible, but small, neat and purpose built with a bunch of usability features that we felt were missing from others in the ecosystem.

You can deploy it in two ways: as a standalone service that you manage with files and operator commands, or as a Kubernetes controller that reads Gateway API resources from your cluster. Each option delivers configuration to running Arion proxies.

Arion remains the data plane: it accepts connections and forwards traffic. Arion-ctl supplies listeners, routes, clusters, endpoints, and secrets as configuration dynamically over xDS. Once a proxy has its configuration, handling a request does not require any interaction with the control plane.

Choose a deployment option

The standalone service and Kubernetes controller are separate deployment options. Choose the one that fits how you manage your gateways. You do not need to install or configure the standalone service before using the Kubernetes controller.

Deployment option	How you manage configuration	Where configuration is stored
Standalone service	Apply YAML resource files with arionctl, or use the optional arionctl shell	A durable snapshot saved by the standalone service
Kubernetes controller	Create Gateway API resources through the Kubernetes API	Kubernetes cluster state

The command line and interactive shell are two interfaces to the same standalone service. The Kubernetes controller has its own setup and reads its configuration from Kubernetes resources. Both deployments use the same library internally to deliver configuration over xDS.

Standalone deployment: files and an optional shell

Choose standalone when you want to run a control-plane service without depending on Kubernetes. The service stores your configuration and supplies it to connected proxies. You can manage it entirely through YAML files and arionctl commands; the Elixir shell is optional.

A fleet groups the configuration delivered to a set of proxies. Each proxy selects its fleet with node.cluster in its bootstrap. The standalone default is arion.

Apply configuration from a file

The standalone service accepts YAML resource documents. Each identifies a fleet, resource kind, and name, followed by the resource specification. These are the standalone service's resource documents; Kubernetes Gateway API uses its own resource definitions. The proxy still has a separate bootstrap containing its identity and control-plane connection settings.

For a working example, follow the standalone walkthrough through the initial HTTP configuration. This starts backend A on port 18081, the control plane on 15051, and a proxy listener on 18080. Keep those processes running and retain the operator terminal's release environment.

Save the overview's Cluster example as backend.yaml. It describes the existing backend resource in fleet arion, pointing to backend A. From the operator terminal:

arionctl validate -f backend.yaml
arionctl apply -f backend.yaml
arionctl get --fleet arion
arionctl status
curl --fail http://127.0.0.1:18080/

The request returns hello from backend-a. Reapplying this Cluster preserves the Listener from the walkthrough. Apply replaces each supplied resource's full specification and keeps identities absent from the input. Removal uses an explicit delete command. Editing the file alone changes nothing; the service loads it when instructed.

The service validates the whole input, writes and syncs a temporary snapshot, renames it into place, then publishes the desired state to the fleet caches. Invalid input is rejected before saving. If writing the snapshot fails, the previous desired state remains active.

get shows saved configuration. status shows resource counts and connected proxies' acknowledgement state. An accepted version appears as ["ack", "<version>"]; a rejected update appears as ["nack", "<version>", "<error>"]. A successful apply means the desired state was saved and published. Proxy responses arrive separately, and a real request checks whether the intended routing works.

Optional: update the same service from the shell

If you want to construct configuration in code, attach to the running standalone service through IEx, Elixir's interactive shell. This uses the same configuration store as the file commands above. Start a second backend in another terminal, from the control-plane repository:

python3 docs/examples/backend.py --port 18082 --name backend-b

In the operator terminal, attach to the service:

arionctl shell

At the IEx prompt, construct a Cluster and apply it:

alias Arion.ControlPlane.Helpers.Cluster

cluster_b = Cluster.static("backend", "127.0.0.1", [18082], connect_timeout: 1)
{:ok, _} = Arion.Ctl.apply_resources([cluster: cluster_b], fleet: "arion")
Arion.Ctl.get("arion")
Arion.Ctl.status()

Cluster.static builds a protobuf value. apply_resources converts it into the same resource documents used by the file interface, validates them, and saves them through Arion.Ctl. The new backend survives a service restart using the same state directory.

From an ordinary terminal, request the proxy again:

curl --fail http://127.0.0.1:18080/

After the update is accepted, it returns hello from backend-b. The Listener stayed in place. The proxy and control plane kept running.

The shell also permits ordinary Elixir functions, loops, and transformations. You can compute a collection of resources, inspect it, then apply the collection together. Keep standalone mutations on Arion.Ctl; publishing directly through the library's Service API would bypass the snapshot.

To leave the attached shell, press Ctrl+G, then enter q. Back in the operator terminal, export the desired state:

(umask 077; arionctl get --fleet arion > arion-resources.json)
arionctl validate -f arion-resources.json

The JSON can be applied again with the same CLI. Exports can contain private keys from Secret resources, so keep them in protected storage. The shell walkthrough covers inspection, deletion, and restart verification.

Kubernetes deployment: Gateway API

Choose this option when you want Kubernetes resources to define your gateways and routes. Install the Kubernetes controller with the Helm chart, then manage configuration through the Kubernetes API, using tools such as kubectl or your GitOps workflow. This is an alternative to the standalone setup above; you do not apply these resources with arionctl or use its shell.

A GatewayClass selects the controller, a Gateway defines traffic entry points, and routes attach rules to them. The controller watches these Kubernetes objects, resolves references and attachment permissions, then translates the result into the xDS configuration that Arion proxies receive.

HTTPRoute covers matching, weighted backends, header changes, redirects, rewrites, timeouts, and retries. GRPCRoute adds service and method matching. Kubernetes Secrets supply certificates for HTTPS, while TCPRoute and TLSRoute describe stream forwarding and TLS handling. Services resolve to ready EndpointSlice addresses. InferencePool integration uses an endpoint picker and the pool's ready Pods. The controller reference describes the supported fields and prerequisites for each feature.

The AI inference gateway overview explains how Arion asks the endpoint picker to select a ready model server Pod for each request. The inference routing guide shows the Gateway API and InferencePool resources that configure it.

Each Gateway gets a fleet named gateway/<namespace>/<name>. Every controller replica derives and serves its own configuration from Kubernetes. With the chart's clustering enabled, an elected leader handles Kubernetes status and proxy provisioning. Replicas do not share a standalone snapshot or replicate xDS stream state.

The controller can provision a Deployment, Service, ServiceAccount, and bootstrap ConfigMap for each Gateway. ArionGatewayParameters controls details such as image, replicas, and service type. Within this Kubernetes deployment, you can instead manage the proxy workloads yourself while the controller continues to deliver their configuration.

Why Elixir fits the design

A controller repeatedly loads or watches resource definitions, resolves their relationships, transforms them into configuration a data plane understands, and delivers updates as the desired state changes. We chose Elixir because it makes both the transformations and the ongoing coordination simpler to express and maintain.

A runtime built for distributed systems

Elixir runs on the Erlang virtual machine, the BEAM. It provides lightweight, isolated processes that communicate by passing messages. Erlang's OTP libraries add supervision: processes can be organised into trees that define how their work is started and restarted after failures. Communication also extends across connected BEAM nodes, with built-in remote messaging, function calls, and process monitoring.

These capabilities make writing distributed applications much simpler. A controller needs to keep resource watches running, schedule reconciliation, track connected proxies, and coordinate work across replicas. The Erlang process model gives us a consistent way to organise those activities, backed by runtime features and libraries built for that purpose. In Arion, this also supports direct interaction with a running service through remote commands and an interactive shell.

A concise language for resource transformations

Much of a controller's work is data transformation: take resources in one representation, resolve what they mean together, and construct another representation. Elixir's concise syntax, pattern matching, collection functions, and pipelines suit this work particularly well. We can express the transformation as a sequence of small functions, each handling one part of the problem. Immutable data makes the inputs and outputs of each step easier to follow.

In other controllers we've worked with, we've seen many thousands of lines of code devoted just to translating between representations. A language can be straightforward to get started with and still require a surprising amount of code for that task. We think that is a reason to question the fit between the language and the problem. Elixir lets us express these transformations more directly, keeping the implementation smaller and easier to understand.

More room for the operator experience

A smaller, simpler implementation also gives us more room to build a richer operator experience. The functions used to construct, inspect, validate, and apply configuration are available in the live Elixir shell. An operator can explore the current state, transform a collection of resources, and apply the result without building a separate tool for each operation.

In the standalone service, the CLI and shell call the same application API. They share validation and durable storage, so using the programmable interface keeps the same persistence behaviour as applying a file. Elixir makes that kind of interactive experience a natural extension of the implementation.

For Elixir developers: embed the library

An Elixir application can also supervise Arion.ControlPlane directly and construct configuration from its own service data. This is useful when you are building a custom control plane. The application owns persistence and recovery, and publishes through Arion.ControlPlane.Service; Service.replace replaces a fleet's complete resource set. See the complete library example for startup and publication details.

State ownership and operational checks

Recovery follows the configuration owner. Standalone restores its snapshot before accepting xDS connections and requires one service instance per state directory. Kubernetes controller replicas rebuild their configuration from cluster resources and wait for their initial lists and first publication before serving discovery.

A cache replacement does not make updates across resource types atomic at every proxy. ACKs describe accepted resource versions; they do not establish backend reachability. Inspect status and proxy logs, then send traffic through the route being changed.

Fleets separate configuration but do not authenticate clients. Keep discovery and operator connections on trusted networks. The deployment guide covers placement, storage, and availability responsibilities.

To get started, choose the standalone walkthrough for files and operator commands, or the Helm guide for Kubernetes Gateway API. The control-plane overview has more examples, and the general xDS guide explains the proxy side of the connection.