Arion - Blog

Code mode: let the model write the code

2026-10-13T00:00:00+00:00

Arion MCP gateway turns REST services, OpenAPI documents and upstream MCP servers into one catalog of tools that agents can discover and call. Arion Code mode provides an alternative way for agents to discover and interact with MCP tools.

Arion MCP code mode works similarly to how code mode works in agent harnesses, but it runs in the context of the gateway. Every tool call made through Arion's code mode SDK goes through the full gateway policy stack in the same way a direct MCP tool call would.

Why support Code Mode

With conventional MCP, an agent calls one tool, waits, reads the result and decides the next call, over and over. Take a routine support request:

Find the customer by email.
Fetch their recent tickets.
Fetch their billing plan.
Open a ticket, at high priority if they are on the enterprise tier.

In many agent harnesses, this results in at least four tool calls, with model turns and eight crossings between the model and the gateway. This introduces additional latency. Worse, if the agent harness does not provide code mode natively or a similar optimization method, then all the raw responses are added to the model's context, including the ticket history it needed only long enough to count. This is both inefficient and wasteful.

A large catalog can add a lot of unncessary cost before any work starts. In many settings, adding a few MCP servers can result in tens and often hundreds of tool definitions that the model has to select from. Tool search helps, but, until MCP progressive discovery is standardized, the harness again needs custom code to effectively use tool search.

The model ends up doing coordination work that a few lines of code could do faster, more cheaply and more reliably.

The approach: one tool, one script

Code mode is enabled in the MCP gateway configuration. When code mode is enabled, it exposes exactly one tool called execute. The model, following instructions from the execute tool description generates code to call the Arion SDK and passes the program to Arion, Arion then runs the provided code in a sandbox. The code can call any MCP tool the agent is allowed to use, combine results, loop, branch and run calls in parallel, then return only the overall result of the program.

Figure 1. The tool calls do not disappear: they move to the gateway's side of the boundary.

The support request becomes one script:

const customer = await tools["crm.find_customer"]({ email: "ada@example.com" });
if (!customer.ok) return { error: "unknown customer" };

const [history, plan] = await Promise.all([
  tools["support.recent_tickets"]({ customer_id: customer.data.id }),
  tools["billing.current_plan"]({ customer_id: customer.data.id }),
]);

const ticket = await tools["support.create_ticket"]({
  ticket: {
    customer_id: customer.data.id,
    priority: plan.ok && plan.data.tier === "enterprise" ? "high" : "normal",
    summary: "Refund not received",
  },
});
return { ticket: ticket.data.id, seen: history.data.items.length };

The history and plan lookups run in parallel, the script branches on the plan, and the model gets back a small object instead of four raw responses. The four calls never touch its context.

Some small design choices help make this practical.

The tool describes itself, per caller. The execute description is written fresh for each caller. It lists the toolkits and namespaces that caller is allowed to see, and how to find more. It has a 4 KiB budget, so a gateway with thousands of tools still gives the model a compact starting point instead of a wall of text.

Discovery happens inside the script. A small catalog API lets the script search, list and describe tools on demand, so the model pulls in only the definitions it needs. catalog.tools.search({ query }) uses the same BM25 or embedding ranking as tool search, but without the extra round trip.

Every call returns one predictable shape. A tool call resolves to { ok: true, data } or { ok: false, error: { code, message, retryable } }. A failure is a value, not an exception, so the script checks ok and moves on, and failures stay readable.

On top of that, operators can give the model a head start. A toolkit is a named bundle of related tools with a short "how to use them together" note, authored in configuration. Where the raw tool list tells the model what exists, a toolkit guides the model in how common tasks can be performed: find the customer, then open a ticket, then notify the owner. A toolkit appears only if the caller may use every tool in it, so partial bundles never leak, and toolkits can be added or removed live over xDS. Toolkits are somewhat similar to skills, but are more tightly coupled to describing how to use a subset of tools from the catalog to achieve a specific task.

How it runs

Code mode is just an alternative operating mode for Arion MCP gateway. It changes how tools are called, never what is allowed.

Policy is always the same for both modes. A tool call from a script travels the same path as an ordinary tools/call. It lands on the one shared tool registry, which checks that the caller is permitted, validates the arguments against the tool's schema, enforces size and time limits, and records the same metrics. A script cannot reach anything its caller could not already reach, and it cannot skip a single check. The registry was reorganized into small, focused pieces so that both modes share it with no duplicated logic to drift apart.

Sandbox runtime Agent provided code runs in WASM using Wasmtime, a mature WebAssembly runtime, on QuickJS, a small and fast JavaScript engine compiled as a WASI component. Inside, the module JS runtime is highly constrained: no files, no network, no environment. The only interaction with outside world is through a very limited set of host calls: logging, tool invocations or tool catalog search.

Fast to start, nothing running when idle. The engine is initialized ahead of time with Wizer, so each run starts from a snapshot rather than booting an interpreter. Instances come from a pool, so starting one is quick. Each run still gets a fresh, empty environment, so one caller can never see another's data. The thread that enforces the compute budget runs only while a script is executing, so an idle gateway does no background work.

Two clocks, and limits on everything. A 5-second compute budget stops runaway code. A total-time limit, 30 seconds by default, bounds the whole run. Waiting on a slow backend does not count against compute time. Memory (64 MiB), tool calls per run (100), runs at once per listener (32), parallel calls within a run (8), result size (1 MiB) and script size (256 KiB) are all bounded too. A script that hits the call limit cannot hide the error by catching it. Excess runs are turned away rather than queued. Operators also set a process-wide ceiling on sandboxes and memory, and per-listener settings cannot exceed it.

Result Composition A run returns a single object: a status, the script's return value, any log lines, and a list of every tool it called with timing and outcome. If that object is too big, the gateway keeps the actual answer, trims the extras and says exactly what it trimmed. A script error gets a plain reason and a line number for the agent to resolve.

What it changes

For the support request above, the comparison is plain:

	Tools mode	Code mode
Model turns	4	1
Crossings between model and gateway	8	2
Tool responses in the model's context	4, in full	0; one small object
Tool definitions offered up front	every listed tool	one, `execute`

Those are counts from the example. The backend calls take as long as they always did, and the two lookups now run in parallel. The model turns in between are gone, and those turns are usually where the time and the tokens go. Workflows with more steps or more fan-out gain more. We have not published timings for code mode yet, so we are not quoting any here.

The catalog scales the same way. tools/list returns one tool whether the gateway serves ten tools or ten thousand, and the model reads a definition only when its script asks for one.

Trade-offs

The model has to write working code. The errors are built for a model to act on: a parse error gives a line and column, and a failed call is a value with a code. Toolkits cut the guesswork further. If it takes a model multiple attempts to generate the correct program, some benefits are lost.
A listener is one mode or the other. A code-mode listener serves only execute. Clients that expect the classic tool list need a separate listener in tools mode. Both can front the same tools.
Scripts keep no state. Every run starts fresh. Anything that must survive between runs belongs in your tools.
The TypeScript is strip-only. Annotations are removed, but enum, namespace declarations and decorators are rejected. In practice, they are typically not necessary.
It is opt-in at build time. Code mode sits behind the codemode Cargo feature, which adds the Wasmtime runtime and a sandbox component of just under 1 MiB. A build without it rejects a code-mode configuration cleanly instead of misbehaving.

Turning it on

Build with --features codemode, then pick the mode per listener:

typed_config:
  "@type": type.googleapis.com/arion.extensions.filters.http.mcp.mcp_gateway.v3.McpGateway
  server_info:
    name: support-tools
    version: "1.0.0"
  code_mode: {}

Every setting has a sensible default. Tools, upstream MCP servers, OpenAPI sources and toolkits can all be delivered and changed at runtime over xDS. It is still standard MCP, so existing clients keep working; the model just sees one tool. The sandbox itself can be swapped for a vetted custom build, which Arion checks when the listener loads.

What is next

To try it end to end, follow code mode for agents. It builds the support workflow above against three REST services. The code mode page covers when to use it, and the SDK reference lists every call, error code and limit. For a quick way to fill the catalog, pair it with OpenAPI tools: point a source at an API's spec, and every operation becomes a tool that a script can call.

Introducing the Arion control plane

2026-10-06T00:00:00+00:00

When we set out to build Arion, we made the decision to keep close compatability the Envoy configuration model. We made this choice because, whilst the Envoy configuration model has some ergonomic issues and had become a little bloated in places, it had extensive coverage. Coupled with the xDS discovery protocol, Envoy led the way in terms of dynamic (re)configuration in Cloud L7 networking. In choosing Envoy's native configuration model over xDS, we could also leverage the existing Envoy ecosystem of xDS control plane software.

Having worked with a bunch of the controllers in the Envoy xDS ecosystem over a number of years, however, we got frustrated and decided to build our own. Arion-ctl is our own control plane, still fully xDS compatible, but small, neat and purpose built with a bunch of usability features that we felt were missing from others in the ecosystem.

You can deploy it in two ways: as a standalone service that you manage with files and operator commands, or as a Kubernetes controller that reads Gateway API resources from your cluster. Each option delivers configuration to running Arion proxies.

Arion remains the data plane: it accepts connections and forwards traffic. Arion-ctl supplies listeners, routes, clusters, endpoints, and secrets as configuration dynamically over xDS. Once a proxy has its configuration, handling a request does not require any interaction with the control plane.

Choose a deployment option

The standalone service and Kubernetes controller are separate deployment options. Choose the one that fits how you manage your gateways. You do not need to install or configure the standalone service before using the Kubernetes controller.

Deployment option	How you manage configuration	Where configuration is stored
Standalone service	Apply YAML resource files with `arionctl`, or use the optional `arionctl shell`	A durable snapshot saved by the standalone service
Kubernetes controller	Create Gateway API resources through the Kubernetes API	Kubernetes cluster state

The command line and interactive shell are two interfaces to the same standalone service. The Kubernetes controller has its own setup and reads its configuration from Kubernetes resources. Both deployments use the same library internally to deliver configuration over xDS.

Standalone deployment: files and an optional shell

Choose standalone when you want to run a control-plane service without depending on Kubernetes. The service stores your configuration and supplies it to connected proxies. You can manage it entirely through YAML files and arionctl commands; the Elixir shell is optional.

A fleet groups the configuration delivered to a set of proxies. Each proxy selects its fleet with node.cluster in its bootstrap. The standalone default is arion.

Apply configuration from a file

The standalone service accepts YAML resource documents. Each identifies a fleet, resource kind, and name, followed by the resource specification. These are the standalone service's resource documents; Kubernetes Gateway API uses its own resource definitions. The proxy still has a separate bootstrap containing its identity and control-plane connection settings.

For a working example, follow the standalone walkthrough through the initial HTTP configuration. This starts backend A on port 18081, the control plane on 15051, and a proxy listener on 18080. Keep those processes running and retain the operator terminal's release environment.

Save the overview's Cluster example as backend.yaml. It describes the existing backend resource in fleet arion, pointing to backend A. From the operator terminal:

arionctl validate -f backend.yaml
arionctl apply -f backend.yaml
arionctl get --fleet arion
arionctl status
curl --fail http://127.0.0.1:18080/

The request returns hello from backend-a. Reapplying this Cluster preserves the Listener from the walkthrough. Apply replaces each supplied resource's full specification and keeps identities absent from the input. Removal uses an explicit delete command. Editing the file alone changes nothing; the service loads it when instructed.

The service validates the whole input, writes and syncs a temporary snapshot, renames it into place, then publishes the desired state to the fleet caches. Invalid input is rejected before saving. If writing the snapshot fails, the previous desired state remains active.

get shows saved configuration. status shows resource counts and connected proxies' acknowledgement state. An accepted version appears as ["ack", "<version>"]; a rejected update appears as ["nack", "<version>", "<error>"]. A successful apply means the desired state was saved and published. Proxy responses arrive separately, and a real request checks whether the intended routing works.

Optional: update the same service from the shell

If you want to construct configuration in code, attach to the running standalone service through IEx, Elixir's interactive shell. This uses the same configuration store as the file commands above. Start a second backend in another terminal, from the control-plane repository:

python3 docs/examples/backend.py --port 18082 --name backend-b

In the operator terminal, attach to the service:

arionctl shell

At the IEx prompt, construct a Cluster and apply it:

alias Arion.ControlPlane.Helpers.Cluster

cluster_b = Cluster.static("backend", "127.0.0.1", [18082], connect_timeout: 1)
{:ok, _} = Arion.Ctl.apply_resources([cluster: cluster_b], fleet: "arion")
Arion.Ctl.get("arion")
Arion.Ctl.status()

Cluster.static builds a protobuf value. apply_resources converts it into the same resource documents used by the file interface, validates them, and saves them through Arion.Ctl. The new backend survives a service restart using the same state directory.

From an ordinary terminal, request the proxy again:

curl --fail http://127.0.0.1:18080/

After the update is accepted, it returns hello from backend-b. The Listener stayed in place. The proxy and control plane kept running.

The shell also permits ordinary Elixir functions, loops, and transformations. You can compute a collection of resources, inspect it, then apply the collection together. Keep standalone mutations on Arion.Ctl; publishing directly through the library's Service API would bypass the snapshot.

To leave the attached shell, press Ctrl+G, then enter q. Back in the operator terminal, export the desired state:

(umask 077; arionctl get --fleet arion > arion-resources.json)
arionctl validate -f arion-resources.json

The JSON can be applied again with the same CLI. Exports can contain private keys from Secret resources, so keep them in protected storage. The shell walkthrough covers inspection, deletion, and restart verification.

Kubernetes deployment: Gateway API

Choose this option when you want Kubernetes resources to define your gateways and routes. Install the Kubernetes controller with the Helm chart, then manage configuration through the Kubernetes API, using tools such as kubectl or your GitOps workflow. This is an alternative to the standalone setup above; you do not apply these resources with arionctl or use its shell.

A GatewayClass selects the controller, a Gateway defines traffic entry points, and routes attach rules to them. The controller watches these Kubernetes objects, resolves references and attachment permissions, then translates the result into the xDS configuration that Arion proxies receive.

HTTPRoute covers matching, weighted backends, header changes, redirects, rewrites, timeouts, and retries. GRPCRoute adds service and method matching. Kubernetes Secrets supply certificates for HTTPS, while TCPRoute and TLSRoute describe stream forwarding and TLS handling. Services resolve to ready EndpointSlice addresses. InferencePool integration uses an endpoint picker and the pool's ready Pods. The controller reference describes the supported fields and prerequisites for each feature.

The AI inference gateway overview explains how Arion asks the endpoint picker to select a ready model server Pod for each request. The inference routing guide shows the Gateway API and InferencePool resources that configure it.

Each Gateway gets a fleet named gateway/<namespace>/<name>. Every controller replica derives and serves its own configuration from Kubernetes. With the chart's clustering enabled, an elected leader handles Kubernetes status and proxy provisioning. Replicas do not share a standalone snapshot or replicate xDS stream state.

The controller can provision a Deployment, Service, ServiceAccount, and bootstrap ConfigMap for each Gateway. ArionGatewayParameters controls details such as image, replicas, and service type. Within this Kubernetes deployment, you can instead manage the proxy workloads yourself while the controller continues to deliver their configuration.

Why Elixir fits the design

A controller repeatedly loads or watches resource definitions, resolves their relationships, transforms them into configuration a data plane understands, and delivers updates as the desired state changes. We chose Elixir because it makes both the transformations and the ongoing coordination simpler to express and maintain.

A runtime built for distributed systems

Elixir runs on the Erlang virtual machine, the BEAM. It provides lightweight, isolated processes that communicate by passing messages. Erlang's OTP libraries add supervision: processes can be organised into trees that define how their work is started and restarted after failures. Communication also extends across connected BEAM nodes, with built-in remote messaging, function calls, and process monitoring.

These capabilities make writing distributed applications much simpler. A controller needs to keep resource watches running, schedule reconciliation, track connected proxies, and coordinate work across replicas. The Erlang process model gives us a consistent way to organise those activities, backed by runtime features and libraries built for that purpose. In Arion, this also supports direct interaction with a running service through remote commands and an interactive shell.

A concise language for resource transformations

Much of a controller's work is data transformation: take resources in one representation, resolve what they mean together, and construct another representation. Elixir's concise syntax, pattern matching, collection functions, and pipelines suit this work particularly well. We can express the transformation as a sequence of small functions, each handling one part of the problem. Immutable data makes the inputs and outputs of each step easier to follow.

In other controllers we've worked with, we've seen many thousands of lines of code devoted just to translating between representations. A language can be straightforward to get started with and still require a surprising amount of code for that task. We think that is a reason to question the fit between the language and the problem. Elixir lets us express these transformations more directly, keeping the implementation smaller and easier to understand.

More room for the operator experience

A smaller, simpler implementation also gives us more room to build a richer operator experience. The functions used to construct, inspect, validate, and apply configuration are available in the live Elixir shell. An operator can explore the current state, transform a collection of resources, and apply the result without building a separate tool for each operation.

In the standalone service, the CLI and shell call the same application API. They share validation and durable storage, so using the programmable interface keeps the same persistence behaviour as applying a file. Elixir makes that kind of interactive experience a natural extension of the implementation.

For Elixir developers: embed the library

An Elixir application can also supervise Arion.ControlPlane directly and construct configuration from its own service data. This is useful when you are building a custom control plane. The application owns persistence and recovery, and publishes through Arion.ControlPlane.Service; Service.replace replaces a fleet's complete resource set. See the complete library example for startup and publication details.

State ownership and operational checks

Recovery follows the configuration owner. Standalone restores its snapshot before accepting xDS connections and requires one service instance per state directory. Kubernetes controller replicas rebuild their configuration from cluster resources and wait for their initial lists and first publication before serving discovery.

A cache replacement does not make updates across resource types atomic at every proxy. ACKs describe accepted resource versions; they do not establish backend reachability. Inspect status and proxy logs, then send traffic through the route being changed.

Fleets separate configuration but do not authenticate clients. Keep discovery and operator connections on trusted networks. The deployment guide covers placement, storage, and availability responsibilities.

To get started, choose the standalone walkthrough for files and operator commands, or the Helm guide for Kubernetes Gateway API. The control-plane overview has more examples, and the general xDS guide explains the proxy side of the connection.

Introducing Arion

2026-09-29T00:00:00+00:00

Arion is an open-source L7 gateway written in Rust. It reads Envoy configuration natively and runs as an API gateway and proxy, an LLM gateway and an MCP gateway — one data plane for services, models and tools.

This is the first post on this blog, so before we write about internals and benchmarks, we want to start with the story: where Arion came from, and why we built it.

Where Arion came from

The project behind Arion began as a research project and was later donated to the Kmesh GitHub organisation as an open-source subproject called Orion. Arion is an independent fork of Orion, led by its original authors and long-term maintainers. Arion builds on what we learned over a number of years of developing a high-throughput data plane for cloud services.

We started this work initially while employed in Huawei's European research labs, aiming to address the many challenges that Cloud service teams and product teams had running large fleets of proxies and gateway services at scale. It was clear that the tools available to them each solved only half of the problem.

We used nginx heavily, and on raw performance it is outstanding. But its configuration model comes from a world where configuration changes rarely: pushing changes across large fleets of nginx instances, continuously and safely, is brittle and error-prone.

Envoy solves exactly that problem: it is battle-tested, and its dynamic configuration over xDS lets whole fleets reconfigure continuously without dropping traffic. But Envoy is heavy. It uses a lot of CPU and memory for the traffic it moves, and its throughput sits far below nginx's.

Neither system gave us both. We wanted the performance of nginx and the operability of Envoy in one data plane.

A better Envoy, in Rust

We set out to build a better Envoy. The requirements were written down early and never really changed:

Dynamically reconfigurable. Configuration arrives over xDS and applies without pausing traffic.
Stable and predictable. Every request passes through the proxy, so it has to run reliably for long periods and fail in predictable ways.
Simple to deploy and operate. One binary, one config file or one control-plane connection.
Lightweight. Frugal with memory and CPU, because at fleet scale resource usage translates directly into cost and capacity.

Rust made those goals compatible. It gives us memory safety without a garbage collector: whole classes of memory bugs are ruled out at compile time, with no runtime cost on the request path. And speaking Envoy's configuration natively, rather than inventing our own, meant the existing ecosystem of control planes and tooling carried over.

The performance target was clear from production experience running high-throughput services: nginx. For our scenarios it has always been the fastest option available, and that was the bar we set for Arion.

From prototype to production

The codebase behind Arion grew in stages: first a prototype, then a pre-production system, then versions of it carrying real production traffic.

Arion builds on learning and experience gained from an internal production version deployed across thousands of instances powering AI cloud services. We now carry that work forward independently through Arion.

Building for stability in production shaped the design. A few early decisions — configuration updates delivered as messages rather than guarded by locks, a data path that scales by adding whole runtimes, nothing on the hot path that can block — were made for stability and operability first. The same decisions, it turns out, are what make a proxy fast. The architecture page describes how those pieces fit together.

We kept tuning, and release after release the gap to nginx narrowed, then closed. Arion now matches nginx, and beats it in a number of scenarios. Against agentgateway, Envoy and other gateways with similar feature sets, Arion delivers multiples of their throughput. The benchmarks show the numbers and exactly how we produce them, so you can rerun them yourself.

What Arion is today

A short tour, with the details left to the docs:

An L7 and L4 proxy and API gateway. Routing, load balancing, TLS, JWT authentication, rate limiting, circuit breaking, health checks — the most commonly used Envoy features, described as standard Envoy configuration. Start with What is Arion.
Envoy-compatible, strictly. If Arion accepts a configuration meant for Envoy, it behaves like Envoy. If it cannot honor one, it refuses to start rather than silently ignoring part of it. The compatibility page defines the supported subset, and there is a guide for migrating from Envoy.
A gateway for AI traffic. An LLM gateway puts one governed front door in front of model providers, and an MCP gateway serves tools to agents — including code mode, where an agent sends one program instead of many tool calls.
Programmable. WebAssembly filters with a Rust SDK, native Rust filters, and external processing over gRPC, for the cases configuration cannot express.

What's next

A lot has changed in the last couple of years. AI and agentic workloads emerged as a new and important class of traffic. It turned out that the properties we built for scale — a fast, safe, dynamically configurable data plane that is simple to extend — are exactly what that traffic needs. Most of Arion's recent development has gone into these AI features, and most of what is coming next will too. The posts that follow this one will go deeper: how we benchmark, how the runtime is structured, and what we learned building AI features on a proxy.

If you want to try Arion, the quickstart proxies your first request in five minutes. The code is on GitHub, and the community page lists where to ask questions. We look forward to seeing what you build with Arion.