Arion routes inference requests to self-hosted model servers using the Kubernetes Gateway API Inference Extension. An HTTPRoute sends traffic to an InferencePool, whose endpoint picker (EPP) selects a model server Pod for each request. The Arion controller tracks the pool's ready Pods and configures the proxy to reach the selected endpoint.
The configured EPP owns the scheduling decisions. Depending on its implementation, it can use signals such as queue depth, loaded LoRA adapters, or KV-cache locality. Arion exchanges request and response information with the EPP and forwards traffic to the selected pool endpoint. For one endpoint over hosted model providers, see the LLM gateway.
The Kubernetes resources
| Resource | Role |
|---|---|
GatewayClass | Selects the Arion controller; the controller's Helm chart creates it. |
Gateway | Defines the listeners that accept inference traffic. |
HTTPRoute | Matches requests and references an InferencePool as its backend. |
InferencePool | Selects model server Pods by label, declares their target ports, and references the EPP Service. |
The controller watches these resources and Pod readiness, then sends listeners, routes, clusters, and endpoints to Arion over xDS. The pool becomes an EDS cluster: its endpoint list contains matching, ready Pods on the pool's target ports and updates as those Pods change. The EPP and model servers are deployed separately from the Arion controller.
How a request is handled
- Match the inference route. The request reaches a Gateway listener and matches an HTTPRoute referencing the pool. The controller enables EPP processing for inference routes; ordinary Service routes on the same listener bypass it.
- Send the request to EPP for processing. Arion uses ext_proc over bidirectional gRPC. Headers, bodies, and trailers participate in the exchange. Request bodies can be sent before EPP answers the request-header message, allowing a scheduler to read the body before choosing an endpoint. Both request and response bodies use
FULL_DUPLEX_STREAMEDprocessing. - Select a pool endpoint. EPP sets
x-gateway-destination-endpointto the selectedip:port. The pool'sOverrideHostload-balancing policy checks the selection against its current healthy endpoints. A comma-separated list supplies candidates in failover order. - Forward the request and report the result. Arion connects to the selected model server and streams the response through response processing. The response-header message to EPP includes the endpoint actually used as
x-gateway-destination-endpoint-servedinenvoy.lbmetadata. This can differ from the original selection after fallback or failover.
Failure and fallback
endpointPickerRef.failureMode controls processor failures. The default, FailClose, fails the request if EPP processing fails. FailOpen allows it to continue; when there is no usable destination header, Arion selects a ready pool endpoint using round robin.
Endpoint selection also has its own fallback: a missing header, or a header naming no healthy pool endpoint, uses round robin under either failure mode. FailClose does not reject every unusable endpoint selection. Without a usable pool endpoint, the request fails.
Build it
The inference routing guide configures a Gateway, InferencePool, and HTTPRoute, then checks their status and sends a model request. It also explains the EPP connection, timeout, and destination-header handling.
The same underlying proxy configuration works in standalone mode. Apply the equivalent ext_proc, endpoint-pool, and load-balancing configuration directly to Arion to connect it to an EPP-style inference scheduler; Kubernetes resources are how the controller generates that configuration.