--- title: "Inference routing" url: "/guides/inference/inference-routing/" kind: "guide" summary: "Route Kubernetes inference traffic through an InferencePool, with an external endpoint picker and fallback to ready model Pods." --- > Arion site index: [/llms.txt](/llms.txt) # Inference routing Route Kubernetes inference traffic through an InferencePool, with an external endpoint picker and fallback to ready model Pods. Connect an Arion Gateway to a pool of model servers using the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/). An endpoint picker (EPP) chooses a pool member for each request. Its scheduling policy can use signals such as queue depth, loaded adapters or cache locality, depending on the EPP you deploy. ## Outcome - A Kubernetes `Gateway` with an HTTP listener on port 8080. - An `HTTPRoute` that sends `/v1` requests to an `InferencePool`. - EPP selection among ready model Pods, with round-robin fallback within the pool when no usable selection is available. ## Prerequisites - The Arion controller Helm chart installed with compatible controller and proxy images, and its default `GatewayClass` named `arion`. Follow the [Kubernetes deployment overview](@/docs/operate/deploy.md#run-on-kubernetes). - Gateway API and Inference Extension CRDs installed, including `inference.networking.k8s.io/v1` `InferencePool`. - Ready model Pods in the `default` namespace, labelled `app: llm-pool`, serving an OpenAI-compatible HTTP API on port 8000 over HTTP/1.1. - A compatible EPP already configured for `default/llm-pool`, exposed through the `default/llm-epp` Service on port 9002 with gRPC over TLS. It must support the full-duplex external processing flow described below. - `kubectl` access to the cluster and `curl` on your workstation. Adjust the namespace, Pod labels, target port and EPP Service reference to match your deployment. The example uses controller-managed Gateway provisioning. ## The manifest Download [k8s.yaml](/recipes/inference-routing/k8s.yaml). It reuses the installed GatewayClass and creates three resources: ````yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: inference-gateway namespace: default spec: gatewayClassName: arion listeners: - name: http protocol: HTTP port: 8080 allowedRoutes: namespaces: from: Same --- apiVersion: inference.networking.k8s.io/v1 kind: InferencePool metadata: name: llm-pool namespace: default spec: selector: matchLabels: app: llm-pool targetPorts: - number: 8000 appProtocol: http endpointPickerRef: name: llm-epp port: number: 9002 failureMode: FailClose --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: inference namespace: default spec: parentRefs: - name: inference-gateway sectionName: http rules: - matches: - path: type: PathPrefix value: /v1 backendRefs: - group: inference.networking.k8s.io kind: InferencePool name: llm-pool ```` The `Gateway` supplies the listener. The `HTTPRoute` attaches to that listener and references `llm-pool` as an `InferencePool` backend. The model server ports belong in the pool's `targetPorts`, so the route has no backend `port`. The pool fields describe membership and scheduling: | Field | Purpose in this example | | --- | --- | | `selector.matchLabels` | Selects Pods with `app: llm-pool` in the pool's namespace. | | `targetPorts` | Exposes each selected Pod's port 8000 as a model endpoint. | | `appProtocol` | Uses HTTP/1.1 for connections to model servers; `http` is also the default. | | `endpointPickerRef` | Names the EPP Service and its Service port, 9002. | | `endpointPickerRef.failureMode` | Uses `FailClose`, the default for processor failures. | See the [InferencePool API reference](https://gateway-api-inference-extension.sigs.k8s.io/reference/spec/) for the field definitions. ## How a request reaches a model The controller watches the pool's selected Pods and publishes ready, non-terminating Pod addresses and target ports through xDS Endpoint Discovery Service (EDS). It also configures the listener, route and EPP connection. The EPP chooses among these pool members; the controller keeps the proxy's membership current as Pods change. For a matching request, the proxy opens a bidirectional `ext_proc` gRPC stream to the pool's EPP. It sends request headers and streams request bodies using `FULL_DUPLEX_STREAMED`. Body data can reach EPP before it answers the request-header message, allowing the picker to read fields such as `model` before choosing a destination. EPP returns a header mutation setting `x-gateway-destination-endpoint` to a selected `ip:port`. Arion's `OverrideHost` load balancer uses that address only if it matches a healthy endpoint in the pool. If the header is missing, malformed or names no usable pool member, the pool's round-robin balancer selects an endpoint. An empty pool still cannot serve the request. Response headers, bodies and trailers also participate in external processing, with `FULL_DUPLEX_STREAMED` for response bodies. Arion forwards the actual served endpoint to EPP in the untyped `envoy.lb` metadata namespace, so response feedback identifies the endpoint that served the request, including after fallback. The controller enables EPP processing on inference routes and disables it for ordinary Service routes. ## Apply and verify Save the downloaded manifest as `k8s.yaml`, then apply it: ```bash kubectl get gatewayclass arion kubectl -n default get pods -l app=llm-pool -o wide kubectl -n default get service llm-epp kubectl apply -f k8s.yaml kubectl -n default wait --for=condition=Programmed gateway/inference-gateway --timeout=120s ``` Inspect the Gateway, route and pool status: ```bash kubectl -n default get gateway inference-gateway -o yaml kubectl -n default get httproute inference -o yaml kubectl -n default get inferencepool llm-pool -o yaml ``` Check `Accepted=True` and `Programmed=True` on the Gateway, and `Accepted=True` and `ResolvedRefs=True` in the route and pool's `status.parents` entries for this Gateway. Each condition's `observedGeneration` should match its resource's `metadata.generation`. These conditions confirm configuration and provisioning; a model request verifies the traffic path. Discover the provisioned gateway Service by its Gateway label and forward its listener to your workstation: ```bash GATEWAY_SERVICE=$(kubectl -n default get service \ -l gateway.networking.k8s.io/gateway-name=inference-gateway \ -o jsonpath='{.items[0].metadata.name}') kubectl -n default port-forward "service/$GATEWAY_SERVICE" 8080:8080 ``` Keep that command running. In another terminal, replace `your-model` with the model ID served by your Pods and send a request: ```bash curl --fail-with-body http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"your-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":32}' ``` You should receive a completion from the model server. Check your EPP's logs or telemetry to verify its selection and the served-endpoint feedback. If the request fails, inspect the resource conditions, model Pod readiness, EPP connectivity and gateway proxy logs: ```bash kubectl -n default logs \ -l gateway.networking.k8s.io/gateway-name=inference-gateway \ -c arion --tail=100 ``` ## Operational details **Processor failures and endpoint fallback are separate.** `FailClose` stops a request when EPP processing fails; processor errors return 500 and processor message timeouts return 504. Set `endpointPickerRef.failureMode: FailOpen` to permit continuation after a processor failure. If no usable destination header remains, the pool's round-robin balancer chooses the endpoint. Missing or unusable destination selections also use that fallback after successful processing, even with `FailClose`. **The current processor message timeout is 1,000 seconds.** This is the controller's generated `message_timeout`, not an overall request deadline. The pool manifest does not expose a setting to change it. Account for this when setting client deadlines and assessing how long requests may wait for a stalled EPP. **EPP connections use HTTP/2 over TLS without certificate verification.** The current controller accepts the EPP certificate without validating its chain or hostname. Keep this behavior in mind when defining access to the EPP Service and the trust boundary around it. **The destination header needs deliberate handling.** Arion does not automatically strip a client-supplied `x-gateway-destination-endpoint`. EPP header mutations overwrite existing values by default, so configure EPP to set its choice explicitly. Pool membership prevents selecting arbitrary addresses outside the pool, but does not stop a retained client header from selecting a valid member. Enforce header handling at a trusted entry point if clients must not influence that choice, including when using `FailOpen`. ## Next - [AI inference gateway overview](@/docs/ai-inference-gateway/overview.md) - [External processing](@/docs/extend/ext-proc.md) - [Control plane](@/docs/operate/arion-control-plane.md)