Inference routing

Connect an Arion Gateway to a pool of model servers using the Gateway API Inference Extension. An endpoint picker (EPP) chooses a pool member for each request. Its scheduling policy can use signals such as queue depth, loaded adapters or cache locality, depending on the EPP you deploy.

Outcome

A Kubernetes Gateway with an HTTP listener on port 8080.
An HTTPRoute that sends /v1 requests to an InferencePool.
EPP selection among ready model Pods, with round-robin fallback within the pool when no usable selection is available.

Prerequisites

The Arion controller Helm chart installed with compatible controller and proxy images, and its default GatewayClass named arion. Follow the Kubernetes deployment overview.
Gateway API and Inference Extension CRDs installed, including inference.networking.k8s.io/v1 InferencePool.
Ready model Pods in the default namespace, labelled app: llm-pool, serving an OpenAI-compatible HTTP API on port 8000 over HTTP/1.1.
A compatible EPP already configured for default/llm-pool, exposed through the default/llm-epp Service on port 9002 with gRPC over TLS. It must support the full-duplex external processing flow described below.
kubectl access to the cluster and curl on your workstation.

Adjust the namespace, Pod labels, target port and EPP Service reference to match your deployment. The example uses controller-managed Gateway provisioning.

The manifest

Download k8s.yaml. It reuses the installed GatewayClass and creates three resources:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
  namespace: default
spec:
  gatewayClassName: arion
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
      allowedRoutes:
        namespaces:
          from: Same
---
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: llm-pool
  namespace: default
spec:
  selector:
    matchLabels:
      app: llm-pool
  targetPorts:
    - number: 8000
  appProtocol: http
  endpointPickerRef:
    name: llm-epp
    port:
      number: 9002
    failureMode: FailClose
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inference
  namespace: default
spec:
  parentRefs:
    - name: inference-gateway
      sectionName: http
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: llm-pool

The Gateway supplies the listener. The HTTPRoute attaches to that listener and references llm-pool as an InferencePool backend. The model server ports belong in the pool's targetPorts, so the route has no backend port.

The pool fields describe membership and scheduling:

Field	Purpose in this example
`selector.matchLabels`	Selects Pods with `app: llm-pool` in the pool's namespace.
`targetPorts`	Exposes each selected Pod's port 8000 as a model endpoint.
`appProtocol`	Uses HTTP/1.1 for connections to model servers; `http` is also the default.
`endpointPickerRef`	Names the EPP Service and its Service port, 9002.
`endpointPickerRef.failureMode`	Uses `FailClose`, the default for processor failures.

See the InferencePool API reference for the field definitions.

How a request reaches a model

The controller watches the pool's selected Pods and publishes ready, non-terminating Pod addresses and target ports through xDS Endpoint Discovery Service (EDS). It also configures the listener, route and EPP connection. The EPP chooses among these pool members; the controller keeps the proxy's membership current as Pods change.

For a matching request, the proxy opens a bidirectional ext_proc gRPC stream to the pool's EPP. It sends request headers and streams request bodies using FULL_DUPLEX_STREAMED. Body data can reach EPP before it answers the request-header message, allowing the picker to read fields such as model before choosing a destination.

EPP returns a header mutation setting x-gateway-destination-endpoint to a selected ip:port. Arion's OverrideHost load balancer uses that address only if it matches a healthy endpoint in the pool. If the header is missing, malformed or names no usable pool member, the pool's round-robin balancer selects an endpoint. An empty pool still cannot serve the request.

Response headers, bodies and trailers also participate in external processing, with FULL_DUPLEX_STREAMED for response bodies. Arion forwards the actual served endpoint to EPP in the untyped envoy.lb metadata namespace, so response feedback identifies the endpoint that served the request, including after fallback. The controller enables EPP processing on inference routes and disables it for ordinary Service routes.

Apply and verify

Save the downloaded manifest as k8s.yaml, then apply it:

kubectl get gatewayclass arion
kubectl -n default get pods -l app=llm-pool -o wide
kubectl -n default get service llm-epp
kubectl apply -f k8s.yaml
kubectl -n default wait --for=condition=Programmed gateway/inference-gateway --timeout=120s

Inspect the Gateway, route and pool status:

kubectl -n default get gateway inference-gateway -o yaml
kubectl -n default get httproute inference -o yaml
kubectl -n default get inferencepool llm-pool -o yaml

Check Accepted=True and Programmed=True on the Gateway, and Accepted=True and ResolvedRefs=True in the route and pool's status.parents entries for this Gateway. Each condition's observedGeneration should match its resource's metadata.generation. These conditions confirm configuration and provisioning; a model request verifies the traffic path.

Discover the provisioned gateway Service by its Gateway label and forward its listener to your workstation:

GATEWAY_SERVICE=$(kubectl -n default get service \
  -l gateway.networking.k8s.io/gateway-name=inference-gateway \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n default port-forward "service/$GATEWAY_SERVICE" 8080:8080

Keep that command running. In another terminal, replace your-model with the model ID served by your Pods and send a request:

curl --fail-with-body http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"your-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":32}'

You should receive a completion from the model server. Check your EPP's logs or telemetry to verify its selection and the served-endpoint feedback. If the request fails, inspect the resource conditions, model Pod readiness, EPP connectivity and gateway proxy logs:

kubectl -n default logs \
  -l gateway.networking.k8s.io/gateway-name=inference-gateway \
  -c arion --tail=100

Operational details

Processor failures and endpoint fallback are separate. FailClose stops a request when EPP processing fails; processor errors return 500 and processor message timeouts return 504. Set endpointPickerRef.failureMode: FailOpen to permit continuation after a processor failure. If no usable destination header remains, the pool's round-robin balancer chooses the endpoint. Missing or unusable destination selections also use that fallback after successful processing, even with FailClose.

The current processor message timeout is 1,000 seconds. This is the controller's generated message_timeout, not an overall request deadline. The pool manifest does not expose a setting to change it. Account for this when setting client deadlines and assessing how long requests may wait for a stalled EPP.

EPP connections use HTTP/2 over TLS without certificate verification. The current controller accepts the EPP certificate without validating its chain or hostname. Keep this behavior in mind when defining access to the EPP Service and the trust boundary around it.

The destination header needs deliberate handling. Arion does not automatically strip a client-supplied x-gateway-destination-endpoint. EPP header mutations overwrite existing values by default, so configure EPP to set its choice explicitly. Pool membership prevents selecting arbitrary addresses outside the pool, but does not stop a retained client header from selecting a valid member. Enforce header handling at a trusted entry point if clients must not influence that choice, including when using FailOpen.