Use Kubernetes Gateway API and the Inference Extension to route to self-hosted model servers. The Arion controller translates an InferencePool into a pool of ready Pods; an external endpoint picker chooses the destination for each request through ext_proc.
AI Inference Gateway
Route inference requests on Kubernetes with Gateway API, InferencePool, and an endpoint picker that selects from ready model server Pods.
1 page