External processing

The ext_proc filter streams request and response phases over a bidirectional gRPC stream to an external processor — a service you write in any language, running anywhere Arion can reach. Use it when your logic should live outside the proxy: existing code, a language other than Rust, or transformations too heavy for in-process filters, at the cost of one network hop per processed message.

Enable it

Add the filter to http_filters and point grpc_service at a cluster for your processor (the cluster must speak HTTP/2):

http_filters:
  - name: envoy.filters.http.ext_proc
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
      failure_mode_allow: false
      message_timeout: 5s
      processing_mode:
        request_header_mode: SEND
        response_header_mode: SKIP
        request_body_mode: NONE
        response_body_mode: NONE
      grpc_service:
        envoy_grpc:
          cluster_name: ext_proc_server

clusters:
  - name: ext_proc_server
    type: STATIC
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address: { address: 127.0.0.1, port_value: 18080 }

grpc_service also accepts google_grpc with a target_uri, and a timeout for the stream.

Processing modes

processing_mode decides which phases reach the processor. Headers: DEFAULT (send), SEND, SKIP. Bodies: NONE (default), STREAMED, BUFFERED, BUFFERED_PARTIAL, FULL_DUPLEX_STREAMED. Trailers: DEFAULT (skip), SKIP, SEND. With no processing_mode at all, both header phases are sent and nothing else. In BUFFERED mode the body is capped at 100 MiB by default; override the cap with max_receive_message_length on envoy_grpc. Oversized bodies are rejected with 413.

An inference endpoint picker may need the whole request body before answering the request headers. Use FULL_DUPLEX_STREAMED with send_body_without_waiting_for_header_response: true to let the body reach the processor while Arion waits for that answer. Response phases can also pass through the processor, including streamed inference responses.

What a processor can return

Header mutations — set and remove headers. Omitting mutation_rules applies no mutation-rule restrictions. A configured empty rules object protects host, :authority, :scheme, :method and x-envoy- headers; use fields such as allow_all_routing, allow_envoy and disallow_system to adjust it. set_headers replaces an existing value by default; append: true requests appending.
Body mutations — replace or clear the body in body-processing modes.
An immediate response — status, body, headers, and optional grpc-status, sent straight to the client. disable_immediate_response: true turns these into errors instead.
Route cache clearing — after mutating routing-relevant headers, the request is re-matched against the route table; route_cache_action (DEFAULT, CLEAR, RETAIN) forces the behaviour.
Mode overrides — with allow_mode_override: true, the processor can switch later phases on or off per request, limited to allowed_override_modes when set, and extend the clock with override_message_timeout.

Timeouts and failure

Arion waits message_timeout (default 200ms) for each processor reply; max_message_timeout caps what override_message_timeout may request. What happens on timeout or stream error is failure_mode_allow:

false (default) — fail closed: processor message timeouts return 504; processor or stream errors return 500.
true — fail open: the request continues after the processor failure.

observability_mode: true makes the stream fire-and-forget — phases are mirrored to the processor, replies are not waited for and cannot mutate traffic.

Per-route override

Routes and virtual hosts can disable the filter or replace its settings via typed_per_filter_config, keyed by the filter's chain name:

typed_per_filter_config:
  envoy.filters.http.ext_proc:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute
    overrides:
      processing_mode:
        request_body_mode: BUFFERED
      failure_mode_allow: true

Set disabled: true instead of overrides to skip the processor on that route. Per-route grpc_service lets one route talk to a different processor. These overrides work with both inline route_config and routes delivered over RDS.

Endpoint feedback

Set metadata_options.forwarding_namespaces.untyped: [envoy.lb] on the filter to send the actual served endpoint back to the processor with the response headers. For a cluster using the OverrideHost load-balancing policy, Arion includes x-gateway-destination-endpoint-served in the envoy.lb namespace of ProcessingRequest.metadata_context. This reports the endpoint that served the response, including when endpoint selection used the fallback balancer.

This is a specific endpoint-feedback capability. Other untyped namespace names are accepted but do not enable arbitrary metadata forwarding. Typed forwarding, receiving metadata and cluster-metadata forwarding are unsupported. Per-route overrides inherit this filter-level setting; they cannot override metadata_options.

Verify

With failure_mode_allow: false and the processor not running, requests fail closed:

curl -si http://127.0.0.1:8080/ | head -1

HTTP/1.1 500 Internal Server Error

Start your processor and the same request goes through.

Limits

The processor must be gRPC; Envoy's http_service variant is not supported and refuses to start.
The GRPC body send mode and attribute forwarding (request_attributes, response_attributes) are not supported. Metadata forwarding is limited to the endpoint feedback described above.
forward_rules (allowed_headers / disallowed_headers) controls which headers are forwarded to the processor; everything else about the stream follows the Envoy ext_proc protocol.

The AI Inference Gateway overview explains how the endpoint picker and proxy work together. Follow the inference routing guide to configure inference traffic; field details are in the ext_proc reference.

Reference and guides