---
title: "Runtime and threading"
url: "/docs/concepts/runtime-and-threading/"
kind: "concept"
summary: "One multi-threaded runtime or one runtime per core: the runtime block, CPU detection, core pinning, and what SO_REUSEPORT changes."
---

> Arion site index: [/llms.txt](/llms.txt)

# Runtime and threading

One multi-threaded runtime or one runtime per core: the runtime block, CPU detection, core pinning, and what SO_REUSEPORT changes.

Arion's threading model is a configuration choice: run the data path on **one multi-threaded Tokio runtime**, or split it into **several independent runtimes, one per core** — and this page explains both modes and the `runtime` block that selects them. Read it when planning deployment for a production scenario.

## Why it matters

The two modes trade flexibility for isolation. A single runtime with many worker threads uses Tokio's work-stealing scheduler: any thread can pick up any connection's next piece of work, which smooths out uneven load. Per-core runtimes are share-nothing: a connection is accepted by one runtime and every byte of its processing stays there, which keeps a connection's state on one core's caches and removes cross-core coordination from the data path. The per-core model wins significantly for high demand, high throughput scenarios with massive scaling needs. Choose based on your traffic shape — switching between them is a simple two-line config change.

## How it works

The `runtime` block sizes everything. `num_cpus` (default: detected) is the total core budget for the data path, `num_runtimes` (default: 1) divides it, and each proxy runtime gets `num_cpus / num_runtimes` worker threads — the division must be exact, or Arion refuses to start. A runtime that gets exactly one thread runs on Tokio's lighter current-thread scheduler. `num_service_threads` (default: 1) sizes the separate [services runtime](@/docs/concepts/architecture.md).

```yaml,name=arion.yaml,arion=fragment
runtime:
  num_cpus: 8          # default: detected, container-aware
  num_runtimes: 4      # 8 / 4 = 2 worker threads per runtime
  num_service_threads: 1
  affinity_strategy:
    type: runtimes
    map: [[0, 1], [2, 3], [4, 5], [6, 7]]
```

{{ <figure src="img/docs/runtime-threading.svg" alt="Two panels comparing one runtime with N worker threads and one socket against four runtimes pinned to cores, each with its own socket on the same port" caption="The same listener, scheduled two ways. On the right, every runtime binds its own SO_REUSEPORT socket on the same port." label="Figure 1." /> }}

**The default `num_cpus` is container-aware.** On Linux, Arion reads the cgroup CPU limit — `cpu.max` on cgroups v2, the CFS quota and period on v1 — and rounds the quota up to whole cores; without a limit it falls back to the system count. A pod limited to `cpus: "2"` therefore defaults to two data-path threads without any configuration. Arion also checks the budget against the cores its process is actually allowed to run on and refuses to start if `num_cpus` exceeds them.

**`SO_REUSEPORT` is what makes multiple runtimes work.** Every listener socket is opened with it, so each proxy runtime binds its *own* socket on the same address and port, and the kernel spreads incoming connections across them. There is no accept thread handing connections around: acceptance, TLS, filters, and upstream I/O for a connection all run on the runtime that accepted it, for the connection's whole life.

**`affinity_strategy` pins runtimes to cores.** With `type: auto`, Arion groups the available cores by NUMA node (read from `/proc/cpuinfo`) and deals runtimes across the nodes round-robin, giving each a contiguous chunk. `type: nodes` does the same with core groups you list explicitly, and `type: runtimes` maps runtime *i* to the *i*-th core list, taking as many cores as that runtime has threads. Pinning keeps a runtime's threads — and with per-core runtimes, each connection — on the same core and memory node; a failed pin logs a warning and continues unpinned.

The remaining keys — `global_queue_interval`, `event_interval`, `max_io_events_per_tick` — pass straight through to Tokio's scheduler and are best left unset until [performance tuning](@/docs/operate/performance-tuning.md) gives you a reason. All of the sizing keys can be overridden per environment with flags or `ARION_*` variables, as described in [the configuration model](@/docs/concepts/configuration-model.md).

## Key terms

- **Proxy runtime** — one Tokio runtime running the data path; `num_runtimes` of them exist.
- **Worker thread** — an OS thread inside a runtime; `num_cpus / num_runtimes` per runtime.
- **`SO_REUSEPORT`** — the socket option letting every runtime bind the same port, with the kernel balancing accepts.
- **Affinity strategy** — the optional pinning of each runtime's threads to specific cores or NUMA nodes.
- **Services runtime** — the control-side runtime, sized independently by `num_service_threads`.

## Next

- Where the runtimes sit in the whole process: [architecture](@/docs/concepts/architecture.md).
- Deployment sizing and cgroup interplay: [deploy](@/docs/operate/deploy.md) and [performance tuning](@/docs/operate/performance-tuning.md).
- Every `runtime` field and default: the [file format reference](@/reference/configuration/file-format.md).