Skip to content

Multi model routing

One model is one quota and one point of failure. When every request goes to the same model, a saturated quota stops your traffic, and an upstream incident stops it too. Distributing requests across a pool of models raises the ceiling on both.

Two policies do the distribution. model-round-robin cycles through the pool in turn. model-weighted-round-robin splits traffic in proportion to a weight you set per model, which suits a pool whose models differ in capacity, cost, or performance.

A model entry can name a provider, so a pool is not confined to one provider. Multi model routing and multi-provider routing overlap rather than divide: this page distributes across models, that page puts several providers behind one OpenAI-compatible endpoint, and one pool can do both at once.

Round robin

Use model-round-robin to cycle deterministically through a list of models. A model entry can include a provider to route that model to an additional provider. When provider is omitted, the model uses the primary provider.

operationPolicies:
  - name: model-round-robin
    version: v1
    paths:
      - path: /chat/completions
        methods: [POST]
        params:
          models:
            - model: gpt-4o
            - model: claude-sonnet-4-5-20250929
              provider: anthropic-provider
          suspendDuration: 30

The policy rewrites the model at the location defined by the provider template. It can rewrite a model in the request payload, a header, a query parameter, or a path parameter.

See Model Round Robin for its complete configuration.

Weighted round robin

Use model-weighted-round-robin to distribute requests in a deterministic weighted cycle. Each entry requires an integer weight of at least 1.

operationPolicies:
  - name: model-weighted-round-robin
    version: v1
    paths:
      - path: /chat/completions
        methods: [POST]
        params:
          models:
            - model: gpt-4o
              weight: 2
            - model: claude-sonnet-4-5-20250929
              provider: anthropic-provider
              weight: 1
          suspendDuration: 30

This example produces the repeating sequence gpt-4o, gpt-4o, claude-sonnet-4-5-20250929 while both targets are available. It provides proportional deterministic distribution, not random or performance-based load balancing.

See Model Weighted Round Robin for its complete configuration.

What happens when a model fails

The failover behavior comes with the round robin policies. When a model returns a 5xx or 429, the policy suspends it for a configurable duration and sends traffic to the rest of the pool, rather than retrying an endpoint that is already failing.

Suspension does not retry the request that failed. The caller sees the error from the model that returned it, and the next request goes to a different target. Set suspendDuration to 0 to disable suspension.

For how long the gateway waits before it treats an upstream as failed, see Timeouts and resilience.