Configure AI Fallbacks and Load Balancing in Portrix

Portrix lets you define fallback models and distribute load across providers so your application stays available even when individual providers hit rate limits or experience outages. Instead of your users seeing a hard error, Portrix silently retries the request against the next model in your chain — all within the same API call.

Why fallbacks matter

AI provider reliability is not guaranteed. Providers can:

Go down — planned or unplanned outages affect even major providers
Rate limit your account — high-traffic periods can trigger 429 responses
Introduce latency spikes — cold model starts or traffic surges slow responses significantly

Without fallbacks, any of these events surfaces directly as an error in your application. With fallbacks configured in Portrix, the gateway retries transparently before returning an error to your code.

Setting up fallbacks

Using the `x-portrix-fallback` header

Pass a comma-separated list of fallback model IDs in the x-portrix-fallback header. Portrix tries the primary model first, then each fallback in the order listed.

curl https://api.portrix.ai/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-portrix-fallback: anthropic/claude-3-5-sonnet, mistral/mistral-large" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [
      {"role": "user", "content": "Summarise the key principles of REST APIs."}
    ]
  }'

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.portrix.ai/v1",
)

response = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[
        {"role": "user", "content": "Summarise the key principles of REST APIs."}
    ],
    extra_headers={
        "x-portrix-fallback": "anthropic/claude-3-5-sonnet, mistral/mistral-large",
    },
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.portrix.ai/v1",
});

const response = await client.chat.completions.create(
  {
    model: "openai/gpt-4o",
    messages: [
      { role: "user", content: "Summarise the key principles of REST APIs." },
    ],
  },
  {
    headers: {
      "x-portrix-fallback": "anthropic/claude-3-5-sonnet, mistral/mistral-large",
    },
  }
);

console.log(response.choices[0].message.content);

In the examples above, Portrix will:

Attempt openai/gpt-4o first
If that fails, attempt anthropic/claude-3-5-sonnet
If that also fails, attempt mistral/mistral-large
If all three fail, return an error to the caller

Fallback conditions

Portrix triggers a fallback when the upstream provider returns any of the following:

Condition	HTTP Status	Description
Provider error	`5xx`	Internal server errors, model unavailability
Rate limited	`429`	Account or IP-level rate limit exceeded
Request timeout	—	Provider did not respond within the configured timeout

Fallbacks are not triggered for client errors (4xx other than 429), such as invalid request shapes or authentication failures. Those are returned immediately.

Load balancing

Using `x-portrix-route: balanced`

When you want to spread traffic evenly across multiple models — distributing cost or staying within per-provider rate limits — set the routing strategy to balanced alongside a fallback list:

response = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[
        {"role": "user", "content": "Generate a product description for a wireless keyboard."}
    ],
    extra_headers={
        "x-portrix-fallback": "anthropic/claude-3-5-sonnet, google/gemini-2.0-flash",
        "x-portrix-route": "balanced",
    },
)

With balanced routing, Portrix distributes requests round-robin across openai/gpt-4o, anthropic/claude-3-5-sonnet, and google/gemini-2.0-flash. If any node is unhealthy, it is temporarily removed from the rotation and the remaining nodes absorb its share. Available routing strategies:

Strategy	Behaviour
`balanced`	Round-robin across all models in the chain
`cheapest`	Always routes to the lowest-cost available model
`fastest`	Routes to the model with lowest current latency

Dashboard configuration

Sending headers on every request works well for dynamic configurations, but for stable production setups you can define named routes in the Portrix dashboard:

Open the Routes section

Navigate to Gateway → Routes in the Portrix dashboard.

Create a new route

Click New Route, give it a name (e.g. production-chat), and add your primary model plus ordered fallbacks.

Set the routing strategy

Choose balanced, cheapest, or fastest from the dropdown.

Use the route name in requests

Reference the route by name instead of a model ID:

response = client.chat.completions.create(
    model="route/production-chat",
    messages=[{"role": "user", "content": "Hello!"}],
)

Named routes are updated centrally — changing a fallback chain in the dashboard immediately affects all requests using that route, with no code changes required.

Testing fallbacks

To verify your fallback chain is working, temporarily swap the primary model for an invalid model ID. Portrix will fail on the primary and cascade to your first fallback:

response = client.chat.completions.create(
    model="openai/this-model-does-not-exist",   # forces immediate fallback
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={
        "x-portrix-fallback": "anthropic/claude-3-5-sonnet",
    },
)

# Response should come from anthropic/claude-3-5-sonnet
print(response.model)

The model field in the response reflects whichever model actually served the request, making it easy to confirm which fallback was used.

Fallback models may have different capabilities, context windows, or output styles. A prompt carefully tuned for openai/gpt-4o may produce different results on mistral/mistral-large. Test every model in your fallback chain with your actual prompts before enabling fallbacks in production.

Get Started

Core Concepts

Guides

Integrations

Configuration

Help

Configure AI Fallbacks and Load Balancing in Portrix

Why fallbacks matter

Setting up fallbacks

Using the `x-portrix-fallback` header

Fallback conditions

Load balancing

Using `x-portrix-route: balanced`

Dashboard configuration

Testing fallbacks

​Why fallbacks matter

​Setting up fallbacks

​Using the x-portrix-fallback header

​Fallback conditions

​Load balancing

​Using x-portrix-route: balanced

​Dashboard configuration

​Testing fallbacks

Why fallbacks matter

Setting up fallbacks

Using the `x-portrix-fallback` header

Fallback conditions

Load balancing

Using `x-portrix-route: balanced`

Dashboard configuration

Testing fallbacks