Auto routing and model fallback

Overview

The routing system automatically selects the best provider for every request based on the features your request requires, real-time metrics, and your optimization preferences. It supports:

Auto model selection — set model: "auto" and let the system choose a compatible model
Provider routing — specify a model and let the system pick the best provider
Multi-provider fallback — if one provider fails, the system retries on the next
Feature degradation — if no provider supports all requested features, less important features are gracefully stripped
ZDR enforcement — when Zero Data Retention is enabled, only ZDR-supporting providers are considered

Basic Usage

Auto Model Selection

Set model: "auto" to let the system select both the model and provider:

curl https://api.concentrate.ai/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "auto",
    "input": "Explain machine learning"
  }'

import requests

response = requests.post(
    "https://api.concentrate.ai/v1/responses",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "model": "auto",
        "input": "Explain machine learning"
    }
)

# Check which model was selected
print(f"Selected model: {response.json()['model']}")

const response = await fetch("https://api.concentrate.ai/v1/responses", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "auto",
    input: "Explain machine learning"
  })
});

const data = await response.json();
console.log(`Selected model: ${data.model}`);

Provider Routing (Pinned Model)

Specify a model without a provider prefix and the system routes to the best provider:

{
  "model": "gpt-4o",
  "input": "Hello",
  "routing": {
    "provider": { "sort": "cost" }
  }
}

This selects the cheapest provider for GPT-4o (e.g., OpenAI vs Azure) based on real-time data.

Pinned Provider

Use provider/model format to pin a specific provider. If the pinned provider fails, routing falls back to the other providers for the same model only when routing.provider.fallbacks or routing.provider.whitelist is set:

{
  "model": "openai/gpt-4o",
  "input": "Hello"
}

Routing Configuration

Control routing behavior with the routing parameter:

string

default:"performance"

How providers are sorted and selected:Static Metrics:

"cost" — Sort by provider pricing (cheapest first)
"performance" — Sort by quality (best first)

Live Metrics:

"avg_latency" — Average response time
"latency" — Minimum observed response time
"p50_latency", "p90_latency", "p99_latency" — Percentile latencies
"avg_e2e_latency", "e2e_latency" — Average / minimum end-to-end latency including overhead
Any percentile from p0 to p100: "p75_latency", "p95_e2e_latency", etc.
"uptime" — Provider availability
"throughput" — Requests per second
"total_requests" — Total request volume
"input_tokens", "output_tokens", "total_tokens" — Average token counts

string

default:"performance"

Accepts the same metric values as routing.provider.sort, for sorting the model pool that model: "auto" picks from.

routing.model.sort previously ranked models for model: "auto", but model-based optimization is not currently applied. Auto routing still selects a compatible model from a curated pool; provider routing, optimization metrics, and fallbacks continue to work normally.

string

default:"15 minutes"

Time window for live metric calculation. Only applies when using live metrics (latency, uptime, etc.). Static metrics (cost, performance) ignore this.Format: "number unit" or "number<shorthand>"

"15 minutes" or "15m" (default, minimum)
"1 hour" or "1h"
"24 hours" or "24h"
"7 days" or "7d"

Valid units: minutes/m, hours/h, days/d, weeks/w, years/y

array

Fallback models tried after the primary model’s providers are exhausted. Accepts model slugs, provider/model format, and "auto".

{
  "model": "gpt-5.6-terra",
  "routing": {
    "model": { "fallbacks": ["claude-sonnet-5", "auto"] }
  }
}

The system tries gpt-5.6-terra first (across all suitable providers), then claude-sonnet-5, then auto-selects from remaining models.

array

Whitelist of providers to consider. When set, only these providers are used for routing. Omit to allow all providers.

{
  "model": "gpt-4o",
  "routing": {
    "provider": { "whitelist": ["openai", "azure"] }
  }
}

array

Blacklist of providers. These providers are never used for routing, including when auto-selecting a model.

{
  "model": "gpt-4o",
  "routing": {
    "provider": { "blacklist": ["azure"] }
  }
}

array

Ordered list of preferred providers. These are tried first, in the order given, ahead of providers you hold a BYOK key for. The remaining providers follow in routing.provider.sort order.

{
  "model": "gpt-4o",
  "routing": {
    "provider": { "fallbacks": ["azure", "openai"] }
  }
}

With a pinned provider/model, setting fallbacks or whitelist lets routing fall back to the other providers for that model if the pinned one fails.

How It Works

1. Feature Detection

When your request arrives, the router scans it and builds a set of required features based on what you’re using:

You send	Required feature
`stream: true`	`stream`
`tools` with functions	`tools.function_calling`
`tools` with web search	`tools.web_search`
`tool_choice: "required"`	`tool_choice.required`
`text.format.type: "json_schema"`	`text.format.json_schema`
`reasoning.effort: "high"`	`reasoning.effort.high`
`temperature`	`temperature`

These features are matched against the capabilities of every model/provider combination.

2. Provider Selection

Providers matching all required features, your whitelist, and your blacklist are sorted by your chosen metric. Ahead of that metric order come, in precedence:

Preferred providers — routing.provider.fallbacks, in the order given
BYOK providers — providers you hold a BYOK key for

For live metrics (latency, uptime, etc.), the system also factors in:

Prompt cache affinity — providers where you have active cached tokens are prioritized
Feature uptime — providers whose success rate for any required feature drops below 90% are excluded

A requested service_tier is a required feature: routing only considers providers that support it, and it is never stripped. See service tiers.

3. Fallback & Retry

If a provider fails, the system automatically tries the next provider in the sorted list:

Request -> Provider A (fails) -> Provider B (fails) -> Provider C (success)

This happens transparently. For non-streaming requests, any error triggers a retry. For streaming requests, retries only happen during the connection phase. Once streaming begins, the request is committed.

4. Feature Degradation

If no provider supports all requested features, the system gracefully strips less important features to find a match. Features are stripped in this priority (least important first):

Cache identity and request attribution (prompt_cache_key, prompt_cache_retention, prompt_cache_options, prompt_cache_breakpoint, context_management, user, safety_identifier)
Output verbosity control (text.verbosity)
Sampling parameters (top_p, temperature)
Parallel tool calls (parallel_tool_calls)
Response metadata includes (include.*, top_logprobs)
Web search (tools.web_search)
Reasoning effort (reasoning.effort.*)
Tool choice controls (tool_choice.*)

Stripping happens before a provider is chosen, and again whenever the providers at the current level are exhausted, so a request can still fall back to a less capable provider after a failure.

Everything else is required and is never stripped: text and image and file input, function calling, custom tools, structured output, and streaming, and the service tier. A request that names one of these is only ever routed to a provider that supports it, and fails if none does, rather than being silently served without it.

Examples

Cost-Optimized with Fallbacks

For high-volume workloads where you want the cheapest option with resilience. Pin a model so provider.sort controls provider cost, not model choice:

{
  "model": "gpt-4o-mini",
  "input": "Summarize this text",
  "routing": {
    "provider": { "sort": "cost" },
    "model": { "fallbacks": ["gemini-2.5-flash"] }
  }
}

Performance-Optimized

For complex reasoning or code generation:

{
  "model": "auto",
  "input": "Design a scalable microservices architecture",
  "routing": {
    "provider": { "sort": "performance", "interval": "1 hour" }
  }
}

With model: "auto", provider.sort only ranks providers for whichever model is chosen from the curated pool. It doesn’t affect which model is selected.

Latency-Optimized

For real-time chat or interactive applications. Pin a model so provider.sort controls provider latency, not model choice:

{
  "model": "gpt-4o-mini",
  "input": "Quick translation: Hello -> Spanish",
  "routing": {
    "provider": { "sort": "p50_latency", "interval": "15 minutes" }
  }
}

Provider-Restricted

Limit routing to specific providers (e.g., for compliance):

{
  "model": "gpt-4o",
  "input": "Process this data",
  "routing": {
    "provider": { "sort": "performance", "whitelist": ["openai", "azure"] }
  }
}

Response Information

The response includes which provider and model were selected:

{
  "id": "resp_xyz789",
  "created_at": 1702934400,
  "status": "completed",
  "model": "anthropic/claude-haiku-4-5",
  "output": [...],
  "usage": {...}
}

Log the model field in responses to understand routing decisions over time.

Error Handling

When all providers are exhausted, the API returns the last provider’s error. Common scenarios:

Status	Meaning
`424`	All providers failed (provider errors)
`429`	All providers rate-limited
`422`	ZDR enabled but no ZDR-supporting providers available for the requested features

If Zero Data Retention is enabled on your API key but no providers support ZDR for the required model/features, you will receive a 422 Unprocessable Entity error rather than falling back to a non-ZDR provider.

To improve reliability:

Add fallback models via routing.model.fallbacks
Use broader provider pools (don’t restrict routing.provider.fallbacks unnecessarily)
Use routing.provider.sort: "performance" (default) for the most stable behavior, since it uses static ranking and doesn’t depend on live metrics availability

Best Practices

Match provider.sort to use case

These metrics rank providers for a given model. They don’t select the model itself:

"cost": Content generation, summarization, simple Q&A
"performance" (default): Complex reasoning, code generation, analysis
"p50_latency": Real-time chat, interactive applications
"uptime": Mission-critical production workloads

Use appropriate intervals

15 minutes (default): Most reactive to provider issues
1 hour: Good balance of stability and responsiveness
24 hours: Stable, long-term patterns
7 days+: Historical trends, less reactive to spikes

Configure fallback models

Add routing.model.fallbacks for critical workloads. If the primary model’s providers all fail, the system automatically tries fallbacks:

{
  "model": "gpt-5.6-terra",
  "routing": {
    "model": { "fallbacks": ["claude-sonnet-5", "gemini-3.6-flash", "auto"] }
  }
}

Placing "auto" last gives the system maximum flexibility as a final fallback.

Monitor selected providers

Track which providers are being selected over time:

selected_providers = {}

for request in requests:
    response = make_request(request)
    model = response["model"]  # e.g. "openai/gpt-4o"
    selected_providers[model] = selected_providers.get(model, 0) + 1

print(selected_providers)
# {'anthropic/claude-haiku-4-5': 45, 'openai/gpt-4o-mini': 32, ...}

Combine with max_output_tokens

Set token limits to control costs even with auto routing:

{
  "model": "auto",
  "input": "Explain quantum physics",
  "routing": { "provider": { "sort": "cost" } },
  "max_output_tokens": 500
}

Create Response

Main endpoint documentation

Supported Models

View all available models

Error Handling

Handle routing failures

Request Parameters

Full parameter reference

Get started

Features

Monitoring

Auto routing and model fallback

Overview

Basic Usage

Auto Model Selection

Provider Routing (Pinned Model)

Pinned Provider

Routing Configuration

How It Works

1. Feature Detection

2. Provider Selection

3. Fallback & Retry

4. Feature Degradation

Examples

Cost-Optimized with Fallbacks

Performance-Optimized

Latency-Optimized

Provider-Restricted

Response Information

Error Handling

Best Practices

Create Response

Supported Models

Error Handling

Request Parameters

​Overview

​Basic Usage

​Auto Model Selection

​Provider Routing (Pinned Model)

​Pinned Provider

​Routing Configuration

​How It Works

​1. Feature Detection

​2. Provider Selection

​3. Fallback & Retry

​4. Feature Degradation

​Examples

​Cost-Optimized with Fallbacks

​Performance-Optimized

​Latency-Optimized

​Provider-Restricted

​Response Information

​Error Handling

​Best Practices

​Related Documentation

Create Response

Supported Models

Error Handling

Request Parameters

Overview

Basic Usage

Auto Model Selection

Provider Routing (Pinned Model)

Pinned Provider

Routing Configuration

How It Works

1. Feature Detection

2. Provider Selection

3. Fallback & Retry

4. Feature Degradation

Examples

Cost-Optimized with Fallbacks

Performance-Optimized

Latency-Optimized

Provider-Restricted

Response Information

Error Handling

Best Practices

Related Documentation