How 5 Managed LLM Gateways Handle Failover: What Their Docs Actually Promise

Checked October 31, 2026 against each vendor's public documentation. Every claim links to its source.

Of the five managed LLM gateways compared here, two reroute a failed request with no configuration (Concentrate and OpenRouter), one reroutes across providers by default but needs a list to try other models (Vercel AI Gateway), and two do nothing until you build a policy (Portkey needs a fallback config, Cloudflare needs a dynamic route). Only Portkey and Cloudflare publish retry counts and backoff. Only Concentrate and Vercel document what happens to a stream that has already started. Nobody publishes a default timeout for managed inference.

This matters because "automatic failover" appears on every gateway's homepage and means five different things. A provider outage or a burst of 429s is handled correctly only if the behavior you assumed is the behavior the gateway has. The table below is what each vendor commits to in writing, not what its marketing page says, and it is deliberately limited to managed gateways, where you cannot read the source to find out.

Failover behavior, from the documentation

Sources: Concentrate auto-routing reference, OpenRouter provider routing and model fallbacks, Vercel model fallbacks and provider timeouts, Portkey fallbacks and automatic retries, Cloudflare dynamic routing and Universal Endpoint.

How does Concentrate handle failover?

Failover is the default, not a feature you turn on. The router sorts providers for the requested model by the metric you pick (routing.provider.sort, default performance; options include cost, latency percentiles, uptime and throughput over a configurable window, default 15 minutes), and on a non-streaming request it retries across that list on any error. A whitelist restricts the pool, a blacklist excludes providers, and fallbacks sets a preferred order tried ahead of BYOK keys. Once the primary model's providers are exhausted, routing.model.fallbacks moves to the models you listed, including "auto". Streaming is the honest caveat: retries happen only during the connection phase; once bytes are flowing the request commits, and your client should handle a broken stream. If no provider supports every requested feature, less critical features are stripped in a fixed order, but text, image and file input, function calling, structured output and streaming are never stripped. Exhaustion is explicit: 424 when all providers failed, 429 when all were rate-limited, and 422 when ZDR is on and no compliant provider fits, with no silent fallback to a non-ZDR provider. Not published: retry counts and timeouts. Source: auto-routing reference; see also fallbacks and reliability.

How does OpenRouter handle failover?

Also default-on. Requests are spread across providers by measured price, throughput, latency and reliability, allow_fallbacks is true unless you turn it off, and with an order list OpenRouter tries providers one at a time and proceeds to other providers if none are operational. The docs are candid that restricting providers with only "may significantly reduce fallback options and limit request recovery." Model-level fallback is the models array; any error moves to the next model, including context-length and moderation failures, and billing follows the model that answered, so a free model followed by a paid one can produce a paid request. Not published: which status codes trigger a retry, retry counts, timeouts, or mid-stream behavior. Sources: provider routing, model fallbacks.

How does Vercel AI Gateway handle failover?

Provider failover for the requested model is automatic: the gateway picks providers by recent uptime and latency, and order or only narrow it. Model fallback requires a models array; the gateway tries every provider for the primary, then each model in the list in turn, and the response carries a modelAttempts array with each provider's status code and response time, which is the most detailed failure record of the five. Timeouts are the asymmetry: providerTimeouts (1s to 789s) apply to BYOK credentials only, and the timer clears on the first stream chunk, so a slow start fails over and a slow middle does not. Not published: default timeout for gateway-managed credentials, retry counts. Sources: model fallbacks, provider timeouts.

How does Portkey handle failover?

The most explicit controls, all opt-in. Nothing fails over until a request uses a config with strategy.mode: fallback and an ordered targets list; then "if the primary LLM fails, Portkey automatically falls back to the next in line," on any non-2xx by default, narrowable with on_status_codes. Retries are a separate retry block: up to 5 attempts, exponential backoff doubling from 1s to 16s, default on 429, 500, 502, 503, 504 and 529, with an option to honor provider Retry-After headers and a 60-second cap on cumulative wait. Portkey is the only gateway here that publishes those numbers. Not published on the fallback page: timeouts and streaming behavior. Portkey does not sell inference, so each target is a provider account you hold. Sources: fallbacks, automatic retries.

How does Cloudflare AI Gateway handle failover?

Two mechanisms, one deprecated. The Universal Endpoint took an ordered array of provider requests and moved to the next on error or requestTimeout, with maxAttempts (up to 5), retryDelay (up to 60s) and a backoff mode per provider; the cf-aig-step header reports which entry answered. Cloudflare now marks it deprecated and points new integrations to dynamic routing, where fallback is a route you build in a visual editor from model, conditional, percentage, rate-limit and budget nodes. Dynamic routes accept only the OpenAI chat completions shape (Anthropic Messages returns 400) and are addressed as dynamic/ followed by the route name. Not published for dynamic routing: retry counts, backoff, or timeouts. Sources: dynamic routing, Universal Endpoint.

What the docs leave out: six questions to ask before production

Every row above has a "not published" cell, and those cells are where incidents live. Ask each vendor these in writing, and ask for the answer as a doc link rather than a sales reply.

1. What is the default timeout on a managed-credential request, and does a timeout count as a failure that triggers failover? Only Vercel documents a timeout, and only for BYOK. For everyone else, a hung provider may hang your request.
2. How many times does the gateway retry the same provider before moving on, and with what backoff? Portkey publishes 5 attempts and 1s to 16s. Cloudflare publishes limits for the deprecated endpoint. The rest do not say, and a gateway that retries a 429 five times with backoff adds up to a minute before your fallback ever runs.
3. What happens to a stream that fails after the first token? Concentrate and Vercel say plainly that the request commits once bytes flow. Assume the same of any gateway that does not say otherwise, and build the client-side retry.
4. When a fallback model answers, how do I know? Vercel's modelAttempts, OpenRouter's model field, Cloudflare's cf-aig-step header and Concentrate's request logs all answer this differently. Your billing and your evals depend on it.
5. Does a policy constraint fail closed or fail open? With ZDR or a provider allowlist set, does the gateway refuse the request (Concentrate returns 422) or quietly route to a provider outside the policy? The docs for most gateways do not say.
6. Does the gateway hold the provider relationship, or do I? If the gateway does not sell the inference (Portkey, and Cloudflare outside Unified Billing), every fallback target is an account, a rate limit and a contract you maintain, and a BAA you negotiate separately.

Then run the two-hour test. Point a staging key at each candidate, force a 429 and a 500 on the primary with a provider you control or a deliberately exhausted key, and record time to first byte from the fallback, which model answered, and what the logs show. Read the gateway's documentation again afterwards and note every place where the behavior and the docs disagree. That list is your real comparison.

FAQ: LLM gateway failover

Which LLM gateways fail over automatically? Concentrate and OpenRouter reroute a failed request across providers by default. Vercel AI Gateway fails over across providers for the requested model by default and needs a models array for other models. Portkey needs a fallback config and Cloudflare needs a dynamic route before any rerouting happens.

What is the difference between a retry and a fallback? A retry repeats the attempt against the same provider. A fallback changes the provider or the model. Portkey documents both separately; most gateways blur them.

Does an LLM gateway recover a broken stream? Not after the first token, on any gateway that documents the case. Concentrate retries only during the connection phase, and Vercel clears its timeout on the first chunk. Handle mid-stream failures in your client.

Which gateway publishes retry counts and backoff? Portkey (up to 5 retries, 1s to 16s exponential, 60s cap) and Cloudflare's deprecated Universal Endpoint (up to 5 attempts, delay up to 60s). Concentrate, OpenRouter and Vercel do not publish retry counts.

Can I see which provider or model actually answered? Yes on all five, but differently: Vercel returns modelAttempts metadata, OpenRouter's response model field names the model used, Cloudflare sets a cf-aig-step header, Portkey shows attempts under a Trace ID, and Concentrate records provider and request IDs in logs.

Does failover work with zero data retention? On Concentrate, ZDR removes non-ZDR providers from the fallback pool and returns 422 if none remain. On OpenRouter, zdr: true restricts routing to ZDR endpoints. Check whether other gateways fail closed before trusting a policy during an outage.

Does Concentrate sign a BAA? Yes. BAA use is $499/month, with no per-request fees and no markup on tokens; non-BAA use has no subscription fee. Failover under a BAA stays on BAA-eligible providers. See concentrate.ai/compliance.

Checked October 31, 2026. We re-read these documentation pages monthly; if a vendor's behavior has changed, tell us and we will update the row.