BlogRoute simple queries to cheaper models and escalate only when confidence drops.
Concentrate.ai
LLM Gateway
Change models, providers, add fallbacks, and more by running all AI usage through one platform.
Most production systems send every request to the same model, no matter how hard the request actually is. A survey published in Transactions on Machine Learning Research names this directly as the core cost-performance dilemma of running large language models in production: simple queries waste money on a powerful model that didn't need to be involved, and complex queries can still fail even when the system bothered to use the expensive model. Static deployment, one model handling the full range of traffic, cannot fix either problem at once. The price gap between a frontier model and a smaller one runs into the orders of magnitude, so even a small share of traffic shifted from the costly tier to the cheap one saves a large amount of money once volume is high enough. Spend climbs even while the price per token stays flat, because the number of tokens per request is what's actually growing.
Confidence-threshold escalation works like this: the cheap model answers first, it produces a confidence signal alongside that answer, and the system sends the request on to a stronger model only when that signal drops below a set threshold. Without that confidence gate, a routing system is just guessing how hard a query is before anyone has tried to answer it.
Three routing strategies sit along a spectrum of complexity, and it helps to place this one against the other two. Rule-based routing uses plain conditional logic, things like query length or keyword matches, and it's fast and easy to audit, but it breaks down at the edges where a rule doesn't generalize. Classifier-based routing adds a lightweight model that predicts, before any generation happens, which LLM will produce the best answer; it catches cases rules miss but adds latency up front. Cascade (or waterfall) routing is the approach this piece covers: the cheap model runs first, a scorer evaluates what it produced, and escalation only fires when the resulting confidence score falls below a threshold T.
The survey draws a formal line between model routing, a single decision made before generation starts, and model cascading, a sequential process where escalation depends on a quality estimate made after the cheap model has already answered. The structural edge cascading has over upfront routing is that nobody has to predict how hard a query will be before seeing an answer. The two-stage cascaded framework built by ADAPT Centre and Huawei Research shows what this looks like in practice: Stage 1 uses semantic clustering and cost-aware routing to send each query to the cheapest model likely to handle it well, and Stage 2 runs a quality-estimation classifier on the output and escalates anything judged low-quality to the stronger model. That escalation step only needs labels for whether a task was done correctly, not extra annotation work.
The escalation gate is only as good as the signal feeding it. Picking the wrong kind of confidence measurement makes the gate fire at the wrong moments, escalating constantly on queries the cheap model actually handled fine, or staying quiet on queries it got wrong.
Three classes of signal exist, and they differ a lot in reliability. Verbalized self-assessment, where the model says something like "I'm not sure about this" in plain language, is the weakest option, because models are not well calibrated at describing their own uncertainty in words. Probe-based or activation-based methods go a layer deeper, training a small classifier to read a model's internal activations and predict when it's about to be wrong. These are the most accurate of the three, but they need access to the model's internals, so they only work with open-weight or white-box models, not with closed APIs.
The survey treats uncertainty-based routing as its own paradigm and identifies response-level signals, confidence scores, log-probabilities, and automated verifier outputs, as the category of information that post-generation cascade decisions actually run on. CP-Router, one method the survey references, applies Conformal Prediction to the routing step: it pulls the logits for each candidate answer, runs them through softmax to get a probability distribution, and computes an uncertainty score as 1 minus the softmax probability for each option. It then routes based on how large the resulting prediction set is. A query with one clearly dominant answer stays on the standard model; a query with several plausible answers escalates to the stronger one.
The threshold T is the one number that decides the whole cost-quality trade-off, and getting it wrong in either direction causes real damage. A threshold set too high causes the system to escalate constantly, costing more than sending everything to the expensive model from the start would have. A threshold set too low lets the cheap model keep handling queries it genuinely can't answer, quietly lowering quality without throwing any error anyone would notice.
Picture a support chatbot built on this pattern. Neither failure appears as a crash or an error log. Both look, from the outside, like a system working normally.
Calibrating correctly means testing the cheap model's confidence signal against a sample of real production queries where the correct answers are already known, finding the point where the signal best predicts the model's own mistakes, and setting T there, not from a guess and not from tuning against a benchmark that doesn't resemble actual traffic. The cascaded framework paper from ADAPT Centre and Huawei Research builds this into the system directly with an interpretable hyperparameter called λ, which controls the cost budget for Stage 1 routing, giving calibration a working model for running as an automated offline step. A threshold calibrated for factual question-answering does not transfer cleanly to open-ended generation. Calibration has to be redone for each task type and each domain the system touches in production.
Every escalation event means paying for two inferences: the cheap model's first attempt, and the expensive model's retry on top of it. The cheap model's generation time gets added to the total response time for anything that ends up escalated, so the slowest path through a cascade is slower than just going straight to the strong model.
For real-time chat, voice interfaces, or any synchronous feature where a user is sitting and waiting on a response, that added latency can erase the gains the cascade was built to capture. For these use cases, cascade routing is often the wrong pattern, and for most other production AI work, batch processing, asynchronous pipelines, document analysis, background classification jobs, the extra time barely registers, and the cost savings dominate the decision.
On AIME 2024 mathematical reasoning, the cascaded framework's two-stage system kept nearly all of the strongest model's accuracy while running at 18% lower Time Per Output Token. That means a cascade can actually come out faster overall than a single strong model, because Stage 1 clears the easy cases quickly and only the hard fraction pays the cost of a second inference. For teams genuinely limited by latency, a few mitigations help: running the confidence check on a signal that resolves before full generation finishes, so slow cases get flagged early; caching high-confidence responses from the cheap model so repeat or near-duplicate queries skip the cheap-model step on future hits; or keeping pre-generation classifier routing for the latency-critical paths and reserving cascade escalation for the asynchronous parts of the workload.
A cascade that's calibrated correctly doesn't just lower the bill. It can raise quality above what either model would produce on its own across the full range of incoming queries, because hard questions consistently reach a model capable of answering them and easy questions never carry the overhead of a model they didn't need. Well-designed routing systems can outperform even the strongest individual model in the pool, by using each model's specialized strengths where they actually apply. The ceiling for the whole system sits higher than the ceiling of any one model inside it.
The two-stage cascaded framework backs this up with a concrete number: it retains 97 to 99% of the strongest model's accuracy on its test datasets while reducing Time Per Output Token, a genuine Pareto improvement in which teams gain better cost without sacrificing accuracy, at the price of a bit of routing complexity.
The risk on the other side of this is what happens when the threshold is wrong and nobody notices. Users just receive worse answers, silently. This failure mode deserves a name: silent regression. The only real defense against it is a monitoring habit built into the system from the start: log every escalation event, track the cheap model's error rate broken out by query cluster and task type, and re-calibrate whenever that distribution shifts, not once a year at contract renewal, but as an ongoing part of running the system.
When escalation logic gets written into application code, every team that builds a service ends up reinventing threshold management, confidence-signal parsing, fallback chains, and cost attribution on its own. That produces inconsistency across services, coverage gaps nobody can see from the outside, and no single place where anyone in the organization can answer a simple question: what's escalating right now, and why? Production AI workloads need things application code struggles to deliver well on its own: unified access across model providers, rate limiting measured in tokens, cost attribution that updates in real time, audit trails showing how every routing decision got made, and the ability to change a threshold value without shipping a new application release. The survey frames the routing mechanism as its own orchestration layer, separate from the models it chooses between, the architectural role a gateway plays.
A gateway sitting at that layer gives a team one place to change threshold values, test them against each other, and roll them back if something goes wrong, without touching application logic. Concentrate is built around exactly this role: a managed LLM gateway connecting teams to more than 130 models through one unified API, with routing, cost attribution, and observability built directly into the platform, so escalation logic gets configured once and then enforced the same way across every service, without each team managing its own provider keys or threshold code.
Building this correctly follows a sequence, and skipping a step reproduces the failure modes covered earlier in this piece.
Start by choosing a confidence signal that matches what access is actually available. If the API in use exposes logprobs, token-probability-based confidence should be the default signal, since it's more reliable than self-assessment and doesn't require special access. Where logprobs aren't available, a lightweight external quality-estimation classifier trained on task-correctness labels pulled from production logs is the better fallback. Verbalized self-assessment shouldn't be the primary signal in either case, given how poorly models judge their own uncertainty in plain language.
Next comes calibration, and this is where the real engineering work sits. Find the point where the signal best predicts the model's own errors, and that point is the threshold. The cascaded framework paper selects thresholds automatically through offline hyperparameter tuning against a held-out test set, a discipline that should carry over into a live production setup.
From there, the cascade needs to live in the infrastructure layer, not inside any one application. Requests should route through a gateway that applies the escalation policy centrally, logs every escalation event, and attributes the resulting cost to the team and project that triggered it. The target for the "strong model" should be a configuration value the gateway can update on its own, without anyone redeploying application code.
Once it's running, escalation rate becomes a core operational metric worth watching closely. A rising escalation rate usually means the query distribution has shifted or the cheap model's relative capability has changed; it is a signal to re-calibrate. An escalation rate falling toward zero needs the opposite kind of attention: confirm the confidence signal is still discriminating between easy and hard queries, rather than assuming everything has simply gotten easier. Tracking the cheap model's error rate on the queries it handles without escalation, separately from the escalated ones, is what catches a silent regression before it does real damage.
Finally, escalation compounds well with other cost controls layered on top of it. Semantic caching lets high-confidence responses from the cheap model get reused for repeat or near-duplicate queries, skipping both model calls entirely on a second hit. Per-team and per-project spend budgets enforced at the gateway level make escalation events visible in real time, instead of appearing as a surprise on an invoice at the end of the month. None of this works as a one-time setup. Getting the threshold logic right, and keeping it right as traffic and models shift, is what separates genuine savings from a quality problem nobody notices until a customer does.