From LiteLLM Deployment to Instant Inference Without a Proxy to Maintain
Self-hosted AI gateways demand more operational work than most teams anticipate.
Concentrate.ai
Self-hosted AI gateways demand more operational work than most teams anticipate.
Concentrate.ai
LLM Gateway
Change models, providers, add fallbacks, and more by running all AI usage through one platform.
Every production AI system now needs a layer sitting between the application and the model providers it calls. The gateway handles routing, load balancing, fallbacks, authentication, rate limiting, cost tracking, caching, and observability for all LLM traffic in one place, instead of leaving it scattered across individual services. Once traffic crosses more than one provider or model, teams can no longer treat this layer as optional.
A traditional API gateway solves a narrower problem: route requests, check auth, enforce limits. An AI gateway carries all of that plus a layer of complexity unique to language models: providers charge by token rather than by request, models vary in latency and quality even when answering the same prompt, and outputs are probabilistic. That's a different kind of engineering problem than routing REST calls to a database. The real question facing engineering teams today isn't whether to run this layer, but who operates it and what that costs over time.
LiteLLM earned its place as the default open-source entry point for a simple reason: it works, and it's free to start. Maintained by BerriAI, LiteLLM normalizes more than 140 providers behind a single OpenAI-compatible endpoint, with no license fee attached. A team can point existing OpenAI SDK code at a different model provider by changing a base URL, which is a meaningful unlock for anyone juggling multiple vendors.
The feature set explains the adoption curve. LiteLLM bundles a unified API, virtual keys with per-key budgets and model allowlists, rate limiting, fallback logic, spend tracking, and logging hooks, all inside one process a team can run itself. The proxy runs as a FastAPI server, distinct from the lighter Python SDK that some teams embed directly into application code. Running the proxy as infrastructure, not just importing the SDK into a script, is what turns LiteLLM into a real gateway.
Getting that proxy to a credible production state asks for more than a single running container. The official production architecture guidance calls for two or more replica instances behind a load balancer, because the proxy itself holds no state. It calls for a private, managed PostgreSQL database to store keys, teams, spend logs, and configuration, backed up automatically and migrated through a separate job. It calls for Redis, version 7.0 or newer, to share rate limits, routing state, and cache data across every replica. It calls for a secret store to hold provider credentials and master keys, metrics and logs and traces that don't leak prompt content, and an upgrade path that keeps a rollout from colliding with a database schema change mid-flight.
Configuration lives in version-controlled files, and because the virtual key system attributes spend and access down to the individual workload, it depends on that PostgreSQL instance to function. None of this is optional hardening layered on top of a working setup; it's the floor. A single container exposed to the internet is a demo. A production gateway is that stack, running continuously, correctly, under load.
Standing up that stack once is the easy part. Keeping it running is the ongoing cost that most teams underestimate when they first adopt LiteLLM, and it grows heavier as usage scales.
Patching alone is a recurring job. Someone on the team has to track every LiteLLM release, evaluate what changed, and decide when to upgrade. Pinning a version protects against unvetted code reaching production, but it also means a person has to make every upgrade decision, on a schedule nobody else is keeping.
Memory behavior under load is a known hazard. If there's no hard memory limit enforced at the process level, the proxy can run out of RAM under sustained high concurrency, and the operating system can kill it, often during off-hours when nobody is watching the dashboard.
Key rotation carries a quieter trap. Rotate a provider API key and reload the config, and the old key often stays alive in the in-memory cache. Requests keep flowing on the revoked key with no errors, until the provider actually invalidates it. At that point every request to that provider starts returning 401 errors, all at once, and the fix is an explicit cache purge or a worker restart that nobody scheduled.
Retry behavior changes the cost math without most teams noticing. The default of two retries turns one failed call into three times the token spend for that request. On 4xx errors, that same retry logic can trigger rate-limit cascades that multiply cost within minutes, and suppressing that behavior takes deliberate reconfiguration, not a default setting.
Visibility gaps compound the problem. The built-in Prometheus metrics, while present for per-provider latency percentiles, per-provider error rates, per-team cost breakdown, and cache hit and miss ratios, still leave on-call engineers guessing at the root cause during an incident: was it LiteLLM, the database, or the provider itself. Throughput adds its own ceiling. Independent benchmarking shows the proxy's P99 latency climbing and its memory spiking as concurrency rises toward what a single instance can handle, so capacity planning has to account for that curve instead of assuming linear scaling.
Governance is the final gap, and it is a licensing constraint. SSO, SAML, SCIM, audit logs, and formal SLAs sit behind LiteLLM's Enterprise commercial license (SSO is free for up to five users, and an enterprise license is required beyond that). Any team that needs Okta integration or audit evidence for a compliance review can't get it from the open-source version alone.
In March 2026, a supply chain compromise hit LiteLLM directly, and it changed how teams should weigh the risk when they run any self-hosted proxy. Two compromised package versions, 1.82.7 and 1.82.8, sat on PyPI carrying a credential-stealing payload that harvested SSH keys, cloud provider credentials, Kubernetes configuration files, and API keys from any machine that installed them.
The packages were live on PyPI for roughly 40 minutes before quarantine. Detection wasn't the result of a security scan catching the payload. A bug in the malware itself caused affected machines to crash: a fork bomb triggered by its own launcher re-triggered on every interpreter startup, and that crash is what surfaced the incident.
Two more vulnerabilities followed close behind, separate from the supply chain compromise but part of the same period. A critical SQL injection flaw in API-key verification affected versions 1.81.16 through 1.83.6. A command injection vulnerability reachable through an MCP test endpoint affected versions 1.74.2 through 1.83.6.
The structural exposure behind all three incidents is the same one every self-hosted proxy carries: it holds API keys for every provider configured in its environment, in one place. A single compromised process can harvest every one of those credentials at once, across OpenAI, Anthropic, Google, and whatever else the team has configured. Operating safely after this class of incident means version pinning, running official container images, dependency scanning, and a controlled upgrade policy, each one an added responsibility for the team running the proxy.
None of this reflects a flaw in LiteLLM's routing logic. The vulnerability sat in the distribution and dependency chain, the path a package takes from a registry to a running machine, and that risk applies to any self-hosted open-source infrastructure pulled from a package registry, not to LiteLLM specifically. Running a proxy in-house is a legitimate choice for teams equipped to own that chain end to end. For teams without a dedicated security function watching PyPI advisories in real time, the risk sits unpriced until an incident like this one forces the accounting. Managed gateways isolate provider credentials server-side rather than in a team-controlled process, and many support Zero Data Retention APIs and bring-your-own-key arrangements for teams with strict data residency rules, which removes that particular blast radius from the equation.
Once a team stops spending engineering hours on patching and incident response, what's left is the actual capability a gateway is supposed to deliver: smart routing and real cost control. A production-grade gateway can route by latency, sending each request to whichever provider responds fastest; by usage, sending it to the deployment carrying the lowest token-per-minute load; by cost, sending it to the cheapest model that still qualifies for the task; or by how busy each endpoint is at that moment, measured by concurrent request count.
Tiered dispatch is the highest-leverage move a team can make on cost. Sending the bulk of queries to a budget-tier model, a smaller share to a mid-tier model, and only the hardest cases to a premium model cuts average per-query cost substantially compared to routing everything through one premium model by default. Semantic caching adds to that effect anywhere queries repeat: near-duplicate requests get served from cache with a sharp drop in latency, and exact matches save even more.
None of that saves money without accurate attribution, and attribution depends on metadata tagged consistently on every single request: input tokens, output tokens, model used, and tags for team, feature, and environment. Without that tagging, a team is left with one aggregate cost number that offers no basis for action. A managed gateway enforces that tagging automatically instead of trusting each developer to add it correctly by hand, so finance can see cost by business unit and engineering can see cost by feature and model, both reading from the same underlying data.
Budget caps close the loop. Without them enforced automatically, one team running a high-volume generation job can burn through a quarter's AI budget in days, and catching that after the fact in a monthly invoice is too late to matter. You need to enforce that policy at the gateway layer, before the spend happens, not catch it later with a dashboard someone checks occasionally.
Any team sending real user data to a third-party model provider needs to think about these controls from the first production request, not just once a compliance audit is scheduled. PII redaction, stripping sensitive data out of a prompt before it ever reaches OpenAI, Anthropic, or any other provider, is a control that matters under GDPR, HIPAA, and PCI DSS alike. Building that redaction logic separately into every application or every provider integration creates inconsistency, and that inconsistency appears as a gap during an audit.
Zero-data-retention routing solves a related problem by sending requests only to providers contractually committed to not retaining or training on prompt data, a rule that can be enforced centrally at the gateway without touching a single line of application code. Data loss prevention scanning, catching PII, financial data, and healthcare data in both the prompt going out and the response coming back, belongs in that same infrastructure layer rather than bolted onto individual features one at a time.
Access control and audit trails follow the same logic. Running a self-hosted gateway means the team owns that audit trail directly: every update gets its own internal security review, and every compliance framework needs its own evidence gathered by hand. Managed gateways handle SOC 2 compliance, penetration testing, and infrastructure security on the provider's side, and the team still keeps full control over its own routing logic and policies.
Credential sprawl, keys scattered across providers and configuration files with no central control, is the structural risk in a multi-provider setup. Static API keys stored in a proxy's configuration file make a high-value target for an attacker, and the March 2026 incident showed exactly that. A managed gateway keeps those credentials in isolated, audited infrastructure instead of in files an engineering team maintains by hand, which removes one of the more common paths to a breach before it opens.
Headline uptime numbers from model providers don't tell you what you'll actually experience when you run traffic in production. OpenAI had a major outage in June 2025 that lasted many hours and affected ChatGPT, the API, and Sora all at once. Anthropic logged numerous incidents across a 90-day window in early 2026, with a median resolution time past an hour for the major ones. Across the industry, average API uptime fell between Q1 2024 and Q1 2025, a decline a vendor's advertised SLA figure doesn't capture, since it tends to describe the best case.
A routing layer that handles fallback automatically changes what an outage looks like to the end user. When a primary provider is rate-limited or down, redirecting to an equivalent model on a different provider turns what would have been a hard outage into something closer to a brief slowdown at slightly higher cost, invisible to whoever is on the other end of the application.
Naive, stateless failover can keep uptime metrics looking clean while quietly dropping conversation history mid-session, which degrades the user's experience even though every dashboard says the system is healthy. Building fallback chains correctly, testing them, and staying on call for when a provider failure exceeds a configured threshold is real engineering work in a self-hosted setup. A managed gateway handles that logic at the infrastructure level, as a property of the platform rather than a project a team has to maintain indefinitely.
Every operational cost described above, the patching cycle, the memory limits, the retry storms, the supply chain exposure, the compliance gaps, the fallback engineering, is the specific tax a managed gateway is built to remove. Concentrate provides routing, governance, security, and observability as managed infrastructure, reachable through one unified API, with no proxy to deploy, patch, or babysit.
That single API connects engineering teams to more than 130 LLM models, including those from OpenAI, Anthropic, and Google, without separate provider keys, custom integration code for each one, or any self-hosted infrastructure to run. There's no PyPI dependency and no self-hosted attack surface like the one the March 2026 LiteLLM incident exposed, because provider credentials live in isolated, audited infrastructure rather than in a configuration file on a team-managed server.
Routing intelligence, cost-based, latency-based, and fallback, is managed at the infrastructure level with full transparency into where each request goes, rather than opaque model-swapping happening somewhere a team can't inspect. Spend visibility is real-time and granular by team, project, key, model, and provider, drawn directly from live usage data rather than reconstructed after the fact from an invoice at the end of the month, and budget caps are enforced automatically. PII redaction, audit logs, RBAC, and SSO are available to teams of any size from day one, not held back behind an enterprise license tier the way LiteLLM's governance features are. Concentrate charges no per-token platform fee on top of provider pricing, so teams pay provider rates, and no markup eats into their AI spend as usage grows.
The choice between self-hosting LiteLLM and routing through a managed gateway is a real tradeoff, and it depends on whether an engineering team wants to own that infrastructure layer as a permanent responsibility or hand it to a platform built to carry it instead.