Access Controls Audit Logs and ZDR as Enterprise LLM Gateway Selection Criteria
Gateways become critical when AI agents multiply access points and compliance demands rise.
Concentrate.ai
Gateways become critical when AI agents multiply access points and compliance demands rise.
Concentrate.ai
LLM Gateway
Change models, providers, add fallbacks, and more by running all AI usage through one platform.
Teams running large language models in production no longer treat the gateway as plumbing. It has become the primary control surface for deciding who can call which models, under what conditions, and with what data attached to the request. That shift didn't happen because vendors marketed it that way. It happened because the underlying workload changed shape.
Gartner projects that roughly two-fifths of enterprise applications will carry task-specific AI agents by the end of 2026, up from a small fraction in 2025. Each of those agents fires off multiple model calls per task, so the number of places where something can go wrong, unauthorized access, leaked data, spend nobody approved, multiplies fast. Layer that on top of the fact that most production setups now run several providers side by side (OpenAI here, Anthropic there, Bedrock and Vertex and Mistral scattered across different teams) and you get fragmented credentials, rate limits that don't line up, and no single record of what happened unless something in the stack is built to keep one. Regulated industries have already drawn the conclusion: they expect compliance-grade isolation, audit logging, and single sign-on to come from the gateway itself, not get bolted onto the application layer after the fact.
A proxy that passes requests through and a gateway that governs them look similar in a demo. They behave completely differently once real teams, real budgets, and real auditors show up.
A basic proxy forwards a request with little or no transformation. An enterprise-grade gateway terminates that request, applies policy against it, normalizes the quirks between providers, enforces spending limits, and hands back signed telemetry before the response ever reaches the client. In dev-tool setups, it's common to find one shared provider key passed around a whole engineering org, no way to tell which team burned through the budget, and security features listed as "coming soon" on a roadmap slide. Enterprise gateways work differently by default: virtual keys that never expose the underlying provider credential, budget checks enforced at the moment of the request rather than after, RBAC tied into the company's existing identity provider, and logs that are signed and immutable, all standard rather than something reserved for a pricier tier.
The gap between those two categories rarely appears during initial rollout, and instead becomes visible later, usually when a compliance review lands on someone's desk and the answers aren't there. Picture a mid-size engineering team that picks a lightweight, open-source gateway to get a chatbot prototype into production fast. It works fine for months. Then a customer in a regulated industry asks for a SOC 2 report, and someone discovers the gateway has no audit trail granular enough to show who touched what data, no RBAC tied to the company's SSO, and no way to prove sensitive fields were ever redacted. The team ends up migrating gateways mid-flight, re-plumbing every service that talks to a model provider, while the deal sits on hold. That's the exact scenario a proper evaluation up front is meant to prevent.
Shared static provider keys are the single most common failure mode in AI deployments that lack governance.
Without a virtual key layer, anyone holding a provider API key has open-ended access to every model that key can reach. There's no record of who used it for what, and the only way to cut someone off is to rotate the key for the entire team. Virtual keys fix that by enforcing budgets, rate limits, and access rules at several levels at once, per key, per team, per customer, so one runaway agent or one bad retry loop can't drain a shared budget meant for everyone else.
At any real scale, the rule is simple: developers and internal applications should never hold a static provider key directly. Authentication should route through the company's existing identity provider, Okta or Microsoft Entra ID, using OAuth 2.0 or OIDC. SCIM 2.0 provisioning closes a related gap that's easy to overlook: when someone leaves the company, their access to every downstream model endpoint gets revoked automatically, rather than lingering because nobody remembered to rotate a shared key.
This isn't a hypothetical risk. IBM's Cost of a Data Breach research found that a large majority of organizations that had an AI-related security incident also lacked proper AI access controls. Separately, a Cloud Security Alliance survey found that most organizations had discovered an AI agent or workflow running somewhere in their environment that security or IT never knew existed, and a majority of those organizations had already suffered an AI agent security incident in the same period. Shadow AI is the visible symptom of missing role-based access control.
Different gateways handle this at different depths. Bifrost ships virtual keys with hierarchical governance across key, team, and customer levels, along with SAML-based SSO and RBAC, as Enterprise-tier features. Portkey builds workspaces, RBAC, SSO, and SCIM integration into its managed product. Concentrate provides RBAC and team-scoped access controls through a managed gateway, without asking engineering teams to build or maintain that identity layer themselves.
An auditor doesn't ask whether a request happened. They ask something closer to: show every request that touched protected health data this quarter, name who made each one, and prove the record hasn't been altered since. An HTTP access log can confirm a request occurred. It can't answer any of the rest.
A compliance-grade audit log is a different artifact. For each request, it needs to capture a timestamp, the actor's identity, the virtual key used, the provider and model called, a hashed digest of the input rather than the raw text, which guardrails fired, which tool calls executed, token counts, and a cryptographic signature over the whole entry. That signature is what turns a log from a record into evidence. Hash chains, HMAC signatures, or WORM storage make the log tamper-evident, and that property is specifically what security teams ask to see during SOC 2, HIPAA, and ISO 27001 review, because it proves nobody altered the record after the fact.
Agentic workloads make this substantially harder to get right. A single agent task can spawn dozens of model calls, invoke tools, and kick off sub-agents, producing a full call graph rather than one tidy log line. A gateway that only logs the top-level request misses exactly the inner tool calls where sensitive data tends to travel. Bifrost includes audit trails and compliance logging as enterprise features. Portkey offers observability by app and model, with user and workspace context, though not broken out by internal team. Concentrate provides granular, real-time audit logs across teams, projects, keys, models, and providers, structured so a security or finance reviewer can reconstruct a full session without ever needing access to raw prompt content.
Picture a team that flips on zero data retention with its model provider, satisfied the box is checked, while its gateway keeps caching full prompt text for performance and its audit log stores the raw input alongside every entry. Nothing about the provider's ZDR contract touches either of those layers. Turning off retention at one point in the pipeline doesn't mean the data stops existing somewhere else in it.
Zero data retention has to work as a control that spans the full stack, the gateway, the cache, and the upstream provider all at once, enforced at the moment each request is processed rather than assumed after the fact. That means the gateway commits to never logging prompt content, the cache either gets disabled entirely for ZDR-flagged traffic or stores nothing more than non-identifying hashes, and a policy layer actively blocks any configuration change that would quietly weaken the guarantee. One gateway vendor shipped exactly this as an explicit enterprise control, dated September 4, 2026, describing it in its changelog as a separate rule that keeps the gateway, cache, and provider layers aligned and blocks incompatible changes before they can undo the guarantee.
For regulated data categories, personal information, health records, payment card data, ZDR has to work hand in hand with redaction. Sensitive fields need to get detected and stripped before the request ever leaves the network perimeter, that redaction event needs its own log entry (what got redacted, under which policy), and the raw content should never sit in storage anywhere in the stack. Concentrate's position on this is direct: sensitive customer data, PCI, PHI, should never reach a model provider unguarded, and redaction should be reviewable and enforceable, not a best-effort filter.
RBAC, audit logs, and ZDR don't function as three separate boxes on a feature comparison sheet. They form a closed loop, where access controls decide who can make a request, audit logs prove what that request contained and how it got handled, and ZDR decides what portion of that content actually gets retained and can be proven so. Break any one link and the other two stop meaning much.
RBAC without audit logs means access can be restricted, but there's no way to prove that restriction actually held under real traffic. Audit logs without ZDR turn the log itself into a liability, since a log full of raw prompts is exactly the kind of data a breach would expose. ZDR without RBAC applies a retention policy to traffic nobody can trace back to a specific person or team. A security reviewer works through these in a fairly predictable order: who has access, using RBAC and SSO as evidence; can the reviewer see what those people actually did, judged by how complete and immutable the audit log is; and can the team prove sensitive content was never retained, backed by ZDR policy and proof of enforcement. A gateway that answers two of those three questions still fails the review.
Agentic workloads raise the stakes further. A multi-step agent calling tools, pulling from external data sources, and routing sub-tasks across different models touches all three criteria inside a single session. The gateway has to attribute every one of those calls back to an actor, log the entire call graph down to the tool calls, and apply ZDR at each hop rather than just the first request in the chain. That's why the sensible move for buyers is to test these three capabilities together, in something close to a real working environment, against a workload that resembles actual production traffic, rather than checking them off one at a time in a spec sheet. Concentrate ships all three as integrated capabilities. Bifrost and Portkey place RBAC and audit logs behind higher-tier or enterprise licensing rather than folding them into the base product. Any gateway that treats one of these three as an upsell creates precisely the gap a review is most likely to expose.
Teams that evaluate gateway security posture systematically, with concrete questions tied to RBAC, audit log completeness, and ZDR enforcement, can identify compliance gaps before adoption rather than during an incident or audit.
On access control, ask whether virtual keys keep application code from ever touching a raw provider credential, and whether RBAC enforces per-team and per-project limits the moment a request is made, rather than only showing up as a warning in a dashboard afterward. Confirm SSO plugs into the identity provider already in use, Okta or Entra ID, over OAuth 2.0 or OIDC, and that SCIM propagates an offboarding event automatically instead of leaving a stale key active. Check whether access policies can be locked down so a developer can't quietly escalate their own permissions without a second person signing off.
On audit logs, confirm every request entry includes the actor's ID, the virtual key, the provider and model, guardrails applied, tool calls, and a cryptographic signature such as HMAC. Check that the log also covers configuration changes, policy edits, key rotations, not just request traffic. For agentic use cases, verify the log captures the full call graph down to inner tool calls, rather than only the top-level request. And check whether logs can export into an existing SIEM without a custom integration project.
On zero data retention, confirm ZDR applies at the gateway layer itself, not just at the provider, and that it covers the cache and audit log storage as separate concerns rather than assuming provider-level ZDR reaches them. Ask whether the gateway can produce a log entry proving a request was processed under ZDR policy without ever storing the content that request contained. Confirm PII redaction happens before data leaves the network perimeter, and that the redaction event itself gets logged along with the policy that triggered it. Check whether a configuration change that would weaken ZDR gets blocked automatically by policy, or just waits for someone to notice during manual review.
A gateway that only runs in managed, hosted-by-someone-else mode routes every prompt through a third party's infrastructure, and for regulated workloads that's often flatly incompatible with data residency requirements, regardless of how strong the gateway's security features look on paper. Self-hosted, in-VPC, or air-gapped deployment options are table stakes for teams in that position. They're table stakes.