Two years ago, most enterprises using generative AI had one model provider, a handful of applications and a single team that knew where every API key lived. That arrangement has not survived contact with scale. Organisations now run several commercial models, open-weight models on their own infrastructure, and dozens of applications built by different teams, each calling models in its own way.
The LLM gateway has emerged as the architectural answer: a layer that sits between applications and models, through which every request passes. It authenticates callers, routes requests to the right model, applies safety checks, records what happened and counts the cost. Above it, a growing number of organisations are building what is often called an AI control plane, the place where policies, budgets and model approvals are set for the whole estate.
This piece explains why the layer appears, what it must do, what it costs to run, and how Indian regulatory requirements shape its design. Its main contribution is a capability map: eight domains of function that a gateway must cover, with the minimum each one needs and what goes wrong without it.
Why the LLM gateway layer appears
Without a gateway, every application team solves the same problems separately. Microsoft's Azure Architecture Center guidance on accessing models through a gateway, last updated in July 2026, sets out what is left to client code when applications call model endpoints directly. Failover and retry logic become client responsibilities; the guidance notes that "failover logic remains your responsibility regardless of deployment type" and that throttled requests leave clients "solely responsible for handling appropriate back-off and retry logic." Authentication happens at the resource or project level rather than the individual model deployment, which makes least-privilege access harder, and clients outside the organisation's identity tenant may end up sharing a single full-access key. Cost attribution depends on aggregating telemetry after the fact, and monitoring does not happen in real time.
Each of those gaps is manageable for one application. Across thirty, they become an operating risk: inconsistent safety checks, keys copied into configuration files, no single view of spend, and no way to change a model provider without asking every team to redeploy.
The gateway moves those concerns out of applications and into a shared layer. Microsoft describes the general approach as the gateway offloading pattern; its multi-backend guidance calls the gateway "a programmable routing layer in front of your model back ends" that lets routing change "without needing to redeploy client code or change client configuration."
What a gateway is, and what a control plane adds
It helps to separate two ideas that are often merged.
The gateway is a data-plane component. It handles each request in real time: who is calling, which model should serve the request, whether the input or output breaks a rule, what to log and how many tokens were used. It must be fast and highly available, because every model call depends on it.
The control plane is where the rules the gateway enforces are defined and governed. It holds the inventory of approved models and applications, the policies each application is subject to, budgets and quotas, and the approval workflow for adding a new model or use case. It also aggregates the gateway's logs into the views risk, finance and engineering teams need.
An organisation can run a gateway without a mature control plane, configuring policies by hand. It cannot run a control plane without something like a gateway, because policy that is not enforced on the request path is advisory.
A capability map: what an LLM gateway must do
The map below sets out eight capability domains. The four most often discussed, routing, guardrails, logging and cost, are necessary but not sufficient; the other four determine whether the gateway can be trusted with production traffic. For each domain, the minimum is what a gateway should have before it carries regulated or customer-facing workloads.
Three observations come from the map. First, the domains depend on each other. Cost attribution (6) needs identity (1); residency (4) depends on routing (2); incident response depends on logging (5). Buying a gateway strong in one domain and weak in another produces controls with gaps between them.
Second, the minimum column is modest. Most of it is achievable with a well-configured API management product. The mature column is where specialised AI gateway products differentiate.
Third, change management is the most neglected domain. Model providers update models on their own schedules, and a gateway that does not pin versions or detect behaviour changes leaves the enterprise exposed to changes it did not choose. Microsoft's routing guidance makes a related point for failover: "Don't fail over from version X to version X+1," because the difference "can cause unexpected behavior in the clients."
Routing and resilience in practice
The Azure multi-backend guidance gives a useful picture of what serious routing involves. It recommends treating HTTP 429, 500 and 503 responses as signals to break the circuit to a model instance for a period, rather than retrying it repeatedly, and honouring the provider's Retry-After header. It describes spillover: slightly under-provisioning reserved capacity and bursting overflow to pay-as-you-go deployments. And it warns against a single gateway spanning geopolitical regions where data residency applies: "Don't implement a unified gateway across geopolitical regions when data residency and compliance is required." The recommended pattern is one isolated gateway per boundary.
Routing is also where cost and quality meet. Sending simple classification requests to a small model and complex reasoning to a large one is a routing policy. So is sending requests that involve customer personal data only to models hosted in the organisation's own environment. A separate PanScience Research piece covers small language models and when they are good enough; the gateway is where that decision is enforced.
Guardrails at the gateway, and their limits
The gateway is a natural place for guardrails because every request passes through it. Detecting personal data before a prompt reaches an external model, blocking prohibited content in outputs and enforcing output formats can all be done once, consistently, rather than in each application.
But the gateway sees text, not intent or downstream consequences. It cannot know whether an agent is permitted to approve a refund or delete a record; that check belongs in the system that performs the action. The OWASP Top 10 for LLM Applications 2025 makes this point in its guidance on excessive agency, recommending that authorisation be implemented in downstream systems rather than relying on a model to decide whether an action is allowed. A gateway is one layer of defence. A separate PanScience Research piece on AI security for enterprise systems sets out the full threat picture.
Logging and cost: the evidence layer
For most enterprises, the first visible benefit of a gateway is a single view of usage and spend. Cloudflare's AI Gateway documentation, for example, describes analytics on "the number of requests, tokens, and the cost it takes to run your application", logging of requests and errors, caching to "serve requests directly from Cloudflare's cache instead of the original model provider", rate limiting, and request retry and model fallback. Those are now standard features across AI gateway products and cloud API management services.
Two design decisions matter more than the feature list. The first is what to log. Metadata on every call is essential; full prompts and responses are valuable for debugging and audit but are themselves sensitive data, and their retention must follow the same rules as the source data. The second is how cost is attributed. A gateway that knows which application, team and use case made each call allows chargeback, and chargeback changes behaviour faster than any policy memo. A separate PanScience Research piece covers the unit-cost model for enterprise AI.
What a gateway costs
A gateway is not free, and the costs are architectural as well as financial. Microsoft's guidance lists them plainly. A gateway "can introduce a single point of failure". It "introduces added latency for each API invocation" and may become a throughput bottleneck. Streaming responses require long-lived connections without premature timeouts. The gateway "expands the workload's attack surface", and because it sees raw requests and responses, data compliance and regulatory scope extend to it. It carries runtime and operational costs, and some of its functions overlap with features model platforms already provide.
The guidance closes with two rules worth adopting verbatim as design principles: do not implement a gateway if it jeopardises the workload's ability to meet agreed service-level objectives, and do not implement one if it leaves the workload unable to protect the confidentiality, integrity or availability of itself or its users' data. In practice, that means running the gateway with at least the redundancy of the most critical application behind it.
Indian requirements that shape the design
For Indian enterprises, the data protection and residency domain carries particular weight. The Reserve Bank of India's payment data localisation circular of 6 April 2018 requires payment system data to be stored in India. SEBI's framework for the adoption of cloud services by regulated entities, issued on 6 March 2023, sets governance and data location expectations for market intermediaries. The Digital Personal Data Protection Rules, notified in November 2025, include Rule 15 on cross-border transfers and additional obligations, including local storage of specified data, for significant data fiduciaries.
None of these rules mentions gateways. But each makes the gateway the most practical enforcement point: the place where a request containing payment or personal data is detected and routed only to a model hosted in a permitted location, and where evidence of that routing is recorded. The Azure guidance's recommendation of one gateway per jurisdictional boundary maps directly onto these requirements.
From gateway to control plane in practice
PanScience's portfolio includes two ventures that sit in this layer and illustrate the split between data plane and control plane. Proteus describes itself as an LLM gateway that routes, secures and monitors model traffic (company-reported). Own360 describes itself as a control plane for AI-native enterprises (company-reported). The distinction between the two, one on the request path and one governing the policies applied on it, is the same distinction this piece recommends enterprises draw in their own architecture, whether they build, buy or configure.
A staged approach suits most organisations.
Conclusion
The LLM gateway is becoming as standard in enterprise AI architecture as the API gateway became in service architecture, and for similar reasons: once many applications call many back ends, shared concerns need a shared place to live. The capability map in this piece argues that routing, guardrails, logging and cost are the visible half of the job. Identity, residency, observability and change management are the half that decide whether the layer can carry regulated work.
The control plane is the harder organisational problem. A gateway enforces rules; someone has to own them. Organisations that assign that ownership early, and treat the gateway's logs as the evidence base for AI governance, will find that adding the next model or the next application is a configuration change rather than a project.
Methodology and sources
Gateway challenges, capabilities, drawbacks and the two design rules are from Microsoft's Azure Architecture Center, "Access Foundry Models and other language models through a gateway" (learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-guide, updated July 2026). Routing, failover, spillover and residency guidance are from the same centre's guidance on routing across multiple model back ends (learn.microsoft.com/en-us/azure/architecture/ai- ml/guide/azure-openai-gateway-multi-backend, updated May 2026). Feature descriptions are from Cloudflare's AI Gateway documentation (developers.cloudflare.com/ai-gateway). The excessive agency point is from the OWASP Top 10 for LLM Applications 2025 (genai.owasp.org/llmrisk/llm062025-excessive-agency). Indian regulatory references (RBI circular of 6 April 2018; SEBI cloud framework of 6 March 2023; DPDP Rules, November 2025) are as verified by the editor from rbi.org.in, sebi.gov.in and PIB. Vendors are cited for their published documentation, not endorsed. The eight-domain capability map, the minimum and mature levels, and the staged approach are PanScience Research's analytical contribution and are illustrative rather than derived from a measured dataset. Proteus and Own360 descriptions are company-reported from panscience.xyz.