AI Gateway Tier of Azure API Management: Dedicated Governance for Models, MCP Servers, and Agents

By Jeff | SMF Works | July 29, 2026

The governance surface after agents ship

Teams ship agents that call models, invoke MCP tools, and collaborate via A2A. Then reality arrives: every application hard-codes provider keys or endpoints, token budgets disappear into shared pools, observability is fragmented, and platform teams cannot inventory which tools or models are actually in use.

Calling providers directly works for a prototype. It does not scale when multiple teams, multiple providers, and production agents share the same estate.

On July 27, 2026, Microsoft introduced the AI Gateway tier of Azure API Management in public preview. It is not another policy layer on classic APIM. It is a purpose-built experience and resource type optimized for AI workloads: publishing models, exposing MCP servers and tools, applying governance through intuitive policy cards, and delivering unified telemetry.

This post walks through what the tier provides, how it differs from prior AI gateway capabilities, the architecture for production agents, and concrete steps to get value this week.

What the AI Gateway tier delivers

The tier is a fully managed gateway resource you provision in your subscription. Applications call a single gateway endpoint with a runtime access key. The gateway authenticates, evaluates policies, routes to the chosen backend (model or tool), and emits OpenTelemetry GenAI metrics.

Key differences from calling providers directly or using only classic APIM AI capabilities:

Dedicated portal and control plane structured around models, MCP servers, and tools rather than generic APIs.
Card-based policy configuration (no XML or expressions required for common controls).
Runtime access keys scoped to the gateway; provider credentials stay on the gateway.
Self-service catalog so developers discover approved assets with samples.
Multi-provider support out of the box, with consistent policies across them.
MCP and A2A coverage in the same governance surface.

The preview is available in East US 2 and Sweden Central. Pricing and the final business model will be announced later.

Sources: Tech Community announcement (July 27), Azure updates, Learn: AI Gateway tier overview.

Models: one endpoint, many providers

Applications send requests to a stable gateway URL (for example .../models/openai/v1 for OpenAI-compatible or the Anthropic Messages path). The gateway routes by the model field (or equivalent) using an exact match on the name you assigned when publishing the model.

Supported today:

Microsoft Foundry (including OpenAI models hosted there)
Azure OpenAI
AWS Bedrock
Google Vertex AI
OpenAI
Anthropic

A guided wizard helps import models from a Foundry resource. For other providers you supply the endpoint and model names plus backend credentials (API key, managed identity, etc.).

Model aliases let you decouple client-facing names from backend deployments. Change the target of an alias and clients continue using the same name. This simplifies upgrades, A/B testing, and vendor swaps.

Developers can call a /models discovery endpoint on the gateway to list available aliases.

MCP servers and tools: making enterprise capabilities agent-callable

The tier turns MCP servers and tools into governed assets behind one endpoint.

You can:

Expose an existing remote MCP server (SSE or Streamable HTTP).
Upload an OpenAPI spec and turn selected operations into tools.
Use 1,400+ connector-backed tools from Power Platform and Logic Apps with no server to host.
Federate multiple backends behind a single MCP server name so an agent sees a unified tool surface.

Backend authentication options include API key, OAuth client credentials, managed identity, or mTLS. The gateway handles the auth; agents never see the provider secrets.

Once published, agents call the gateway’s MCP endpoint (.../toolservers/<server-name>/mcp) with the gateway runtime key.

Content safety policies now extend to MCP tool-call arguments, MCP responses, and A2A payloads in addition to LLM traffic.

Policies, keys, and observability

Governance is expressed as policy cards in the portal (also manageable as code/JSON). Common cards include:

Token rate limit (tokens per rolling minute)
Request rate limit
Token quotas
Azure AI Content Safety
IP filter
Fallback to a secondary model

Policies apply per asset (model or MCP server/tool). The gateway returns appropriate errors (e.g., 429) when limits are hit.

Administrators sign in with Microsoft Entra ID. Applications authenticate with runtime access keys created on the Keys page. One key per application/environment is the recommended pattern; grant least privilege.

Telemetry uses OpenTelemetry with GenAI semantic conventions. Send to Application Insights, Datadog, Grafana Cloud, or any OTLP endpoint. Token metrics include prompt, completion, total, and additional types such as cached or reasoning tokens where providers expose them.

Better together with Microsoft Foundry

The tier complements the Foundry control plane announced earlier in July. Foundry gives project owners a way to create or associate a gateway and apply project-level token limits. AI Gateway tier provides the deeper, purpose-built experience for platform teams who need to publish models and tools, manage keys, apply advanced policies, and maintain a self-service catalog across the broader AI estate (Foundry-hosted and external providers).

Foundry-hosted agents can consume tools from Foundry toolboxes while the underlying MCP servers and APIs are governed through the gateway.

The result is consistent guardrails whether an agent runs in Foundry, on App Service, or in another runtime.

Practical architecture for production agents

Client / agent runtime → AI Gateway (runtime key) → backend model or MCP/tool

Agent code stays unchanged if it already targets an OpenAI-compatible or Anthropic endpoint — point it at the gateway instead.
Provider keys and connection details live only on the gateway.
Platform team owns the shared governance surface; application teams consume via catalog and keys.
Telemetry flows to your chosen destinations with rich attributes for chargeback, capacity planning, and auditing.

Private networking is supported (inbound Private Link, outbound VNet integration). Data residency follows the full request path (gateway + chosen backends + telemetry).

Getting started this week

Provision an AI Gateway tier resource in a supported region (East US 2 or Sweden Central) via the dedicated portal at ai.gateway.azure.com or the Azure portal experience.
Add models:
- Use the import wizard for Foundry resources.
- Add custom providers for Bedrock, Vertex, etc.
Publish MCP servers and tools from OpenAPI specs, existing MCP endpoints, or connectors.
Create runtime access keys and distribute to applications/agents.
Configure policy cards for rate limits, quotas, and content safety.
Point an agent at the gateway endpoint and test end-to-end.
Wire telemetry to Application Insights and build initial dashboards.

Samples are referenced in the announcement and Learn docs. Start with the quickstart to create a gateway, add a model, and make a first call in 20–30 minutes.

What this changes for builders and platform teams

No more scattered keys. One gateway endpoint and runtime keys replace per-app provider credentials.
Consistent policies across providers. The same rate limits, safety checks, and quotas apply whether the model is in Foundry, on Vertex, or Anthropic.
MCP and tools become first-class governed assets. Agents discover and call enterprise capabilities through the same surface as models.
Observable by default. Token and request metrics with GenAI conventions land in the systems you already use for chargeback and alerting.
Self-service with guardrails. Developers browse the catalog and integrate faster while platform teams retain control.

For teams already using classic APIM AI gateway capabilities, the dedicated tier offers a streamlined experience tailored to AI assets. Classic tiers remain available for general-purpose needs.

Sourced facts and references

Announcement and tier overview (July 27, 2026). Dedicated AI Gateway tier in public preview; card-based policies; models from Foundry + external providers; MCP server publishing and federation; runtime access keys; OpenTelemetry metrics. Source: Tech Community – AI Gateway tier of API Management now in public preview. Azure updates and Learn confirm regional availability and preview nature.
Multi-provider and Unified Model API context. Earlier AI gateway capabilities (Build 2026) introduced unified OpenAI-compatible surface and expanded to Anthropic/Vertex. The new tier builds on that with a purpose-built resource and portal. Sources: prior AI gateway capabilities posts and Learn: AI gateway capabilities.
MCP and tool exposure. Gateway can front remote MCP servers, OpenAPI-derived tools, and 1,400+ connectors. Content safety extends to MCP/A2A. Source: announcement and Learn docs.
Integration with Foundry. Complements project-level control plane and toolbox consumption. Agents and tools governed through the gateway while Foundry owns the agent runtime and project economics. Sources: announcement + earlier Foundry AI Gateway control plane coverage.

Additional reading:

AI Gateway tier (preview) overview – Microsoft Learn
Azure updates – AI Gateway tier
Related samples and quickstarts linked from the announcement.

What to do this week

Provision a gateway in a supported region and import one Foundry model.
Publish a simple MCP server or connector-backed tool.
Update one non-production agent to call through the gateway.
Enable token metrics in Application Insights and review the first day of data.
Review policy cards for rate limits and content safety; apply to the published assets.

The tier is in public preview. Treat it as production-capable for pilots with clear rollback plans and monitor quota/limits as they are documented.

This is the control surface Microsoft is giving platform teams to make AI assets — models, tools, and the agents that use them — governable at scale inside the Microsoft ecosystem.

Live URL (after publish): https://www.smfclearinghouse.com/blog/2026-07-29-ai-gateway-tier-azure-api-management/

Sources: Primary Microsoft Tech Community, Azure updates, and Microsoft Learn documentation (July 2026). All claims trace to the linked posts and docs.