Alauda AI Platform Gateway

Agent Router

Alauda AI Platform Gateway is based on Agent Router, formerly known as Envoy AI Gateway. Agent Router is a Kubernetes-native, AI-specific gateway layer built on top of Envoy Gateway, providing intelligent traffic management, routing, and policy enforcement for AI inference workloads.

Main components and capabilities include:

  • AI-Aware Routing: Routes inference requests to the appropriate backend model service based on request content, model name, and backend availability — enabling transparent multi-model serving behind a single endpoint.
  • OpenAI-Compatible API: Exposes a unified, OpenAI-compatible API surface (/v1/chat/completions, /v1/completions, /v1/models) for all downstream inference services, regardless of the underlying runtime.
  • Per-Model Rate Limiting & Policies: Enforces fine-grained rate limiting, token quotas, and traffic policies at the individual model level, preventing resource starvation and ensuring fair usage across tenants.
  • Backend Load Balancing: Distributes inference requests across multiple replicas of the same model using configurable load-balancing strategies, with health checking and automatic failover.
  • Envoy Gateway Integration: Runs as an extension of Envoy Gateway, inheriting its Kubernetes Gateway API-native control plane, TLS termination, and observability features (metrics, access logs, distributed tracing).
  • Gateway API Inference Extension (GIE): Integrates with the Kubernetes SIG Gateway API Inference Extension for advanced, inference-aware scheduling and load balancing decisions based on real-time backend state.

Alauda AI Platform Gateway is a required dependency of Alauda AI Platform Inference Management for exposing inference services.

Alauda AI Platform Gateway is installed and managed by Alauda AI Platform through the envoyAIGateway component in the default AmlCluster. The component key is unchanged by the rename. For installation and component configuration, see Install Alauda AI Platform.

Guides

The following guides configure Agent Router as a multi-tenant model serving control plane:

Create the gateway in a dedicated namespace

Create the Gateway and its AIGatewayRoute in a dedicated namespace, such as maas-system, rather than in the Envoy Gateway control-plane namespace envoy-gateway-system. This keeps tenant gateways separate from the control plane, and avoids an issue on some versions where a gateway placed in the control-plane namespace does not get the AI Gateway request-processing (ext_proc) filter or SecurityPolicy rules applied to its listener — which silently breaks model routing, token quotas, and authentication. The data-plane proxy pods are created in envoy-gateway-system either way; only the Gateway resource's namespace matters here.

Documentation

Agent Router upstream documentation and related resources: