Trendy Ingress for AKS: Introducing Utility Gateway for Containers (AGC)

Good day People!

This Microsoft Azure Infra Summit 2026 classes one is for everybody operating manufacturing workloads on AKS. Jack Stromberg from the Utility Gateway product workforce walked us by your entire historical past of load balancing in about ten minutes (no small feat) after which dropped the information that Utility Gateway for Containers (AGC) is selecting up a model new trick: AI gateway capabilities by the CNCF Gateway API Inference Extension. In case you have ever watched one pod get hammered whereas the others sit idle, this session goes to really feel private.

📺 Watch the session:

The NGINX Ingress Controller retirement is pushing loads of groups to select a brand new entrance door for his or her AKS clusters. AGC is Microsoft’s managed, Azure-native reply. It’s Layer 7, it speaks each the Kubernetes Ingress API and the newer Gateway API, and it runs exterior the cluster so you aren’t paying with pod CPU to terminate TLS.

Right here is the quick checklist of why this session issues:

• The web runs on load balancers, and the load balancer story has been largely secure for many years. That’s altering quick.
• AI inference visitors doesn’t appear like internet visitors. Lengthy-lived, GPU-bound, costly. Conventional Layer 7 guidelines can’t sustain.
• AGC now ships AI gateway capabilities (in preview) that route by mannequin, route by load, and cease slamming a busy pod when others are idle.
• Measured numbers from the demo: zero timeouts versus 15 failed requests, and a P99 time-to-first-token of about 1,000 ms versus 16,000 ms on a vanilla HTTPRoute.

In case your AKS clusters are beginning to host self-hosted fashions alongside common companies, you should know this exists.

AGC is the evolution of the older Utility Gateway Ingress Controller (AGIC). The structure is the half that issues:

• Utility Gateway for Containers useful resource: the managed Azure useful resource that really handles visitors, TLS, WAF, and routing.
• Frontends: listeners and IP addresses that settle for incoming visitors.
• Associations: the hyperlink between the AGC useful resource and your AKS VNet/subnet.
• ALB Controller: a small controller that runs inside your AKS cluster, watches your Gateway API and Ingress assets, and reconciles the AGC configuration in close to actual time.

Two deployment fashions. You possibly can let the ALB Controller handle the lifecycle of the AGC useful resource for you, or you may go Convey Your Personal (BYO) and handle it with Bicep, Terraform, ARM, or the CLI. I like BYO for manufacturing as a result of it offers the platform workforce a clear separation between Kubernetes intent and the Azure useful resource lifecycle.

Function-wise it covers what you’d count on from a contemporary Layer 7: automated retries, autoscaling, zone resiliency, weighted/break up visitors for blue-green, header and URL rewrites, WAF integration, mTLS (entrance, again, or end-to-end), HTTP/2, WebSocket, gRPC, and server-sent occasions. Routing by hostname, path, header, question string, or HTTP technique.

Jack opened with a tour of the OSI mannequin that I genuinely loved. The TL;DR:

• Layer 4 (community load balancer): stateless, quick, easy. Supply IP, vacation spot IP, ship. This nonetheless powers many of the web.
• Layer 7: TLS, session affinity, WAF, mutual auth, clever routing based mostly on the contents of the request. That is the place fashionable internet purposes reside.
• Kubernetes ingress: the Layer 7 load balancer is now not a {hardware} equipment you rack and stack. Pods come and go continuously. You want a controller that interprets Kubernetes intent into load balancer config in close to actual time.
• Gateway API: the CNCF’s vendor-neutral technique to specific that intent. AGC is a first-class Gateway API implementation, so the YAML you write is transportable throughout clouds and on-prem.

In Gateway API land you’re employed with three major assets:

1. A Gateway (the entrance door, together with listeners and hostnames).
2. An HTTPRoute that claims “for requests matching this host/path/header, ship them to this backend.”
3. A backend, historically a Kubernetes Service.

That’s the image for regular internet apps. AI workloads break it.

Right here is the sincere tradeoff Jack laid out. Conventional Layer 7 load balancers had been tuned for short-lived, low-latency requests. They distribute visitors with easy algorithms (spherical robin, least connections) as a result of they assume backends are interchangeable.

AI inference flips each a kind of assumptions:

• Requests are long-lived. A mannequin reasoning over an enormous immediate can take many seconds.
• Backends should not interchangeable. Pod A may be operating a picture mannequin, Pod B a coding mannequin.
• Compute is dear. GPUs should not low cost, and a sizzling pod with a full KV cache can’t simply settle for extra work.
• A standard load balancer will fortunately pile new requests on a pod that’s already saturated, whereas idle pods sit ready.

The repair is the CNCF Gateway API Inference Extension. It introduces 4 primitives, and AGC implements them so that you do not need to bolt the items collectively your self:

• Physique-Primarily based Router (BBR): reads the immediate physique to tug out the requested mannequin. In AGC, this runs contained in the load balancer itself. Nothing further so that you can deploy.
• Endpoint Picker (EPP): runs in your cluster, scrapes metrics out of your mannequin pods (issues like KV cache utilization), and tells the load balancer which pod is definitely an excellent goal proper now.
• InferencePool: a brand new Kubernetes useful resource that teams pods serving the identical mannequin together with the EPP and goal port.
• InferenceObjective: permits you to specific precedence when a number of requests are competing for a similar pool.

The neat half is that InferencePool plugs in as a backendRef on an HTTPRoute, proper subsequent to conventional Service backends. You possibly can combine AI and normal workloads behind the identical gateway. No new gateway, no new product, simply a further backend kind.

Jack ran two demos that I wish to name out:

• Mannequin-aware routing: a script fires curl requests with totally different prompts. AGC’s body-based router pulls the requested mannequin out of the JSON physique, units a header, the HTTPRoute matches the header, and visitors lands on the proper InferencePool. The vanilla HTTPRoute facet can’t do that and loads of requests fail.
• Load-aware routing: three pods, one already busy. Fireplace 50 requests on the AI-enabled route and 50 on the conventional route. The endpoint picker steers visitors to the idle pods. P99 time-to-first-token: about 1,000 ms with AGC’s AI gateway, about 16,000 ms with out. Timeouts: zero with AGC, 15 with out.

That could be a actual quantity, on the identical backend, in the identical cluster. The one factor that modified is the load balancer understanding what is going on behind it.

If you wish to do that, right here is the trail I might take:

• Rise up an AKS cluster with Azure CNI, OIDC issuer, and Workload Identification enabled. AGC wants these.
• Register the suppliers: Microsoft.ContainerService, Microsoft.Community, Microsoft.NetworkFunction, Microsoft.ServiceNetworking.
• Create a user-assigned managed identification for the ALB Controller and grant it the AGC Configuration Supervisor position.
• Set up the ALB Controller by way of Helm or as an AKS add-on, then create your Gateway useful resource in opposition to the azure-alb-external GatewayClass.
• Begin with a plain HTTPRoute in opposition to a daily Kubernetes Service. Get that inexperienced first.
• If you find yourself prepared for inference, deploy your mannequin, drop in an InferencePool and an Endpoint Picker, and level your HTTPRoute on the pool.

Pierre’s tip: the AI gateway capabilities are in preview. The workforce explicitly requested for suggestions from anybody operating self-hosted fashions in AKS, particularly round mannequin combine, request patterns, and what you’re doing immediately to maintain sizzling pods from getting buried.

Catch each session from the Microsoft Azure Infra Summit 2026 on the official playlist: https://www.youtube.com/playlist?checklist=PLjt5SKzX1iI8con7FJDB56G6hHqxGm7ki. There’s a companion session on AKS Safety with AGC and Managed Cilium that pairs rather well with this one.

Cheers!

Pierre Roman