Azure Maia 300 architecture: Microsoft’s co-designed LLM inference stack with rack-scale power and memory tradeoffs
Microsoft says Maia 300 is co-designed across software, silicon, racks, networking, and datacenters for cloud-scale agentic LLM inference, with claimed gains in memory bandwidth, rack density, and token throughput versus Maia 200.
What happened
Microsoft’s Azure blog says Maia 300 is a third-generation system built for LLM inference, with software, silicon, rack infrastructure, networking, and datacenters co-designed as one platform for cloud-scale agentic workloads. The post says the design aims to keep data closer to compute, using SDLA control, explicit DMA management, NUMA across chiplets, and a memory hierarchy with 288 GB of HBM4 and 18 TB/s per accelerator. Microsoft also reports rack-level integration of 96 accelerators in a 167-kW rack and claims up to 2.5× better performance per watt and up to 7× greater rack-scale token throughput versus Maia 200. Those comparisons are Microsoft’s own reported results, not independent reproduction.
Why it matters
FACT: Microsoft says Maia 300 is co-designed across software, silicon, rack infrastructure, networking, and datacenters for cloud-scale agentic LLM inference, and it reports up to 2.5× better performance per watt than Maia 200. INFERENCE: The decision implication is to evaluate inference platforms as a power-and-locality problem, not just a chip benchmark. Limitations: these are Microsoft’s own claims, not independent validation.
Engineering impact
FACT: Microsoft describes SDLA-controlled data movement, NUMA across chiplets, and 288 GB of HBM4 with 18 TB/s bandwidth per accelerator. INFERENCE: For engineering teams, the relevant constraint is whether their workload needs KV-state residency and short data paths more than raw FLOPS, because the post frames memory bandwidth and locality as the gating factors. Limitations: workload fit remains unknown from this source alone.
Leadership impact
FACT: Microsoft says Maia 300 places 96 accelerators in a 167-kW rack and claims up to 7× greater rack-scale token throughput versus Maia 200. INFERENCE: Leadership should treat datacenter power, cooling, and rack density as capacity-planning constraints, because the reported gain is tied to rack integration rather than only device-level speed. Limitations: the throughput figure is vendor-reported and setup-specific.
Editorial judgment
This changes the infrastructure decision from “buy faster chips” to “optimize the full inference system under a fixed power envelope.” The useful lesson is architectural: memory locality, rack integration, and switch design are presented as first-class levers for agentic inference, but only as Microsoft’s reported design and performance claims.
Watch next
FACT: Microsoft says Maia 300 uses three liquid-cooled switch trays with six high-radix switch ASICs and that next month it plans to share more AI strategy and technology-adoption details at Ignite 2026. INFERENCE: The next decision point is whether Microsoft provides comparative evidence on real agentic workloads, failure tolerance, and rack-level efficiency under matched conditions. Limitations: this source does not supply independent benchmarks or deployment data.
Limitations & uncertainty
All conclusions are constrained to Microsoft’s own Azure blog descriptions and performance claims. The source does not independently verify the reported comparisons, does not provide full benchmark conditions, and does not establish real-world serving outcomes across other workloads or operators.