Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCm
A deep dive into vLLM's 7 attention backends for AMD ROCm, featuring 1.2-4.4x throughput gains with ROCM_AITER_FA and MLA optimizations.
At Embedded LLM, our mission is to push the boundaries of what’s possible on open-standard hardware. For years, the narrative surrounding AMD in the LLM space focused on "parity"—simply getting kernels to run.
Today, we are shifting that narrative to performance leadership.
In collaboration with the AMD AI Framework team, we have moved beyond simple code porting into the realm of architectural co-design. By orchestrating software and hardware primitives together, we’ve unlocked massive throughput gains for the MI300 and the next-generation MI355X series.
We recently co-authored a comprehensive technical deep-dive on the official AMD ROCm blog detailing the internal orchestration of vLLM’s 7 new attention backends.
Key Performance Breakthroughs:
- Massive TPS Gains:
ROCM_AITER_FAdelivers 2.7-4.4x higher TPS than legacy ROCM_ATTN on MHA models. - DeepSeek/MLA Optimization:
ROCM_AITER_MLAdelivers 1.2-1.5x higher TPS on complex MLA architectures via our assembly decode kernels. - Seamless Scaling: Proven performance scaling across MI300X, MI325X, and MI355X hardware.
- Explicit 3-Path Routing: A deliberate architectural shift to route workloads at the software layer for prefill, extend, and decode paths.
This isn't just porting; it is purpose-built, infrastructure-grade optimization.