Achieving BF16 Accuracy at FP8 Speeds: The PTPC-FP8 Breakthrough on AMD ROCm
When deploying Large Language Models at scale, infrastructure teams are usually forced into a painful compromise: run in full precision (BF16) and burn through expensive VRAM, or use standard FP8 quantization and watch your model's reasoning capabilities degrade.
That tradeoff is officially obsolete on AMD hardware.
In collaboration with the AMD AI Framework team, we recently detailed a major breakthrough now supported in vLLM (v0.7.3+): PTPC-FP8 Quantization.
PTPC-FP8 (Per-Token-Activation, Per-Channel-Weight FP8) solves the infamous "outlier problem" in LLMs by applying dual-granularity scaling factors. It delivers the massive throughput and memory savings of 8-bit computation while maintaining accuracy that is virtually indistinguishable from BF16.
The Technical Differentiator: Fused Hardware Operations
Granular scaling should be slow—a naive two-step implementation of this math ruins throughput by wasting memory bandwidth.
However, by leveraging AMD ROCm's fused FP8 rowwise scaled GEMM operation on the MI300X architecture, the matrix multiplication and scaling are combined into a single on-chip hardware operation. This eliminates redundant memory writes, resulting in up to a 2.5x kernel speedup over naive implementations.
Verified Benchmark Breakthroughs (Llama-3.1-70B on MI300X)
Our comprehensive testing reveals that PTPC-FP8 is now the definitive standard for deploying LLMs on ROCm:
- 🚀 Zero Throughput Penalty: PTPC-FP8 matches (and even slightly beats by 1.01x) the raw requests-per-second throughput of standard, lower-accuracy per-tensor FP8.
- 🧠 Preserved Reasoning (GSM8K): On complex mathematical reasoning benchmarks, PTPC-FP8 achieved 87.3% strict-match accuracy on the 70B model, significantly outperforming standard FP8 and even edging out the BF16 baseline (86.3%).
- 📉 Minimal Perplexity Degradation: On the 8B model, PTPC-FP8 showed a mere 0.86% degradation from the full-precision baseline, preserving the model's core predictive power.
Zero-Friction Deployment
You do not need to pre-quantize your models or rewrite your infrastructure. PTPC-FP8 applies weights on-the-fly directly from Hugging Face.
Simply pull the latest vLLM container and append the flag:
--quantization ptpc_fp8
This isn't just an incremental update; it is a fundamental shift in how efficiently you can run frontier models on AMD open-standard hardware.
Read the Full Technical Breakdown and Benchmarks on the vLLM Blog → →