FP8 MLA KV compression projection
FP8-quantized Multi-head Latent Attention (MLA) KV compression projection with low-rank bottleneck. Compresses hidden_size (7168) to kv_lora_rank + qk_rope_head_dim (576), applies RMSNorm on compressed representation, then expands to num_heads * (qk_nope_head_dim + v_head_dim) (24576). Uses BlockWise1x128 scaling for activations and BlockWise128x128 scaling for weights.
Reported evidence · last observed 2026-08-22. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencynum_heads = 128 · v_head_dim = 128 · hidden_size = 7168 · kv_lora_rank = 512 · hidden_blocks = 56 · kv_lora_blocks = 4 · kv_a_out_blocks = 4 · kv_a_output_dim = 576 · kv_b_out_blocks = 256 · kv_b_output_dim = 32768 · qk_nope_head_dim = 128 · qk_rope_head_dim = 6416 cases
Not measured on H100 for this workload. Challenges →
1162.4µs89% SOL1.00×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
2182.2µs88% SOL1.12×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
3206.6µs86% SOL1.27×Reported · license unknown · no source2026-07-22
Measured exactly what you asked. Reported by source; not independently reproduced.
4234.6µs84% SOL1.44×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
5255.2µs82% SOL1.57×Reported · license unknown · no source2026-08-02
Measured exactly what you asked. Reported by source; not independently reproduced.
6270.7µs81% SOL1.67×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
7275.0µs81% SOL1.69×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
8278.0µs81% SOL1.71×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
9279.2µs81% SOL1.72×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
10279.4µs81% SOL1.72×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
11280.4µs81% SOL1.73×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
12281.0µs81% SOL1.73×Reported · license unknown · no source2026-08-16
Measured exactly what you asked. Reported by source; not independently reproduced.
13283.4µs80% SOL1.74×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
14289.6µs80% SOL1.78×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
15295.9µs80% SOL1.82×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
16306.6µs79% SOL1.89×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
17320.0µs79% SOL1.97×Reported · license unknown · no source2026-07-18
Measured exactly what you asked. Reported by source; not independently reproduced.
18322.2µs78% SOL1.98×Reported · license unknown · no source2026-08-12
Measured exactly what you asked. Reported by source; not independently reproduced.
19331.0µs77% SOL2.04×Reported · license unknown · no source2026-08-11
Measured exactly what you asked. Reported by source; not independently reproduced.
20332.1µs77% SOL2.04×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
21332.9µs77% SOL2.05×Reported · license unknown · no source2026-08-11
Measured exactly what you asked. Reported by source; not independently reproduced.
22335.5µs77% SOL2.07×Reported · license unknown · no source2026-08-12
Measured exactly what you asked. Reported by source; not independently reproduced.
23335.9µs77% SOL2.07×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
24336.2µs77% SOL2.07×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
25337.7µs77% SOL2.08×Reported · license unknown · no source2026-08-09
Measured exactly what you asked. Reported by source; not independently reproduced.
26343.8µs77% SOL2.12×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
27360.7µs76% SOL2.22×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
28365.4µs76% SOL2.25×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
29376.2µs75% SOL2.32×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
30381.6µs75% SOL2.35×Reported · license unknown · no source2026-08-04
Measured exactly what you asked. Reported by source; not independently reproduced.
31384.1µs75% SOL2.36×Reported · license unknown · no source2026-08-09
Measured exactly what you asked. Reported by source; not independently reproduced.
32387.8µs74% SOL2.39×Reported · license unknown · no source2026-08-08
Measured exactly what you asked. Reported by source; not independently reproduced.
33392.2µs74% SOL2.41×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
34454.4µs71% SOL2.80×Reported · license unknown · no source2026-08-08
Measured exactly what you asked. Reported by source; not independently reproduced.
35685.6µs61% SOL4.22×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
362.09ms33% SOL12.87×Reported · license unknown · no source2026-08-09
Measured exactly what you asked. Reported by source; not independently reproduced.
372.64ms28% SOL16.22×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
382.72ms27% SOL16.77×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
3920.8ms5% SOL128.09×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.