MLA fused QKV RoPE split
Fused Multi-head Latent Attention (MLA) QKV projection with RoPE split. Combines query low-rank projection (q_a_proj -> RMSNorm -> q_b_proj) with tensor splitting and key-value joint projection with split into a single operation. Eliminates intermediate memory writes for the latent representations.
Reported evidence · last observed 2026-08-19. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencynum_heads = 128 · q_head_dim = 192 · hidden_size = 7168 · q_lora_rank = 1536 · kv_lora_rank = 512 · q_proj_out_dim = 24576 · kv_proj_out_dim = 576 · qk_nope_head_dim = 128 · qk_rope_head_dim = 6416 cases
Not measured on H100 for this workload. Challenges →
1153.6µs72% SOL1.00×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
2161.7µs69% SOL1.05×Reported · license unknown · no source2026-07-15
Measured exactly what you asked. Reported by source; not independently reproduced.
3162.2µs69% SOL1.06×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
4162.3µs69% SOL1.06×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
5163.8µs68% SOL1.07×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
6164.9µs67% SOL1.07×Reported · license unknown · no source2026-08-24
Measured exactly what you asked. Reported by source; not independently reproduced.
7166.6µs66% SOL1.08×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
8168.6µs66% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
9170.8µs66% SOL1.11×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
10171.0µs65% SOL1.11×Reported · license unknown · no source2026-07-11
Measured exactly what you asked. Reported by source; not independently reproduced.
11172.2µs65% SOL1.12×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
12178.2µs63% SOL1.16×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
13183.3µs62% SOL1.19×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
14183.5µs62% SOL1.19×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
15186.0µs61% SOL1.21×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
16187.9µs61% SOL1.22×Reported · license unknown · no source2026-08-12
Measured exactly what you asked. Reported by source; not independently reproduced.
17188.7µs60% SOL1.23×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
18195.8µs59% SOL1.27×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
19195.9µs59% SOL1.28×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
20196.9µs58% SOL1.28×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
21197.8µs58% SOL1.29×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
22202.6µs57% SOL1.32×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
23226.0µs51% SOL1.47×Reported · license unknown · no source2026-07-21
Measured exactly what you asked. Reported by source; not independently reproduced.
24651.3µs19% SOL4.24×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.