Grouped query attention with RoPE and QK norm
Grouped Query Attention with RoPE and Query/Key RMSNorm. Features 32 query heads, 8 KV heads (4x grouping), per-head RMSNorm on Q/K before RoPE application, and causal attention. This is the core attention mechanism from Qwen3-8B.
Reported evidence · last observed 2026-08-07. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyone = 1 · head_dim = 128 · hidden_size = 4096 · q_proj_out_dim = 4096 · kv_proj_out_dim = 1024 · num_attention_heads = 32 · num_key_value_heads = 8 · num_key_value_groups = 416 cases
Not measured on H100 for this workload. Challenges →
1228.4µs60% SOL1.00×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
2230.1µs59% SOL1.01×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
3236.6µs58% SOL1.04×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
4237.7µs58% SOL1.04×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
5241.9µs58% SOL1.06×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
6242.1µs59% SOL1.06×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
7242.2µs59% SOL1.06×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
8243.4µs55% SOL1.07×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
9243.8µs58% SOL1.07×Reported · license unknown · no source2026-07-11
Measured exactly what you asked. Reported by source; not independently reproduced.
10251.9µs53% SOL1.10×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
11256.4µs54% SOL1.12×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
12257.1µs54% SOL1.13×Reported · license unknown · no source2026-07-15
Measured exactly what you asked. Reported by source; not independently reproduced.
13260.4µs52% SOL1.14×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
14267.8µs52% SOL1.17×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
15272.6µs51% SOL1.19×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
16278.3µs48% SOL1.22×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
17288.1µs50% SOL1.26×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
18293.7µs47% SOL1.29×Reported · license unknown · no source2026-07-27
Measured exactly what you asked. Reported by source; not independently reproduced.
19311.5µs47% SOL1.36×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
20314.8µs45% SOL1.38×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
21316.8µs46% SOL1.39×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
22339.5µs43% SOL1.49×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
23434.3µs36% SOL1.90×Reported · license unknown · no source2026-07-15
Measured exactly what you asked. Reported by source; not independently reproduced.
24554.5µs28% SOL2.43×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
25593.6µs29% SOL2.60×Reported · license unknown · no source2026-08-02
Measured exactly what you asked. Reported by source; not independently reproduced.
26863.4µs21% SOL3.78×Reported · license unknown · no source2026-08-11
Measured exactly what you asked. Reported by source; not independently reproduced.