Attention value projection with transpose
Fused attention value projection with reshape and transpose for GQA attention. Projects hidden states from hidden_size=5120 to num_kv_heads*head_dim=1024, then reshapes and transposes to [batch, num_kv_heads, seq_len, head_dim] format for attention computation.
Reported evidence · last observed 2026-08-22. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhead_dim = 128 · hidden_size = 5120 · num_kv_heads = 8 · kv_projection_size = 102416 cases
Not measured on H100 for this workload. Challenges →
124.9µs51% SOL1.00×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
225.0µs51% SOL1.00×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
325.0µs51% SOL1.00×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
425.4µs51% SOL1.02×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
525.6µs50% SOL1.03×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
625.7µs51% SOL1.03×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
725.7µs51% SOL1.03×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
825.7µs50% SOL1.03×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
926.0µs49% SOL1.04×Reported · license unknown · no source2026-07-31
Measured exactly what you asked. Reported by source; not independently reproduced.
1026.7µs48% SOL1.07×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
1126.8µs48% SOL1.07×Reported · license unknown · no source2026-07-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1226.8µs48% SOL1.07×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1327.0µs48% SOL1.08×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1427.4µs47% SOL1.10×Reported · license unknown · no source2026-08-24
Measured exactly what you asked. Reported by source; not independently reproduced.
1528.4µs45% SOL1.14×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1630.0µs43% SOL1.20×Reported · license unknown · no source2026-08-04
Measured exactly what you asked. Reported by source; not independently reproduced.
1736.8µs36% SOL1.48×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.