Attention weights matmul with value projection
Fused attention weights @ value matmul with transpose and reshape. Performs attn_weights @ value_states followed by transpose from (batch, num_heads, seq_len, head_dim) to (batch, seq_len, num_heads, head_dim) and reshape to (batch, seq_len, hidden_size). This is the second major matmul in attention that combines attention weights with value states, appearing 64 times per forward pass.
Reported evidence · last observed 2026-08-17. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhead_dim = 128 · hidden_size = 5120 · num_attention_heads = 4016 cases
Not measured on H100 for this workload. Challenges →
123.6µs75% SOL1.00×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
227.7µs72% SOL1.18×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
328.0µs71% SOL1.19×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
429.4µs70% SOL1.25×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
530.0µs67% SOL1.27×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
630.9µs66% SOL1.31×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
731.4µs65% SOL1.33×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
831.4µs65% SOL1.33×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
931.9µs65% SOL1.35×Reported · license unknown · no source2026-07-15
Measured exactly what you asked. Reported by source; not independently reproduced.
1032.4µs64% SOL1.37×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
1133.5µs64% SOL1.42×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
1233.5µs64% SOL1.42×Reported · license unknown · no source2026-07-12
Measured exactly what you asked. Reported by source; not independently reproduced.
1334.4µs63% SOL1.46×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1434.8µs63% SOL1.48×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1538.7µs60% SOL1.64×Reported · license unknown · no source2026-07-20
Measured exactly what you asked. Reported by source; not independently reproduced.
1646.6µs53% SOL1.98×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.