Attention output projection with residual
Fused attention output projection (o_proj) with residual addition. After computing attention outputs, this operation performs the final linear projection and adds the residual connection. The projection is a matmul (hidden_size x hidden_size) fused with elementwise residual add to eliminate intermediate memory traffic.
Reported evidence · last observed 2026-06-26. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 256016 cases
Not measured on H100 for this workload. Challenges →
134.6µs52% SOL1.00×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
235.2µs51% SOL1.02×Reported · license unknown · no source2026-07-07
Measured exactly what you asked. Reported by source; not independently reproduced.
335.3µs51% SOL1.02×Reported · license unknown · no source2026-07-07
Measured exactly what you asked. Reported by source; not independently reproduced.
435.4µs51% SOL1.02×Reported · license unknown · no source2026-06-28
Measured exactly what you asked. Reported by source; not independently reproduced.
535.6µs51% SOL1.03×Reported · license unknown · no source2026-07-05
Measured exactly what you asked. Reported by source; not independently reproduced.
635.8µs50% SOL1.04×Reported · license unknown · no source2026-07-07
Measured exactly what you asked. Reported by source; not independently reproduced.
735.9µs50% SOL1.04×Reported · license unknown · no source2026-07-04
Measured exactly what you asked. Reported by source; not independently reproduced.
835.9µs50% SOL1.04×Reported · license unknown · no source2026-07-07
Measured exactly what you asked. Reported by source; not independently reproduced.
935.9µs50% SOL1.04×Reported · license unknown · no source2026-07-04
Measured exactly what you asked. Reported by source; not independently reproduced.
1036.1µs49% SOL1.04×Reported · license unknown · no source2026-07-04
Measured exactly what you asked. Reported by source; not independently reproduced.
1136.3µs49% SOL1.05×Reported · license unknown · no source2026-04-15
Measured exactly what you asked. Reported by source; not independently reproduced.
1236.7µs48% SOL1.06×Reported · license unknown · no source2026-04-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1336.7µs48% SOL1.06×Reported · license unknown · no source2026-04-15
Measured exactly what you asked. Reported by source; not independently reproduced.
1436.8µs48% SOL1.06×Reported · license unknown · no source2026-04-15
Measured exactly what you asked. Reported by source; not independently reproduced.
1536.8µs48% SOL1.07×Reported · license unknown · no source2026-07-09
Measured exactly what you asked. Reported by source; not independently reproduced.
1636.9µs48% SOL1.07×Reported · license unknown · no source2026-06-28
Measured exactly what you asked. Reported by source; not independently reproduced.
1736.9µs48% SOL1.07×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
1836.9µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
1936.9µs48% SOL1.07×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
2037.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
2137.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
21=37.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked; tied ranks share a number. Reported by source; not independently reproduced.
2237.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
2337.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
2437.0µs48% SOL1.07×Reported · license unknown · no source2026-06-27
Measured exactly what you asked. Reported by source; not independently reproduced.
2537.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
2637.0µs48% SOL1.07×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
2737.0µs48% SOL1.07×Reported · license unknown · no source2026-06-30
Measured exactly what you asked. Reported by source; not independently reproduced.
27=37.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked; tied ranks share a number. Reported by source; not independently reproduced.
2837.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
2937.0µs48% SOL1.07×Reported · license unknown · no source2026-06-26
Measured exactly what you asked. Reported by source; not independently reproduced.
3037.0µs48% SOL1.07×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
3137.1µs48% SOL1.07×Reported · license unknown · no source2026-06-30
Measured exactly what you asked. Reported by source; not independently reproduced.
3237.1µs48% SOL1.07×Reported · license unknown · no source2026-06-29
Measured exactly what you asked. Reported by source; not independently reproduced.
3337.1µs48% SOL1.07×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
3437.2µs48% SOL1.08×Reported · license unknown · no source2026-06-29
Measured exactly what you asked. Reported by source; not independently reproduced.
3537.2µs48% SOL1.08×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
3637.2µs48% SOL1.08×Reported · license unknown · no source2026-07-05
Measured exactly what you asked. Reported by source; not independently reproduced.
3737.3µs47% SOL1.08×Reported · license unknown · no source2026-06-29
Measured exactly what you asked. Reported by source; not independently reproduced.
3837.3µs47% SOL1.08×Reported · license unknown · no source2026-06-29
Measured exactly what you asked. Reported by source; not independently reproduced.
3937.4µs47% SOL1.08×Reported · license unknown · no source2026-06-28
Measured exactly what you asked. Reported by source; not independently reproduced.
4037.4µs47% SOL1.08×Reported · license unknown · no source2026-06-08
Measured exactly what you asked. Reported by source; not independently reproduced.
4137.5µs47% SOL1.09×Reported · license unknown · no source2026-06-30
Measured exactly what you asked. Reported by source; not independently reproduced.
4238.1µs46% SOL1.10×Reported · license unknown · no source2026-07-06
Measured exactly what you asked. Reported by source; not independently reproduced.
4338.1µs46% SOL1.10×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
4438.4µs45% SOL1.11×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
4538.5µs45% SOL1.11×Reported · license unknown · no source2026-06-20
Measured exactly what you asked. Reported by source; not independently reproduced.
4638.7µs45% SOL1.12×Reported · license unknown · no source2026-04-14
Measured exactly what you asked. Reported by source; not independently reproduced.
4738.9µs45% SOL1.12×Reported · license unknown · no source2026-05-20
Measured exactly what you asked. Reported by source; not independently reproduced.
4838.9µs45% SOL1.12×Reported · license unknown · no source2026-05-16
Measured exactly what you asked. Reported by source; not independently reproduced.
4939.2µs44% SOL1.13×Reported · license unknown · no source2026-04-14
Measured exactly what you asked. Reported by source; not independently reproduced.
5039.5µs44% SOL1.14×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
5139.9µs43% SOL1.15×Reported · license unknown · no source2026-04-14
Measured exactly what you asked. Reported by source; not independently reproduced.
5240.8µs42% SOL1.18×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
5341.0µs42% SOL1.19×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
5441.0µs41% SOL1.19×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
5541.5µs41% SOL1.20×Reported · license unknown · no source2026-06-18
Measured exactly what you asked. Reported by source; not independently reproduced.
5642.3µs42% SOL1.22×Reported · license unknown · no source2026-06-29
Measured exactly what you asked. Reported by source; not independently reproduced.
5742.7µs39% SOL1.24×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
5842.7µs39% SOL1.24×Reported · license unknown · no source2026-06-19
Measured exactly what you asked. Reported by source; not independently reproduced.
17 more rows in this comparison group. Open all in search →