Fused attention QK matmul scale mask softmax backward
Backward pass for fused attention QK matmul with scaling, causal mask, and softmax. Computes gradients for query and key tensors through the chain: softmax_backward -> scale_backward -> matmul_backward.
Reported evidence · last observed 2026-08-22. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhead_dim = 128 · num_heads = 12816 cases
Not measured on H100 for this workload. Challenges →
1137.0µs71% SOL1.00×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
2138.5µs71% SOL1.01×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
3144.5µs69% SOL1.05×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
4145.5µs69% SOL1.06×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
5147.2µs69% SOL1.07×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
6151.1µs68% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
7154.5µs67% SOL1.13×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
8154.6µs68% SOL1.13×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
9154.8µs67% SOL1.13×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
10155.0µs67% SOL1.13×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
11156.2µs67% SOL1.14×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
12156.8µs67% SOL1.15×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
13159.2µs67% SOL1.16×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
14160.5µs67% SOL1.17×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
15162.5µs66% SOL1.19×Reported · license unknown · no source2026-08-24
Measured exactly what you asked. Reported by source; not independently reproduced.
16166.4µs66% SOL1.21×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
17166.4µs66% SOL1.21×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
18176.0µs64% SOL1.28×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
19176.2µs64% SOL1.29×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
20180.3µs63% SOL1.32×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
21185.6µs62% SOL1.36×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
22187.5µs62% SOL1.37×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
23187.5µs62% SOL1.37×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
24221.5µs56% SOL1.62×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
25229.0µs56% SOL1.67×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
26264.6µs51% SOL1.93×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
271.22ms17% SOL8.89×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.