ScaledDotProductAttention
4 eligible runs
attention
KernelBench level1 problem 97: ScaledDotProductAttention. The computation is the reference PyTorch module's forward pass; the output shape follows the module (mirrored as each implementation's source).
Fastest reported
8.15ms±0.03 · mean of 100 · 1.04× faster than baseline
torch.compile (inductor)PyTorch · MIT · python
Reported evidence · last observed 2026-03-05. Reported by source; not independently reproduced.
Current records
HardwareBest knownImplementationRuns
NVIDIA H100 · env 18.15 mstorch.compile (inductor)2NVIDIA H100 · env 213.2 mstorch.compile (inductor)2Not measured on B200 for this workload. Challenges →
Source-native comparison · GPU NVIDIA H100 · Workload num_heads = 32 · batch_size = 32 · sequence_length = 512 · embedding_dimension = 1024 · fp32 · Protocol KernelBench timing scripts · mean · 2 results · last observed 2026-03-05Record history →
Estimated floor 1.92 ms · record 4.24× above itestimate, not evidence ›
DRAM 1.92 ms · bandwidth-bound on H100 SXM
every declared tensor crosses HBM exactly once (3,350 GB/s, H100 SXM datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1torch.compile (inductor)PyTorch8.15ms±0.031.00×Reported · MIT · source2026-03-05stale
1.04× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.
source mirroredMITno install recipeView source →Run detail →
28.45ms±0.071.04×Reported · MIT · source2026-03-05stale
Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.
source mirroredMITno install recipeView source →Run detail →
Implementations
Implementation
Runtime
Best latency
Evidence
Availability
Semantics
Inputs and outputs
qfp32 [batch_size, num_heads, sequence_length, embedding_dimension]
kfp32 [batch_size, num_heads, sequence_length, embedding_dimension]
vfp32 [batch_size, num_heads, sequence_length, embedding_dimension]
outfloat [out]
Axes and behavior
outvariable
num_headsvariable
batch_sizevariable
sequence_lengthvariable
embedding_dimensionvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha25626510ffb9022…
Sources: KernelBench baseline timings (2026-03-05) · MITlast observed 2026-03-05How records are decidedJSON