Skip to content
KernelIndex
Search⌘K
Sign in

QK norm scaled dot product attention

28 eligible runs
rmsnorm

Scaled dot-product attention with QK LayerNorm normalization. Query and key tensors are normalized using LayerNorm before computing attention scores. This is used in SD3.5 transformer blocks with 24 heads and 64-dim per head.

Fastest reported
93.4µsmean · 83% of SOL
RIAC as wellLicense unknown · unknown

Reported evidence · last observed 2026-08-21. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencydim = 1536 · head_dim = 64 · num_heads = 24 · dim_times_3 = 460816 cases
seq_len
batch_size
dtype
1281fp321284fp322114fp322561fp322564fp322568fp3225616fp322934fp3244916fp325122fp325124fp326138fp3269116fp3210241fp3210244fp3220481fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 193.4 µsRIAC as well17NVIDIA B200 · env 292.9 µsRecursive11

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 17 results · last observed 2026-08-21Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
RIAC as well
93.4µs83% SOL
1.00×
Reported · license unknown · no source
2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
TH3CHARLie | TileGym
94.7µs82% SOL
1.01×
Reported · license unknown · no source
2026-07-28

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Recursive
95.0µs82% SOL
1.02×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Databricks
100.6µs81% SOL
1.08×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Amir M. Mir | SF Tensor
109.6µs80% SOL
1.17×
Reported · license unknown · no source
2026-08-01

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Amir M. Mir | SF Tensor
110.4µs79% SOL
1.18×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
ac4k
112.1µs79% SOL
1.20×
Reported · license unknown · no source
2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
doubleAI
112.2µs79% SOL
1.20×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Geometric
113.0µs79% SOL
1.21×
Reported · license unknown · no source
2026-08-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Geometric
119.6µs78% SOL
1.28×
Reported · license unknown · no source
2026-08-08

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Geometric
124.6µs77% SOL
1.33×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Hyra
128.0µs76% SOL
1.37×
Reported · license unknown · no source
2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Geometric
135.2µs75% SOL
1.45×
Reported · license unknown · no source
2026-08-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
ac4k
142.1µs74% SOL
1.52×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
175.2µs69% SOL
1.87×
Reported · license unknown · no source
2026-08-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Cursor
273.2µs58% SOL
2.92×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Geometric
334.7µs52% SOL
3.58×
Reported · license unknown · no source
2026-07-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Geometric
unknown
113.0µs
1.22×
Reported
license unknown · no source
Hyra
unknown
128.0µs
1.38×
Reported
license unknown · no source
Cursor
unknown
245.2µs
2.64×
Reported
license unknown · no source
Argus
unknown
332.3µs
3.58×
Reported
license unknown · no source
Recursive
unknown
95.0µs
1.02×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
94.7µs
1.02×
Reported
license unknown · no source
Databricks
unknown
96.6µs
1.04×
Reported
license unknown · no source
Recursive
unknown
92.9µs
1.00×
Reported
license unknown · no source
Show all 28 implementations ›
RIAC as well
unknown
93.4µs
1.01×
Reported
license unknown · no source
ac4k
unknown
98.8µs
1.06×
Reported
license unknown · no source
doubleAI
unknown
107.5µs
1.16×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
108.9µs
1.17×
Reported
license unknown · no source
Databricks
unknown
100.6µs
1.08×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
109.6µs
1.18×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
110.4µs
1.19×
Reported
license unknown · no source
doubleAI
unknown
112.2µs
1.21×
Reported
license unknown · no source
ac4k
unknown
112.1µs
1.21×
Reported
license unknown · no source
Geometric
unknown
119.6µs
1.29×
Reported
license unknown · no source
Geometric
unknown
124.6µs
1.34×
Reported
license unknown · no source
doubleAI
unknown
126.2µs
1.36×
Reported
license unknown · no source
Geometric
unknown
135.2µs
1.46×
Reported
license unknown · no source
ac4k
unknown
142.1µs
1.53×
Reported
license unknown · no source
Geometric
unknown
175.2µs
1.89×
Reported
license unknown · no source
Cursor
unknown
273.2µs
2.94×
Reported
license unknown · no source
Geometric
unknown
334.7µs
3.60×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
589.2µs
6.34×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
959.1µs
10.32×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
984.1µs
10.59×
Reported
license unknown · no source

Semantics

Inputs and outputs
hidden_statesfp32 [batch_size, seq_len, dim]
qkv_weightfp32 [dim_times_3, dim]
qkv_biasfp32 [dim_times_3]
q_norm_weightfp32 [head_dim]
q_norm_biasfp32 [head_dim]
k_norm_weightfp32 [head_dim]
k_norm_biasfp32 [head_dim]
out_proj_weightfp32 [dim, dim]
out_proj_biasfp32 [dim]
epsfp32 scalar
outputfp32 [batch_size, seq_len, dim]
Axes and behavior
dimconstant = 1536
seq_lenvariable
head_dimconstant = 64
num_headsconstant = 24
batch_sizevariable
dim_times_3dim_times_3 = dim * 3
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias082_qk_norm_scaled_dot_product_attentionmodelstable-diffusion-3-5-mediumsha256f8e8fbc4b80e…
Sources: NVIDIA SOL-ExecBench (2026-08-21)last observed 2026-08-21How records are decidedJSON