Skip to content
KernelIndex
Search⌘K
Sign in

Fused vision multihead attention with norms backward

36 eligible runs
rmsnorm

Backward pass for fused vision multi-head attention with pre-normalization and residual connection. Computes gradients through: residual connection, dropout, output projection, attention mechanism (softmax, matmul), head split/merge, QKV projection, and LayerNorm.

Fastest reported
608.5µsmean · 91% of SOL
RecursiveLicense unknown · unknown

Reported evidence · last observed 2026-07-10. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyqkv_dim = 3072 · seq_len = 577 · head_dim = 64 · embed_dim = 1024 · num_heads = 1616 cases
batch_size
dtype
1fp322fp323fp325fp327fp3210fp3212fp3214fp3223fp3228fp3232fp3237fp3241fp3247fp3253fp3256fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 1608.5 µsRecursive23NVIDIA B200 · env 2597.6 µsRecursive13

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 23 results · last observed 2026-08-22Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Recursive
608.5µs91% SOL
1.00×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
TH3CHARLie | TileGym
636.7µs91% SOL
1.05×
Reported · license unknown · no source
2026-07-29

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Amir M. Mir | SF Tensor
676.1µs90% SOL
1.11×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
doubleAI
682.7µs90% SOL
1.12×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
ac4k
736.4µs89% SOL
1.21×
Reported · license unknown · no source
2026-07-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Amir M. Mir | SF Tensor
745.5µs89% SOL
1.23×
Reported · license unknown · no source
2026-07-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Hyra
791.1µs88% SOL
1.30×
Reported · license unknown · no source
2026-07-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Proud Wren @ NVIDIA
839.6µs87% SOL
1.38×
Reported · license unknown · no source
2026-07-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
imec.AILabs
860.5µs87% SOL
1.41×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Proud Wren @ NVIDIA
888.3µs87% SOL
1.46×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
imec.AILabs
947.7µs86% SOL
1.56×
Reported · license unknown · no source
2026-07-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
971.0µs86% SOL
1.60×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Amir M. Mir | SF Tensor
999.2µs85% SOL
1.64×
Reported · license unknown · no source
2026-07-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
999.4µs85% SOL
1.64×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
1.09ms84% SOL
1.78×
Reported · license unknown · no source
2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
1.09ms84% SOL
1.79×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
AKO
1.18ms82% SOL
1.93×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Amir M. Mir | SF Tensor
1.46ms79% SOL
2.40×
Reported · license unknown · no source
2026-07-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Geometric
1.56ms78% SOL
2.56×
Reported · license unknown · no source
2026-08-09

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
20
Gilded Quail
1.58ms78% SOL
2.60×
Reported · license unknown · no source
2026-07-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
21
Amir M. Mir | SF Tensor
1.59ms78% SOL
2.62×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
22
Geometric
1.79ms76% SOL
2.95×
Reported · license unknown · no source
2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
23
Geometric
1.81ms75% SOL
2.98×
Reported · license unknown · no source
2026-08-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Recursive
unknown
597.6µs
1.00×
Reported
license unknown · no source
doubleAI
unknown
654.7µs
1.10×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
676.1µs
1.13×
Reported
license unknown · no source
Hyra
unknown
791.1µs
1.32×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
831.6µs
1.39×
Reported
license unknown · no source
imec.AILabs
unknown
860.5µs
1.44×
Reported
license unknown · no source
Geometric
unknown
971.0µs
1.62×
Reported
license unknown · no source
AKO
unknown
1.09ms
1.83×
Reported
license unknown · no source
Show all 36 implementations ›
Gilded Quail
unknown
1.55ms
2.60×
Reported
license unknown · no source
Cursor
unknown
3.85ms
6.45×
Reported
license unknown · no source
Arcane Vulture
unknown
4.91ms
8.22×
Reported
license unknown · no source
Recursive
unknown
608.5µs
1.02×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
745.5µs
1.25×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
839.6µs
1.40×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
888.3µs
1.49×
Reported
license unknown · no source
imec.AILabs
unknown
947.7µs
1.59×
Reported
license unknown · no source
Geometric
unknown
999.4µs
1.67×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
999.2µs
1.67×
Reported
license unknown · no source
Geometric
unknown
1.09ms
1.82×
Reported
license unknown · no source
Geometric
unknown
1.09ms
1.82×
Reported
license unknown · no source
AKO
unknown
1.15ms
1.93×
Reported
license unknown · no source
AKO
unknown
1.18ms
1.97×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.46ms
2.44×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.53ms
2.56×
Reported
license unknown · no source
Gilded Quail
unknown
1.58ms
2.65×
Reported
license unknown · no source
Geometric
unknown
1.56ms
2.61×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.59ms
2.67×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.75ms
2.94×
Reported
license unknown · no source
Geometric
unknown
1.79ms
3.00×
Reported
license unknown · no source
doubleAI
unknown
1.83ms
3.06×
Reported
license unknown · no source
Geometric
unknown
1.81ms
3.03×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.98ms
3.32×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
5.68ms
9.50×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
636.7µs
1.07×
Reported
license unknown · no source
doubleAI
unknown
682.7µs
1.14×
Reported
license unknown · no source
ac4k
unknown
736.4µs
1.23×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputfp32 [batch_size, seq_len, embed_dim]
xfp32 [batch_size, seq_len, embed_dim]
x_meanfp32 [batch_size, seq_len, 1]
x_varfp32 [batch_size, seq_len, 1]
x_normfp32 [batch_size, seq_len, embed_dim]
ln_weightfp32 [embed_dim]
qkv_weightfp32 [qkv_dim, embed_dim]
qfp32 [batch_size, num_heads, seq_len, head_dim]
kfp32 [batch_size, num_heads, seq_len, head_dim]
vfp32 [batch_size, num_heads, seq_len, head_dim]
attn_weightsfp32 [batch_size, num_heads, seq_len, seq_len]
attn_outputfp32 [batch_size, seq_len, embed_dim]
out_weightfp32 [embed_dim, embed_dim]
scalefp32 scalar
norm_epsfp32 scalar
grad_xfp32 [batch_size, seq_len, embed_dim]
grad_qkv_weightfp32 [qkv_dim, embed_dim]
grad_qkv_biasfp32 [qkv_dim]
grad_out_weightfp32 [embed_dim, embed_dim]
grad_out_biasfp32 [embed_dim]
grad_ln_weightfp32 [embed_dim]
grad_ln_biasfp32 [embed_dim]
Axes and behavior
qkv_dimqkv_dim = embed_dim * 3
seq_lenconstant = 577
head_dimconstant = 64
embed_dimconstant = 1024
num_headsconstant = 16
batch_sizevariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias001_fused_vision_multihead_attention_with_norms_backwardmodelmolmo-7b-d-basesha25660ee31285eaf…
Sources: NVIDIA SOL-ExecBench (2026-08-22)last observed 2026-08-22How records are decidedJSON