Skip to content
KernelIndex
Search⌘K
Sign in

Fused residual RMS norm backward

37 eligible runs
rmsnorm

Backward pass for fused residual addition and RMSNorm. Computes gradients for hidden_states, residual, and weight given the gradient of the output.

Fastest reported
42.0µsmean · 83% of SOL
RecursiveLicense unknown · unknown

Reported evidence · last observed 2026-06-06. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhidden_size = 256016 cases
seq_len
batch_size
dtype
1281bf16/fp321311bf16/fp322112bf16/fp322562bf16/fp327732bf16/fp328534bf16/fp3299716bf16/fp3210248bf16/fp32108732bf16/fp3212491bf16/fp32165732bf16/fp32172164bf16/fp3218011bf16/fp3219734bf16/fp32409632bf16/fp32819264bf16/fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 142.0 µsRecursive20NVIDIA B200 · env 235.2 µsRIAC as well17

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.0 · mean · 20 results · last observed 2026-08-24Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Recursive
42.0µs83% SOL
1.00×
Reported · license unknown · no source
2026-06-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Databricks
42.3µs83% SOL
1.01×
Reported · license unknown · no source
2026-06-27

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
doubleAI
43.9µs83% SOL
1.04×
Reported · license unknown · no source
2026-06-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
ac4k
45.3µs81% SOL
1.08×
Reported · license unknown · no source
2026-06-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
doubleAI
52.8µs78% SOL
1.26×
Reported · license unknown · no source
2026-05-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Snappy Macaw
58.0µs78% SOL
1.38×
Reported · license unknown · no source
2026-05-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Argus
62.1µs75% SOL
1.48×
Reported · license unknown · no source
2026-07-05

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Amir M. Mir | SF Tensor
62.5µs75% SOL
1.49×
Reported · license unknown · no source
2026-07-05

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Amir M. Mir | SF Tensor
87.0µs66% SOL
2.07×
Reported · license unknown · no source
2026-04-29

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Zerg AI
87.7µs66% SOL
2.09×
Reported · license unknown · no source
2026-07-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
wychi888
88.8µs67% SOL
2.12×
Reported · license unknown · no source
2026-05-28

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Igor Bragaia
89.1µs67% SOL
2.12×
Reported · license unknown · no source
2026-05-28

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
KernelOpt
116.7µs59% SOL
2.78×
Reported · license unknown · no source
2026-04-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Zerg AI
119.1µs59% SOL
2.84×
Reported · license unknown · no source
2026-07-03

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Arcane Vulture
121.3µs60% SOL
2.89×
Reported · license unknown · no source
2026-06-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Gilded Quail
121.5µs60% SOL
2.89×
Reported · license unknown · no source
2026-06-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Cursor
130.1µs58% SOL
3.10×
Reported · license unknown · no source
2026-04-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Amir M. Mir | SF Tensor
131.7µs57% SOL
3.14×
Reported · license unknown · no source
2026-04-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Yuang
134.5µs56% SOL
3.20×
Reported · license unknown · no source
2026-05-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
20
Yuang
163.5µs51% SOL
3.89×
Reported · license unknown · no source
2026-05-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
RIAC as well
unknown
35.2µs
1.00×
Reported
license unknown · no source
doubleAI
unknown
43.6µs
1.24×
Reported
license unknown · no source
Recursive
unknown
41.9µs
1.19×
Reported
license unknown · no source
ac4k
unknown
45.3µs
1.29×
Reported
license unknown · no source
Geometric
unknown
50.4µs
1.43×
Reported
license unknown · no source
Snappy Macaw
unknown
58.0µs
1.65×
Reported
license unknown · no source
Argus
unknown
62.1µs
1.76×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
64.2µs
1.82×
Reported
license unknown · no source
Show all 37 implementations ›
wychi888
unknown
88.8µs
2.52×
Reported
license unknown · no source
Igor Bragaia
unknown
89.1µs
2.53×
Reported
license unknown · no source
Zerg AI
unknown
87.7µs
2.49×
Reported
license unknown · no source
Runic Kingfisher
unknown
106.3µs
3.02×
Reported
license unknown · no source
Arcane Vulture
unknown
121.3µs
3.44×
Reported
license unknown · no source
Gilded Quail
unknown
121.5µs
3.45×
Reported
license unknown · no source
KernelOpt
unknown
116.7µs
3.31×
Reported
license unknown · no source
Cursor
unknown
130.1µs
3.69×
Reported
license unknown · no source
Yuang
unknown
134.5µs
3.82×
Reported
license unknown · no source
Databricks
unknown
42.3µs
1.20×
Reported
license unknown · no source
doubleAI
unknown
43.9µs
1.25×
Reported
license unknown · no source
Recursive
unknown
42.0µs
1.19×
Reported
license unknown · no source
ac4k
unknown
45.8µs
1.30×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
50.4µs
1.43×
Reported
license unknown · no source
Geometric
unknown
51.9µs
1.47×
Reported
license unknown · no source
Geometric
unknown
51.9µs
1.47×
Reported
license unknown · no source
doubleAI
unknown
52.8µs
1.50×
Reported
license unknown · no source
Geometric
unknown
60.5µs
1.72×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
62.5µs
1.77×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
62.6µs
1.78×
Reported
license unknown · no source
Snappy Macaw
unknown
67.6µs
1.92×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
87.0µs
2.47×
Reported
license unknown · no source
Zerg AI
unknown
119.1µs
3.38×
Reported
license unknown · no source
Cursor
unknown
130.8µs
3.71×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
131.7µs
3.74×
Reported
license unknown · no source
Yuang
unknown
163.5µs
4.64×
Reported
license unknown · no source
Hyra
unknown
39.8µs
1.13×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
36.9µs
1.05×
Reported
license unknown · no source
Databricks
unknown
42.3µs
1.20×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputbf16 [batch_size, seq_len, hidden_size]
xfp32 [batch_size, seq_len, hidden_size]
normalizedfp32 [batch_size, seq_len, hidden_size]
rstdfp32 [batch_size, seq_len, 1]
weightfp32 [hidden_size]
grad_hidden_statesbf16 [batch_size, seq_len, hidden_size]
grad_residualbf16 [batch_size, seq_len, hidden_size]
grad_weightfp32 [hidden_size]
Axes and behavior
seq_lenvariable
batch_sizevariable
hidden_sizeconstant = 2560
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias013_fused_residual_rms_norm_backwardmodelqwen3-4b-instruct-2507sha256f3aca4e58b6a…
Sources: NVIDIA SOL-ExecBench (2026-08-24)last observed 2026-08-24How records are decidedJSON