Skip to content
KernelIndex
Search⌘K
Sign in

Language model decoder prenorm attention FFN residual backward

26 eligible runs
mlp

Backward pass for complete decoder transformer layer with pre-norm attention and FFN with residual connections. Computes gradients for all weights and input through RMSNorm, GQA attention with RoPE, and SwiGLU FFN.

Fastest reported
1.50msmean · 64% of SOL
Proud Wren @ NVIDIALicense unknown · unknown

Reported evidence · last observed 2026-07-12. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 160 · num_heads = 32 · hidden_size = 5120 · num_kv_heads = 8 · half_head_dim = 80 · intermediate_size = 14336 · num_heads_x_head_dim = 5120 · num_kv_heads_x_head_dim = 128016 cases
seq_len
batch_size
dtype
1281bf16/fp321284bf16/fp3212832bf16/fp322112bf16/fp322561bf16/fp322564bf16/fp322568bf16/fp3225632bf16/fp322934bf16/fp323738bf16/fp3244916bf16/fp325122bf16/fp326131bf16/fp326912bf16/fp3210241bf16/fp3210242bf16/fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 11.50 msProud Wren @ NVIDIA20NVIDIA B200 · env 21.46 msdoubleAI6

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 20 results · last observed 2026-08-22Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Proud Wren @ NVIDIA
1.50ms64% SOL
1.00×
Reported · license unknown · no source
2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
RIAC as well
1.54ms64% SOL
1.03×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
RIAC as well
1.54ms64% SOL
1.03×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Amir M. Mir | SF Tensor
1.58ms61% SOL
1.06×
Reported · license unknown · no source
2026-08-05

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
ac4k
1.60ms61% SOL
1.07×
Reported · license unknown · no source
2026-07-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Gleaming Coyote
1.62ms60% SOL
1.08×
Reported · license unknown · no source
2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Rapid Tiger
1.63ms60% SOL
1.08×
Reported · license unknown · no source
2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Electric Cheetah
1.63ms60% SOL
1.08×
Reported · license unknown · no source
2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Geometric
1.75ms56% SOL
1.17×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Hyra
1.82ms55% SOL
1.21×
Reported · license unknown · no source
2026-07-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Geometric
1.83ms54% SOL
1.22×
Reported · license unknown · no source
2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
1.86ms53% SOL
1.24×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Amir M. Mir | SF Tensor
1.89ms53% SOL
1.26×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
1.89ms53% SOL
1.26×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
1.90ms52% SOL
1.27×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
1.91ms52% SOL
1.27×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Geometric
1.93ms52% SOL
1.29×
Reported · license unknown · no source
2026-07-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Geometric
1.93ms52% SOL
1.29×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Geometric
4.09ms27% SOL
2.73×
Reported · license unknown · no source
2026-07-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
20
doubleAI
4.31ms31% SOL
2.88×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Amir M. Mir | SF Tensor
unknown
1.58ms
1.08×
Reported
license unknown · no source
ac4k
unknown
1.60ms
1.09×
Reported
license unknown · no source
Gleaming Coyote
unknown
1.62ms
1.11×
Reported
license unknown · no source
Rapid Tiger
unknown
1.63ms
1.11×
Reported
license unknown · no source
Electric Cheetah
unknown
1.63ms
1.11×
Reported
license unknown · no source
Recursive
unknown
1.63ms
1.12×
Reported
license unknown · no source
Geometric
unknown
1.75ms
1.20×
Reported
license unknown · no source
Hyra
unknown
1.82ms
1.24×
Reported
license unknown · no source
Show all 26 implementations ›
Cursor
unknown
2.02ms
1.38×
Reported
license unknown · no source
RIAC as well
unknown
1.54ms
1.05×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.64ms
1.12×
Reported
license unknown · no source
doubleAI
unknown
1.81ms
1.24×
Reported
license unknown · no source
Geometric
unknown
1.83ms
1.25×
Reported
license unknown · no source
Geometric
unknown
1.86ms
1.27×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.89ms
1.29×
Reported
license unknown · no source
Geometric
unknown
1.89ms
1.29×
Reported
license unknown · no source
Geometric
unknown
1.90ms
1.30×
Reported
license unknown · no source
Geometric
unknown
1.91ms
1.31×
Reported
license unknown · no source
Geometric
unknown
1.93ms
1.32×
Reported
license unknown · no source
Geometric
unknown
1.93ms
1.32×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.86ms
1.95×
Reported
license unknown · no source
doubleAI
unknown
4.31ms
2.95×
Reported
license unknown · no source
Geometric
unknown
4.09ms
2.79×
Reported
license unknown · no source
doubleAI
unknown
1.46ms
1.00×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
1.50ms
1.02×
Reported
license unknown · no source
RIAC as well
unknown
1.54ms
1.05×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputbf16 [batch_size, seq_len, hidden_size]
residualbf16 [batch_size, seq_len, hidden_size]
attn_inputbf16 [batch_size, seq_len, hidden_size]
query_statesbf16 [batch_size, num_heads, seq_len, head_dim]
key_statesbf16 [batch_size, num_kv_heads, seq_len, head_dim]
value_statesbf16 [batch_size, num_kv_heads, seq_len, head_dim]
query_states_rotatedbf16 [batch_size, num_heads, seq_len, head_dim]
key_states_rotatedbf16 [batch_size, num_kv_heads, seq_len, head_dim]
key_states_repeatedbf16 [batch_size, num_heads, seq_len, head_dim]
value_states_repeatedbf16 [batch_size, num_heads, seq_len, head_dim]
cosbf16 [batch_size, seq_len, head_dim]
sinbf16 [batch_size, seq_len, head_dim]
attn_weightsbf16 [batch_size, num_heads, seq_len, seq_len]
attn_outputbf16 [batch_size, seq_len, num_heads_x_head_dim]
residual2bf16 [batch_size, seq_len, hidden_size]
ffn_inputbf16 [batch_size, seq_len, hidden_size]
gatebf16 [batch_size, seq_len, intermediate_size]
upbf16 [batch_size, seq_len, intermediate_size]
silu_upbf16 [batch_size, seq_len, intermediate_size]
swiglu_outputbf16 [batch_size, seq_len, intermediate_size]
input_ln_weightbf16 [hidden_size]
q_weightbf16 [num_heads_x_head_dim, hidden_size]
k_weightbf16 [num_kv_heads_x_head_dim, hidden_size]
v_weightbf16 [num_kv_heads_x_head_dim, hidden_size]
o_weightbf16 [hidden_size, num_heads_x_head_dim]
post_attn_ln_weightbf16 [hidden_size]
gate_weightbf16 [intermediate_size, hidden_size]
up_weightbf16 [intermediate_size, hidden_size]
down_weightbf16 [hidden_size, intermediate_size]
variance1fp32 [batch_size, seq_len, 1]
variance2fp32 [batch_size, seq_len, 1]
hidden_states_normalized1fp32 [batch_size, seq_len, hidden_size]
hidden_states_normalized2fp32 [batch_size, seq_len, hidden_size]
epsfp32 scalar
grad_inputbf16 [batch_size, seq_len, hidden_size]
grad_input_ln_weightfp32 [hidden_size]
grad_q_weightbf16 [num_heads_x_head_dim, hidden_size]
grad_k_weightbf16 [num_kv_heads_x_head_dim, hidden_size]
grad_v_weightbf16 [num_kv_heads_x_head_dim, hidden_size]
grad_o_weightbf16 [hidden_size, num_heads_x_head_dim]
grad_post_attn_ln_weightfp32 [hidden_size]
grad_gate_weightbf16 [intermediate_size, hidden_size]
grad_up_weightbf16 [intermediate_size, hidden_size]
grad_down_weightbf16 [hidden_size, intermediate_size]
Axes and behavior
seq_lenvariable
head_dimconstant = 160
num_headsconstant = 32
batch_sizevariable
hidden_sizeconstant = 5120
num_kv_headsconstant = 8
half_head_dimhalf_head_dim = head_dim // 2
intermediate_sizeconstant = 14336
num_heads_x_head_dimnum_heads_x_head_dim = num_heads * head_dim
num_kv_heads_x_head_dimnum_kv_heads_x_head_dim = num_kv_heads * head_dim
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias056_language_model_decoder_prenorm_attention_ffn_residual_backwardmodelvoxtral-small-24b-2507sha256cad6aabd8b0a…
Sources: NVIDIA SOL-ExecBench (2026-08-22)last observed 2026-08-22How records are decidedJSON