Skip to content
KernelIndex
Search⌘K
Sign in

Decoder layer pre post norm residual

21 eligible runs
rmsnorm

Complete decoder layer with pre-normalization, attention, MLP, and residual connections. Pattern: x = x + attention(norm(x)), x = x + mlp(norm(x)). This is the fundamental building block executed N times per forward pass.

Fastest reported
1.39msmean · 57% of SOL
RIAC as wellLicense unknown · unknown

Reported evidence · last observed 2026-08-14. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 96 · kv_hidden = 768 · qkv_hidden = 6144 · hidden_size = 6144 · half_head_dim = 48 · intermediate_size = 19648 · num_attention_heads = 64 · num_key_value_heads = 816 cases
batch_size
seq_len
dtype
1128bf1611024bf162512bf1621024bf1622048bf164128bf164256bf164293bf164512bf168128bf168256bf168512bf1616128bf1616449bf1632256bf1664128bf16
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 11.39 msRIAC as well14NVIDIA B200 · env 21.39 msdoubleAI7

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 14 results · last observed 2026-08-21Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
RIAC as well
1.39ms57% SOL
1.00×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Hyra
1.40ms60% SOL
1.01×
Reported · license unknown · no source
2026-07-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
ac4k
1.42ms56% SOL
1.02×
Reported · license unknown · no source
2026-07-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Geometric
1.42ms56% SOL
1.02×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Geometric
1.43ms56% SOL
1.03×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Recursive
1.45ms55% SOL
1.04×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Amir M. Mir | SF Tensor
1.46ms54% SOL
1.05×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Geometric
1.47ms54% SOL
1.06×
Reported · license unknown · no source
2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Geometric
1.48ms53% SOL
1.07×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
doubleAI
1.49ms53% SOL
1.07×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Proud Wren @ NVIDIA
1.50ms53% SOL
1.08×
Reported · license unknown · no source
2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
1.51ms52% SOL
1.09×
Reported · license unknown · no source
2026-07-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Q H
1.59ms49% SOL
1.14×
Reported · license unknown · no source
2026-08-08

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Cursor
1.88ms40% SOL
1.35×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
RIAC as well
unknown
1.39ms
1.00×
Reported
license unknown · no source
ac4k
unknown
1.42ms
1.02×
Reported
license unknown · no source
Geometric
unknown
1.42ms
1.02×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.42ms
1.02×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
1.50ms
1.08×
Reported
license unknown · no source
Q H
unknown
1.59ms
1.15×
Reported
license unknown · no source
Cursor
unknown
1.80ms
1.30×
Reported
license unknown · no source
Geometric
unknown
1.43ms
1.03×
Reported
license unknown · no source
Show all 21 implementations ›
Recursive
unknown
1.45ms
1.04×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.46ms
1.05×
Reported
license unknown · no source
Geometric
unknown
1.47ms
1.06×
Reported
license unknown · no source
doubleAI
unknown
1.49ms
1.08×
Reported
license unknown · no source
Geometric
unknown
1.48ms
1.07×
Reported
license unknown · no source
Geometric
unknown
1.51ms
1.09×
Reported
license unknown · no source
doubleAI
unknown
1.58ms
1.14×
Reported
license unknown · no source
Cursor
unknown
1.88ms
1.36×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.24ms
1.61×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.49ms
1.79×
Reported
license unknown · no source
Hyra
unknown
1.40ms
1.01×
Reported
license unknown · no source
doubleAI
unknown
1.39ms
1.00×
Reported
license unknown · no source
Recursive
unknown
1.39ms
1.00×
Reported
license unknown · no source

Semantics

Inputs and outputs
hidden_statesbf16 [batch_size, seq_len, hidden_size]
cosbf16 [batch_size, seq_len, half_head_dim]
sinbf16 [batch_size, seq_len, half_head_dim]
attention_maskbf16 [batch_size, 1, seq_len, seq_len]
input_layernorm_weightbf16 [hidden_size]
q_proj_weightbf16 [qkv_hidden, hidden_size]
k_proj_weightbf16 [kv_hidden, hidden_size]
v_proj_weightbf16 [kv_hidden, hidden_size]
o_proj_weightbf16 [hidden_size, qkv_hidden]
post_attention_layernorm_weightbf16 [hidden_size]
gate_proj_weightbf16 [intermediate_size, hidden_size]
up_proj_weightbf16 [intermediate_size, hidden_size]
down_proj_weightbf16 [hidden_size, intermediate_size]
rms_norm_epsfp32 scalar
outputbf16 [batch_size, seq_len, hidden_size]
Axes and behavior
seq_lenvariable
head_dimconstant = 96
kv_hiddenkv_hidden = num_key_value_heads * head_dim
batch_sizevariable
qkv_hiddenqkv_hidden = num_attention_heads * head_dim
hidden_sizeconstant = 6144
half_head_dimhalf_head_dim = head_dim // 2
intermediate_sizeconstant = 19648
num_attention_headsconstant = 64
num_key_value_headsconstant = 8
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias020_decoder_layer_pre_post_norm_residualmodelreka-flash-3sha256ed519dfa08b5…
Sources: NVIDIA SOL-ExecBench (2026-08-21)last observed 2026-08-21How records are decidedJSON