Skip to content
KernelIndex
Search⌘K
Sign in

QK norm RoPE attention core backward

27 eligible runs
rmsnorm

Backward pass for Gemma3 attention with Q/K RMSNorm, RoPE, and soft-capping. Computes gradients for hidden_states, Q/K/V/O projection weights, and Q/K norm weights.

Fastest reported
464.6µsmean · 78% of SOL
AccountForCollegeLicense unknown · unknown

Reported evidence · last observed 2026-08-16. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 128 · q_proj_out = 3072 · hidden_size = 3072 · kv_proj_out = 1024 · num_attention_heads = 24 · num_key_value_heads = 816 cases
batch_size
seq_len
dtype
1128fp321256fp321512fp3212048fp322512fp322541fp3221024fp3222048fp324128fp324256fp3241024fp328128fp328512fp3216256fp3232128fp3264256fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 1464.6 µsAccountForCollege21NVIDIA B200 · env 2631.5 µsdoubleAI6

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 21 results · last observed 2026-08-22Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
AccountForCollege
464.6µs78% SOL
1.00×
Reported · license unknown · no source
2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Amir M. Mir | SF Tensor
549.9µs74% SOL
1.18×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Amir M. Mir | SF Tensor
607.7µs71% SOL
1.31×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Geometric
614.4µs71% SOL
1.32×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Geometric
660.6µs69% SOL
1.42×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
doubleAI
686.4µs68% SOL
1.48×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Geometric
710.0µs67% SOL
1.53×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Geometric
723.4µs67% SOL
1.56×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
ac4k
749.8µs66% SOL
1.61×
Reported · license unknown · no source
2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Recursive
763.4µs65% SOL
1.64×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Proud Wren @ NVIDIA
763.4µs65% SOL
1.64×
Reported · license unknown · no source
2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Amir M. Mir | SF Tensor
793.4µs64% SOL
1.71×
Reported · license unknown · no source
2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Hyra
802.6µs64% SOL
1.73×
Reported · license unknown · no source
2026-07-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
891.7µs61% SOL
1.92×
Reported · license unknown · no source
2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
1.02ms57% SOL
2.19×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
1.26ms51% SOL
2.71×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Proud Wren @ NVIDIA
1.28ms51% SOL
2.75×
Reported · license unknown · no source
2026-08-01

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Geometric
1.47ms46% SOL
3.17×
Reported · license unknown · no source
2026-07-27

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Geometric
1.74ms42% SOL
3.75×
Reported · license unknown · no source
2026-08-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
20
Amir M. Mir | SF Tensor
3.55ms26% SOL
7.64×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
21
Geometric
8.13ms13% SOL
17.51×
Reported · license unknown · no source
2026-07-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
AccountForCollege
unknown
464.6µs
1.00×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
549.9µs
1.18×
Reported
license unknown · no source
Geometric
unknown
614.4µs
1.32×
Reported
license unknown · no source
Recursive
unknown
693.9µs
1.49×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
763.4µs
1.64×
Reported
license unknown · no source
Hyra
unknown
802.6µs
1.73×
Reported
license unknown · no source
Cursor
unknown
1.57ms
3.38×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
607.7µs
1.31×
Reported
license unknown · no source
Show all 27 implementations ›
Geometric
unknown
660.6µs
1.42×
Reported
license unknown · no source
Geometric
unknown
710.0µs
1.53×
Reported
license unknown · no source
Geometric
unknown
723.4µs
1.56×
Reported
license unknown · no source
Recursive
unknown
763.4µs
1.64×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
793.4µs
1.71×
Reported
license unknown · no source
Geometric
unknown
891.7µs
1.92×
Reported
license unknown · no source
Geometric
unknown
1.02ms
2.19×
Reported
license unknown · no source
doubleAI
unknown
1.14ms
2.45×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
1.28ms
2.75×
Reported
license unknown · no source
Geometric
unknown
1.26ms
2.71×
Reported
license unknown · no source
Geometric
unknown
1.47ms
3.17×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.75ms
3.78×
Reported
license unknown · no source
Geometric
unknown
1.74ms
3.75×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.15ms
4.63×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
3.55ms
7.64×
Reported
license unknown · no source
Geometric
unknown
8.13ms
17.51×
Reported
license unknown · no source
doubleAI
unknown
631.5µs
1.36×
Reported
license unknown · no source
doubleAI
unknown
686.4µs
1.48×
Reported
license unknown · no source
ac4k
unknown
749.8µs
1.61×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputfp32 [batch_size, seq_len, hidden_size]
hidden_statesfp32 [batch_size, seq_len, hidden_size]
cosfp32 [batch_size, seq_len, head_dim]
sinfp32 [batch_size, seq_len, head_dim]
q_weightfp32 [q_proj_out, hidden_size]
k_weightfp32 [kv_proj_out, hidden_size]
v_weightfp32 [kv_proj_out, hidden_size]
o_weightfp32 [hidden_size, q_proj_out]
q_norm_weightfp32 [head_dim]
k_norm_weightfp32 [head_dim]
scalingfp32 scalar
attn_logit_softcappingfp32 scalar
rms_norm_epsfp32 scalar
grad_hidden_statesfp32 [batch_size, seq_len, hidden_size]
grad_q_weightfp32 [q_proj_out, hidden_size]
grad_k_weightfp32 [kv_proj_out, hidden_size]
grad_v_weightfp32 [kv_proj_out, hidden_size]
grad_o_weightfp32 [hidden_size, q_proj_out]
grad_q_norm_weightfp32 [head_dim]
grad_k_norm_weightfp32 [head_dim]
Axes and behavior
seq_lenvariable
head_dimconstant = 128
batch_sizevariable
q_proj_outq_proj_out = num_attention_heads * head_dim
hidden_sizeconstant = 3072
kv_proj_outkv_proj_out = num_key_value_heads * head_dim
num_attention_headsconstant = 24
num_key_value_headsconstant = 8
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias037_qk_norm_rope_attention_core_backwardmodelgemma-3-4b-itsha25697102cae35c3…
Sources: NVIDIA SOL-ExecBench (2026-08-22)last observed 2026-08-22How records are decidedJSON