Skip to content
KernelIndex
Search⌘K
Sign in

Fused GQA QKV RoPE QK norm backward

34 eligible runs
rmsnorm

Backward pass for fused GQA QKV projection with RoPE and QK normalization. Computes gradients through RMS normalization, rotary position embedding, reshape/transpose operations, and linear projection.

Fastest reported
156.9µsmean · 95% of SOL
RIAC as wellLicense unknown · unknown

Reported evidence · last observed 2026-08-21. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 128 · qkv_size = 6144 · rstd_dim = 1 · num_heads = 32 · hidden_size = 4096 · num_kv_heads = 8 · num_kv_groups = 416 cases
seq_len
batch_size
dtype
1281bf161284bf1612816bf1612832bf1612864bf161311bf162561bf162568bf1625616bf1625632bf162932bf165121bf165414bf1610242bf1615711bf1620482bf16
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 1156.9 µsRIAC as well19NVIDIA B200 · env 2159.8 µsDatabricks15

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 19 results · last observed 2026-08-23Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
RIAC as well
156.9µs95% SOL
1.00×
Reported · license unknown · no source
2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Databricks
168.4µs94% SOL
1.07×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Hyra
171.9µs94% SOL
1.10×
Reported · license unknown · no source
2026-07-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Geometric
174.4µs94% SOL
1.11×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Recursive
174.5µs94% SOL
1.11×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
doubleAI
179.6µs93% SOL
1.14×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Amir M. Mir | SF Tensor
189.2µs93% SOL
1.21×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
OpenKernels-Ruibo
196.4µs92% SOL
1.25×
Reported · license unknown · no source
2026-07-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Geometric
197.6µs92% SOL
1.26×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
OpenKernels-Ruibo
198.4µs92% SOL
1.26×
Reported · license unknown · no source
2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
ac4k
201.1µs92% SOL
1.28×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
TH3CHARLie | TileGym
216.3µs91% SOL
1.38×
Reported · license unknown · no source
2026-07-28

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Geometric
219.1µs92% SOL
1.40×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
226.7µs91% SOL
1.44×
Reported · license unknown · no source
2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
233.5µs91% SOL
1.49×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
238.8µs90% SOL
1.52×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Geometric
246.5µs90% SOL
1.57×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
OpenKernels-Ruibo
410.4µs78% SOL
2.62×
Reported · license unknown · no source
2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Cursor
612.3µs72% SOL
3.90×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
RIAC as well
unknown
156.9µs
1.00×
Reported
license unknown · no source
doubleAI
unknown
163.3µs
1.04×
Reported
license unknown · no source
Recursive
unknown
166.0µs
1.06×
Reported
license unknown · no source
Geometric
unknown
174.4µs
1.11×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
179.3µs
1.14×
Reported
license unknown · no source
ac4k
unknown
190.8µs
1.22×
Reported
license unknown · no source
OpenKernels-Ruibo
unknown
196.4µs
1.25×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
216.3µs
1.38×
Reported
license unknown · no source
Show all 34 implementations ›
Argus
unknown
245.6µs
1.57×
Reported
license unknown · no source
Cursor
unknown
577.1µs
3.68×
Reported
license unknown · no source
Zerg AI
unknown
1.46ms
9.33×
Reported
license unknown · no source
doubleAI
unknown
171.0µs
1.09×
Reported
license unknown · no source
Databricks
unknown
177.0µs
1.13×
Reported
license unknown · no source
Recursive
unknown
174.5µs
1.11×
Reported
license unknown · no source
doubleAI
unknown
179.6µs
1.14×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
189.2µs
1.21×
Reported
license unknown · no source
Geometric
unknown
197.6µs
1.26×
Reported
license unknown · no source
ac4k
unknown
201.1µs
1.28×
Reported
license unknown · no source
OpenKernels-Ruibo
unknown
198.4µs
1.26×
Reported
license unknown · no source
Geometric
unknown
219.1µs
1.40×
Reported
license unknown · no source
Geometric
unknown
226.7µs
1.44×
Reported
license unknown · no source
Geometric
unknown
233.5µs
1.49×
Reported
license unknown · no source
Geometric
unknown
238.8µs
1.52×
Reported
license unknown · no source
Geometric
unknown
246.5µs
1.57×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
260.5µs
1.66×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
409.4µs
2.61×
Reported
license unknown · no source
OpenKernels-Ruibo
unknown
410.4µs
2.62×
Reported
license unknown · no source
Cursor
unknown
612.3µs
3.90×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.32ms
8.42×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.32ms
8.44×
Reported
license unknown · no source
Zerg AI
unknown
1.47ms
9.36×
Reported
license unknown · no source
Databricks
unknown
159.8µs
1.02×
Reported
license unknown · no source
Hyra
unknown
171.9µs
1.10×
Reported
license unknown · no source
Databricks
unknown
168.4µs
1.07×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_querybf16 [batch_size, num_heads, seq_len, head_dim]
grad_keybf16 [batch_size, num_kv_heads, seq_len, head_dim]
grad_valuebf16 [batch_size, num_kv_heads, seq_len, head_dim]
hidden_statesbf16 [batch_size, seq_len, hidden_size]
cosbf16 [batch_size, seq_len, head_dim]
sinbf16 [batch_size, seq_len, head_dim]
qkv_weightbf16 [qkv_size, hidden_size]
q_norm_weightbf16 [head_dim]
k_norm_weightbf16 [head_dim]
query_pre_normbf16 [batch_size, num_heads, seq_len, head_dim]
key_pre_normbf16 [batch_size, num_kv_heads, seq_len, head_dim]
q_rstdbf16 [batch_size, num_heads, seq_len, rstd_dim]
k_rstdbf16 [batch_size, num_kv_heads, seq_len, rstd_dim]
epsfp32 scalar
grad_hidden_statesbf16 [batch_size, seq_len, hidden_size]
grad_cosbf16 [batch_size, seq_len, head_dim]
grad_sinbf16 [batch_size, seq_len, head_dim]
grad_qkv_weightbf16 [qkv_size, hidden_size]
grad_q_norm_weightbf16 [head_dim]
grad_k_norm_weightbf16 [head_dim]
Axes and behavior
seq_lenvariable
head_dimconstant = 128
qkv_sizeconstant = 6144
rstd_dimconstant = 1
num_headsconstant = 32
batch_sizevariable
hidden_sizeconstant = 4096
num_kv_headsconstant = 8
num_kv_groupsconstant = 4
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias091_fused_gqa_qkv_rope_qk_norm_backwardmodelhunyuanimage-3-0sha2566698161c6c96…
Sources: NVIDIA SOL-ExecBench (2026-08-23)last observed 2026-08-23How records are decidedJSON