Skip to content
KernelIndex
Search⌘K
Sign in

Multi head QKV projection with RoPE backward

29 eligible runs
rope

Backward pass for multi-head QKV projection with RoPE. Computes gradients for hidden_states, q_weight, k_weight, and v_weight given gradients of query, key, and value states.

Fastest reported
81.9µsmean · 66% of SOL
RecursiveLicense unknown · unknown

Reported evidence · last observed 2026-07-10. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 96 · num_heads = 24 · q_out_dim = 2304 · kv_out_dim = 384 · hidden_size = 1536 · num_kv_heads = 4 · half_head_dim = 4816 cases
seq_len
batch_size
dtype
1282fp3212816fp3212832fp3212864fp322564fp322568fp322932fp325128fp325414fp3210242fp3210244fp3213212fp3220481fp3220484fp3240961fp3281921fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 181.9 µsRecursive20NVIDIA B200 · env 277.2 µsRecursive9

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 20 results · last observed 2026-08-24Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Recursive
81.9µs66% SOL
1.00×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
RIAC as well
82.0µs67% SOL
1.00×
Reported · license unknown · no source
2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
RIAC as well
82.6µs66% SOL
1.01×
Reported · license unknown · no source
2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Amir M. Mir | SF Tensor
83.6µs66% SOL
1.02×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
ac4k
84.7µs66% SOL
1.03×
Reported · license unknown · no source
2026-08-03

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Hyra
85.2µs65% SOL
1.04×
Reported · license unknown · no source
2026-07-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Geometric
88.4µs64% SOL
1.08×
Reported · license unknown · no source
2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Amir M. Mir | SF Tensor
88.9µs64% SOL
1.08×
Reported · license unknown · no source
2026-08-04

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
doubleAI
90.6µs63% SOL
1.11×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Geometric
91.6µs63% SOL
1.12×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Amir M. Mir | SF Tensor
95.6µs61% SOL
1.17×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
100.8µs60% SOL
1.23×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Proud Wren @ NVIDIA
105.1µs59% SOL
1.28×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Lunar Wolf
105.4µs59% SOL
1.29×
Reported · license unknown · no source
2026-07-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
112.0µs56% SOL
1.37×
Reported · license unknown · no source
2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
127.5µs52% SOL
1.56×
Reported · license unknown · no source
2026-08-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Geometric
158.9µs45% SOL
1.94×
Reported · license unknown · no source
2026-07-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Cursor
168.0µs44% SOL
2.05×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Geometric
286.0µs29% SOL
3.49×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
20
Geometric
483.8µs19% SOL
5.91×
Reported · license unknown · no source
2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
RIAC as well
unknown
82.0µs
1.06×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
83.6µs
1.08×
Reported
license unknown · no source
Hyra
unknown
85.2µs
1.10×
Reported
license unknown · no source
doubleAI
unknown
85.6µs
1.11×
Reported
license unknown · no source
Geometric
unknown
88.4µs
1.15×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
103.0µs
1.33×
Reported
license unknown · no source
Lunar Wolf
unknown
105.4µs
1.37×
Reported
license unknown · no source
Cursor
unknown
162.7µs
2.11×
Reported
license unknown · no source
Show all 29 implementations ›
RIAC as well
unknown
82.6µs
1.07×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
88.9µs
1.15×
Reported
license unknown · no source
doubleAI
unknown
90.6µs
1.17×
Reported
license unknown · no source
Geometric
unknown
91.6µs
1.19×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
95.6µs
1.24×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
96.3µs
1.25×
Reported
license unknown · no source
Geometric
unknown
100.8µs
1.31×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
105.1µs
1.36×
Reported
license unknown · no source
doubleAI
unknown
105.0µs
1.36×
Reported
license unknown · no source
Geometric
unknown
112.0µs
1.45×
Reported
license unknown · no source
Geometric
unknown
127.5µs
1.65×
Reported
license unknown · no source
Geometric
unknown
158.9µs
2.06×
Reported
license unknown · no source
Cursor
unknown
168.0µs
2.18×
Reported
license unknown · no source
doubleAI
unknown
170.3µs
2.21×
Reported
license unknown · no source
Geometric
unknown
286.0µs
3.71×
Reported
license unknown · no source
Geometric
unknown
483.8µs
6.27×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.16ms
15.07×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.18ms
15.24×
Reported
license unknown · no source
Recursive
unknown
77.2µs
1.00×
Reported
license unknown · no source
Recursive
unknown
81.9µs
1.06×
Reported
license unknown · no source
ac4k
unknown
84.7µs
1.10×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_queryfp32 [batch_size, num_heads, seq_len, head_dim]
grad_keyfp32 [batch_size, num_kv_heads, seq_len, head_dim]
grad_valuefp32 [batch_size, num_kv_heads, seq_len, head_dim]
hidden_statesfp32 [batch_size, seq_len, hidden_size]
q_weightfp32 [q_out_dim, hidden_size]
k_weightfp32 [kv_out_dim, hidden_size]
v_weightfp32 [kv_out_dim, hidden_size]
cosfp32 [batch_size, seq_len, head_dim]
sinfp32 [batch_size, seq_len, head_dim]
grad_hidden_statesfp32 [batch_size, seq_len, hidden_size]
grad_q_weightfp32 [q_out_dim, hidden_size]
grad_k_weightfp32 [kv_out_dim, hidden_size]
grad_v_weightfp32 [kv_out_dim, hidden_size]
Axes and behavior
seq_lenvariable
head_dimconstant = 96
num_headsconstant = 24
q_out_dimq_out_dim = num_heads * head_dim
batch_sizevariable
kv_out_dimkv_out_dim = num_kv_heads * head_dim
hidden_sizeconstant = 1536
num_kv_headsconstant = 4
half_head_dimhalf_head_dim = head_dim // 2
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias064_multi_head_qkv_projection_with_rope_backwardmodelparakeet-tdt-0-6b-v3sha256e810e0e29d64…
Sources: NVIDIA SOL-ExecBench (2026-08-24)last observed 2026-08-24How records are decidedJSON