Skip to content
KernelIndex
Search⌘K
Sign in

GQA rotary attention core backward

24 eligible runs
rope

Backward pass for Grouped Query Attention with GLM-style Rotary Position Embeddings. Computes gradients for hidden_states and all projection weights (Q, K, V, O) through the complete attention mechanism including RoPE, GQA head expansion, scaled dot-product attention, and output projection.

Fastest reported
1.45msmean · 67% of SOL
RIAC as wellLicense unknown · unknown

Reported evidence · last observed 2026-08-17. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 128 · hidden_size = 8192 · half_head_dim = 64 · kv_hidden_size = 1024 · num_attention_heads = 64 · num_key_value_heads = 816 cases
seq_len
batch_size
dtype
1281bf16/fp321282bf16/fp3212832bf16/fp3212864bf16/fp322568bf16/fp3225616bf16/fp322932bf16/fp323731bf16/fp324492bf16/fp325411bf16/fp326918bf16/fp3285364bf16/fp329194bf16/fp3210241bf16/fp3210242bf16/fp3220481bf16/fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 11.45 msRIAC as well16NVIDIA B200 · env 21.44 msAmir M. Mir | SF Tensor8

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 16 results · last observed 2026-08-21Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
RIAC as well
1.45ms67% SOL
1.00×
Reported · license unknown · no source
2026-08-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
ac4k
1.50ms65% SOL
1.03×
Reported · license unknown · no source
2026-08-04

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Amir M. Mir | SF Tensor
1.50ms65% SOL
1.03×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Gleaming Coyote
1.50ms65% SOL
1.03×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Recursive
1.57ms63% SOL
1.08×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Geometric
1.69ms58% SOL
1.16×
Reported · license unknown · no source
2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Geometric
1.74ms57% SOL
1.19×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Geometric
1.77ms57% SOL
1.22×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Hyra
1.78ms58% SOL
1.23×
Reported · license unknown · no source
2026-07-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Geometric
1.78ms57% SOL
1.23×
Reported · license unknown · no source
2026-07-28

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Proud Wren @ NVIDIA
1.85ms55% SOL
1.27×
Reported · license unknown · no source
2026-07-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Q H
1.88ms55% SOL
1.29×
Reported · license unknown · no source
2026-08-08

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Cursor
2.29ms48% SOL
1.57×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Proud Wren @ NVIDIA
2.47ms44% SOL
1.70×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
doubleAI
3.50ms38% SOL
2.40×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
3.55ms32% SOL
2.44×
Reported · license unknown · no source
2026-07-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
RIAC as well
unknown
1.45ms
1.01×
Reported
license unknown · no source
Gleaming Coyote
unknown
1.50ms
1.04×
Reported
license unknown · no source
ac4k
unknown
1.50ms
1.04×
Reported
license unknown · no source
doubleAI
unknown
1.54ms
1.07×
Reported
license unknown · no source
Geometric
unknown
1.69ms
1.17×
Reported
license unknown · no source
Hyra
unknown
1.78ms
1.24×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
1.85ms
1.28×
Reported
license unknown · no source
Q H
unknown
1.88ms
1.30×
Reported
license unknown · no source
Show all 24 implementations ›
Cursor
unknown
2.09ms
1.45×
Reported
license unknown · no source
Recursive
unknown
1.57ms
1.09×
Reported
license unknown · no source
doubleAI
unknown
1.66ms
1.15×
Reported
license unknown · no source
Geometric
unknown
1.74ms
1.21×
Reported
license unknown · no source
Geometric
unknown
1.77ms
1.23×
Reported
license unknown · no source
Geometric
unknown
1.78ms
1.24×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.02ms
1.40×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.20ms
1.53×
Reported
license unknown · no source
Cursor
unknown
2.29ms
1.58×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
2.33ms
1.61×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
2.47ms
1.71×
Reported
license unknown · no source
doubleAI
unknown
3.50ms
2.42×
Reported
license unknown · no source
Geometric
unknown
3.55ms
2.46×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.44ms
1.00×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.50ms
1.04×
Reported
license unknown · no source
Recursive
unknown
1.51ms
1.04×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputbf16 [batch_size, seq_len, hidden_size]
hidden_statesbf16 [batch_size, seq_len, hidden_size]
q_weightbf16 [hidden_size, hidden_size]
k_weightbf16 [kv_hidden_size, hidden_size]
v_weightbf16 [kv_hidden_size, hidden_size]
o_weightbf16 [hidden_size, hidden_size]
inv_freqfp32 [half_head_dim]
scalingfp32 scalar
grad_hidden_statesbf16 [batch_size, seq_len, hidden_size]
grad_q_weightbf16 [hidden_size, hidden_size]
grad_k_weightbf16 [kv_hidden_size, hidden_size]
grad_v_weightbf16 [kv_hidden_size, hidden_size]
grad_o_weightbf16 [hidden_size, hidden_size]
Axes and behavior
seq_lenvariable
head_dimconstant = 128
batch_sizevariable
hidden_sizeconstant = 8192
half_head_dimhalf_head_dim = head_dim // 2
kv_hidden_sizekv_hidden_size = num_key_value_heads * head_dim
num_attention_headsconstant = 64
num_key_value_headsconstant = 8
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias028_gqa_rotary_attention_core_backwardmodelernie-4-5-300b-a47b-ptsha256d01a7424f6ac…
Sources: NVIDIA SOL-ExecBench (2026-08-21)last observed 2026-08-21How records are decidedJSON