Skip to content
KernelIndex
Search⌘K
Sign in

Attention with QK norm and RoPE

29 eligible runs
rmsnorm

Gemma3 attention mechanism with query/key RMSNorm normalization, rotary position embeddings (RoPE), grouped-query attention (GQA) with 24 query heads and 8 KV heads, attention logit softcapping at 50.0, and causal masking. Includes Q/K/V/O projections.

Fastest reported
177.5µsmean · 79% of SOL
RecursiveLicense unknown · unknown

Reported evidence · last observed 2026-07-10. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 128 · kv_out_dim = 1024 · hidden_size = 3072 · qkv_out_dim = 3072 · num_attention_heads = 24 · num_key_value_heads = 816 cases
seq_len
batch_size
dtype
1281bf1612816bf1612832bf162564bf162568bf1625616bf1625632bf162932bf165121bf165122bf165124bf166138bf1610241bf1620534bf1635571bf1681921bf16
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 1177.5 µsRecursive17NVIDIA B200 · env 2155.9 µsac4k12

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 17 results · last observed 2026-08-23Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Recursive
177.5µs79% SOL
1.00×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Hyra
181.3µs80% SOL
1.02×
Reported · license unknown · no source
2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Amir M. Mir | SF Tensor
189.8µs77% SOL
1.07×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Databricks
191.2µs78% SOL
1.08×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
ac4k
203.4µs75% SOL
1.15×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
doubleAI
216.9µs75% SOL
1.22×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Geometric
219.8µs75% SOL
1.24×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Geometric
222.3µs75% SOL
1.25×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Geometric
225.3µs74% SOL
1.27×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Geometric
227.3µs74% SOL
1.28×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Geometric
237.3µs73% SOL
1.34×
Reported · license unknown · no source
2026-08-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
275.3µs69% SOL
1.55×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
TH3CHARLie | TileGym
276.5µs69% SOL
1.56×
Reported · license unknown · no source
2026-07-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
310.3µs65% SOL
1.75×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
327.3µs64% SOL
1.84×
Reported · license unknown · no source
2026-08-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Cursor
516.3µs51% SOL
2.91×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Geometric
986.0µs34% SOL
5.56×
Reported · license unknown · no source
2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Databricks
unknown
177.7µs
1.14×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
174.1µs
1.12×
Reported
license unknown · no source
doubleAI
unknown
203.9µs
1.31×
Reported
license unknown · no source
Geometric
unknown
219.8µs
1.41×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
276.5µs
1.77×
Reported
license unknown · no source
Argus
unknown
388.6µs
2.49×
Reported
license unknown · no source
Cursor
unknown
460.9µs
2.96×
Reported
license unknown · no source
Igor Bragaia
unknown
1.45ms
9.28×
Reported
license unknown · no source
Show all 29 implementations ›
Recursive
unknown
177.5µs
1.14×
Reported
license unknown · no source
Databricks
unknown
191.2µs
1.23×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
189.8µs
1.22×
Reported
license unknown · no source
ac4k
unknown
203.4µs
1.30×
Reported
license unknown · no source
doubleAI
unknown
216.9µs
1.39×
Reported
license unknown · no source
Geometric
unknown
222.3µs
1.43×
Reported
license unknown · no source
Geometric
unknown
225.3µs
1.44×
Reported
license unknown · no source
Geometric
unknown
227.3µs
1.46×
Reported
license unknown · no source
doubleAI
unknown
228.9µs
1.47×
Reported
license unknown · no source
Geometric
unknown
237.3µs
1.52×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
225.1µs
1.44×
Reported
license unknown · no source
Geometric
unknown
275.3µs
1.77×
Reported
license unknown · no source
Geometric
unknown
310.3µs
1.99×
Reported
license unknown · no source
Geometric
unknown
327.3µs
2.10×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
410.4µs
2.63×
Reported
license unknown · no source
Cursor
unknown
516.3µs
3.31×
Reported
license unknown · no source
Geometric
unknown
986.0µs
6.32×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.40ms
8.96×
Reported
license unknown · no source
ac4k
unknown
155.9µs
1.00×
Reported
license unknown · no source
Recursive
unknown
170.3µs
1.09×
Reported
license unknown · no source
Hyra
unknown
181.3µs
1.16×
Reported
license unknown · no source

Semantics

Inputs and outputs
hidden_statesbf16 [batch_size, seq_len, hidden_size]
cosbf16 [batch_size, seq_len, head_dim]
sinbf16 [batch_size, seq_len, head_dim]
attention_maskbf16 [batch_size, 1, seq_len, seq_len]
q_proj_weightbf16 [qkv_out_dim, hidden_size]
k_proj_weightbf16 [kv_out_dim, hidden_size]
v_proj_weightbf16 [kv_out_dim, hidden_size]
o_proj_weightbf16 [hidden_size, qkv_out_dim]
q_norm_weightbf16 [head_dim]
k_norm_weightbf16 [head_dim]
attn_logit_softcappingfp32 scalar
rms_norm_epsfp32 scalar
attn_outputbf16 [batch_size, seq_len, hidden_size]
Axes and behavior
seq_lenvariable
head_dimconstant = 128
batch_sizevariable
kv_out_dimkv_out_dim = num_key_value_heads * head_dim
hidden_sizeconstant = 3072
qkv_out_dimqkv_out_dim = num_attention_heads * head_dim
num_attention_headsconstant = 24
num_key_value_headsconstant = 8
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias047_attention_with_qk_norm_and_ropemodelgemma-3-12b-itsha256957dedb43419…
Sources: NVIDIA SOL-ExecBench (2026-08-23)last observed 2026-08-23How records are decidedJSON