Skip to content
KernelIndex
Search⌘K
Sign in

Audio attention QKV projection with normalization

29 eligible runs
rmsnorm

Fused QKV projection with per-head RMS normalization for audio attention. Projects hidden states to Q, K, V representations, reshapes to multi-head format, and applies separate RMS normalization to each head. This is a unique pattern where normalization is applied per-head rather than per-sequence.

Fastest reported
29.9µsmean · 93% of SOL
RIAC as wellLicense unknown · unknown

Reported evidence · last observed 2026-08-19. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 128 · num_heads = 8 · hidden_size = 1024 · qkv_out_size = 102416 cases
seq_len
batch_size
dtype
1281fp321282fp321284fp3212864fp321311fp322562fp322564fp322932fp325124fp325128fp3251216fp325414fp327738fp3210242fp3220482fp3240961fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 129.9 µsRIAC as well17NVIDIA B200 · env 232.3 µsDatabricks12

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 17 results · last observed 2026-08-24Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
RIAC as well
29.9µs93% SOL
1.00×
Reported · license unknown · no source
2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Databricks
33.3µs92% SOL
1.11×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Recursive
33.9µs92% SOL
1.13×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
TH3CHARLie | TileGym
34.6µs92% SOL
1.16×
Reported · license unknown · no source
2026-07-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Geometric
34.7µs92% SOL
1.16×
Reported · license unknown · no source
2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
TH3CHARLie | TileGym
39.3µs91% SOL
1.31×
Reported · license unknown · no source
2026-07-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
doubleAI
40.0µs91% SOL
1.34×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Geometric
40.3µs90% SOL
1.35×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Amir M. Mir | SF Tensor
40.4µs90% SOL
1.35×
Reported · license unknown · no source
2026-08-01

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Geometric
41.7µs90% SOL
1.39×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Hyra
42.2µs90% SOL
1.41×
Reported · license unknown · no source
2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Amir M. Mir | SF Tensor
42.6µs90% SOL
1.42×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Geometric
46.2µs89% SOL
1.55×
Reported · license unknown · no source
2026-08-08

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Gleaming Coyote
47.3µs89% SOL
1.58×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
ac4k
47.5µs89% SOL
1.59×
Reported · license unknown · no source
2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Cursor
60.5µs85% SOL
2.02×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
ac4k
85.1µs80% SOL
2.85×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
ac4k
unknown
47.5µs
1.59×
Reported
license unknown · no source
doubleAI
unknown
60.3µs
2.02×
Reported
license unknown · no source
Cursor
unknown
60.5µs
2.02×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
67.4µs
2.25×
Reported
license unknown · no source
ac4k
unknown
85.1µs
2.85×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
285.0µs
9.53×
Reported
license unknown · no source
RIAC as well
unknown
29.9µs
1.00×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
34.6µs
1.16×
Reported
license unknown · no source
Show all 29 implementations ›
Geometric
unknown
34.7µs
1.16×
Reported
license unknown · no source
doubleAI
unknown
38.0µs
1.27×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
40.4µs
1.35×
Reported
license unknown · no source
Hyra
unknown
42.2µs
1.41×
Reported
license unknown · no source
Databricks
unknown
32.3µs
1.08×
Reported
license unknown · no source
Recursive
unknown
32.8µs
1.10×
Reported
license unknown · no source
Databricks
unknown
33.3µs
1.11×
Reported
license unknown · no source
ac4k
unknown
44.4µs
1.49×
Reported
license unknown · no source
Argus
unknown
44.8µs
1.50×
Reported
license unknown · no source
Gleaming Coyote
unknown
47.3µs
1.58×
Reported
license unknown · no source
Cursor
unknown
58.0µs
1.94×
Reported
license unknown · no source
Zerg AI
unknown
295.4µs
9.88×
Reported
license unknown · no source
Bragaia
unknown
391.7µs
13.10×
Reported
license unknown · no source
Recursive
unknown
33.9µs
1.13×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
39.3µs
1.31×
Reported
license unknown · no source
doubleAI
unknown
40.0µs
1.34×
Reported
license unknown · no source
Geometric
unknown
40.3µs
1.35×
Reported
license unknown · no source
Geometric
unknown
41.7µs
1.39×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
41.7µs
1.40×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
42.6µs
1.42×
Reported
license unknown · no source
Geometric
unknown
46.2µs
1.55×
Reported
license unknown · no source

Semantics

Inputs and outputs
hidden_statesfp32 [batch_size, seq_len, hidden_size]
q_proj_weightfp32 [qkv_out_size, hidden_size]
k_proj_weightfp32 [qkv_out_size, hidden_size]
v_proj_weightfp32 [qkv_out_size, hidden_size]
q_norm_weightfp32 [head_dim]
k_norm_weightfp32 [head_dim]
epsfp32 scalar
query_statesfp32 [batch_size, seq_len, num_heads, head_dim]
key_statesfp32 [batch_size, seq_len, num_heads, head_dim]
value_statesfp32 [batch_size, seq_len, num_heads, head_dim]
Axes and behavior
seq_lenvariable
head_dimconstant = 128
num_headsconstant = 8
batch_sizevariable
hidden_sizeconstant = 1024
qkv_out_sizeqkv_out_size = num_heads * head_dim
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias054_audio_attention_qkv_projection_with_normalizationmodelgemma-3n-e4bsha25614daf31d9ad9…
Sources: NVIDIA SOL-ExecBench (2026-08-24)last observed 2026-08-24How records are decidedJSON