Skip to content
KernelIndex
Search⌘K
Sign in

Flash attention GQA ultralong

35 eligible runs
other

Flash Attention for Grouped Query Attention with ultra-long context. Fuses QKV projection, RoPE application, GQA KV repetition, scaled dot-product attention with causal masking, and output projection.

Fastest reported
177.6µsmean · 69% of SOL
HyraLicense unknown · unknown

Reported evidence · last observed 2026-07-16. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 18 cases · mean latencyhead_dim = 128 · q_proj_out = 4096 · hidden_size = 4096 · kv_proj_out = 1024 · num_attention_heads = 32 · num_key_value_heads = 8 · num_key_value_groups = 418 cases
seq_len
batch_size
dtype
1281fp321282fp3212864fp321311fp322561fp322562fp3225616fp322931fp325121fp325122fp325128fp325412fp329971fp3210242fp3220482fp3220531fp3281922fp32163841fp32
scrolls · 18 cases total
Aggregatessuite of 18 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 1177.6 µsHyra22NVIDIA B200 · env 2198.5 µsac4k13

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 18 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 22 results · last observed 2026-08-20Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Hyra
177.6µs69% SOL
1.00×
Reported · license unknown · no source
2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Amir M. Mir | SF Tensor
194.4µs66% SOL
1.09×
Reported · license unknown · no source
2026-07-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
ac4k
212.9µs63% SOL
1.20×
Reported · license unknown · no source
2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Recursive
239.6µs60% SOL
1.35×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
Geometric
243.8µs59% SOL
1.37×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Geometric
244.0µs59% SOL
1.37×
Reported · license unknown · no source
2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Geometric
251.7µs58% SOL
1.42×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Geometric
253.1µs58% SOL
1.42×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Amir M. Mir | SF Tensor
255.1µs58% SOL
1.44×
Reported · license unknown · no source
2026-07-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Q H
259.7µs57% SOL
1.46×
Reported · license unknown · no source
2026-07-30

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Geometric
259.8µs57% SOL
1.46×
Reported · license unknown · no source
2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
260.9µs57% SOL
1.47×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Databricks
264.1µs57% SOL
1.49×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
277.1µs55% SOL
1.56×
Reported · license unknown · no source
2026-08-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Amir M. Mir | SF Tensor
279.9µs55% SOL
1.58×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
ac4k
301.3µs52% SOL
1.70×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
TH3CHARLie | TileGym
308.1µs51% SOL
1.73×
Reported · license unknown · no source
2026-07-27

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Geometric
322.1µs50% SOL
1.81×
Reported · license unknown · no source
2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Geometric
357.2µs46% SOL
2.01×
Reported · license unknown · no source
2026-07-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
20
doubleAI
378.6µs45% SOL
2.13×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
21
Cursor
441.5µs40% SOL
2.49×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
22
ac4k
576.1µs34% SOL
3.24×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
ac4k
unknown
198.5µs
1.12×
Reported
license unknown · no source
Recursive
unknown
237.2µs
1.34×
Reported
license unknown · no source
doubleAI
unknown
223.7µs
1.26×
Reported
license unknown · no source
Geometric
unknown
244.0µs
1.37×
Reported
license unknown · no source
Databricks
unknown
253.2µs
1.43×
Reported
license unknown · no source
Q H
unknown
259.7µs
1.46×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
308.1µs
1.73×
Reported
license unknown · no source
Argus
unknown
317.9µs
1.79×
Reported
license unknown · no source
Show all 35 implementations ›
Iridescent Lemur
unknown
320.9µs
1.81×
Reported
license unknown · no source
Cursor
unknown
362.5µs
2.04×
Reported
license unknown · no source
Recursive
unknown
239.6µs
1.35×
Reported
license unknown · no source
Geometric
unknown
243.8µs
1.37×
Reported
license unknown · no source
Geometric
unknown
251.7µs
1.42×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
255.1µs
1.44×
Reported
license unknown · no source
Geometric
unknown
253.1µs
1.42×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
262.1µs
1.48×
Reported
license unknown · no source
Geometric
unknown
259.8µs
1.46×
Reported
license unknown · no source
Geometric
unknown
260.9µs
1.47×
Reported
license unknown · no source
Databricks
unknown
264.1µs
1.49×
Reported
license unknown · no source
Geometric
unknown
277.1µs
1.56×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
279.9µs
1.58×
Reported
license unknown · no source
ac4k
unknown
301.3µs
1.70×
Reported
license unknown · no source
Geometric
unknown
322.1µs
1.81×
Reported
license unknown · no source
doubleAI
unknown
342.3µs
1.93×
Reported
license unknown · no source
Geometric
unknown
357.2µs
2.01×
Reported
license unknown · no source
doubleAI
unknown
378.6µs
2.13×
Reported
license unknown · no source
Cursor
unknown
441.5µs
2.49×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
466.4µs
2.63×
Reported
license unknown · no source
ac4k
unknown
576.1µs
3.24×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
3.84ms
21.63×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
4.24ms
23.87×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
4.50ms
25.33×
Reported
license unknown · no source
Hyra
unknown
177.6µs
1.00×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
194.4µs
1.09×
Reported
license unknown · no source
ac4k
unknown
212.9µs
1.20×
Reported
license unknown · no source

Semantics

Inputs and outputs
hidden_statesfp32 [batch_size, seq_len, hidden_size]
cosfp32 [batch_size, seq_len, head_dim]
sinfp32 [batch_size, seq_len, head_dim]
q_proj_weightfp32 [q_proj_out, hidden_size]
k_proj_weightfp32 [kv_proj_out, hidden_size]
v_proj_weightfp32 [kv_proj_out, hidden_size]
o_proj_weightfp32 [hidden_size, q_proj_out]
outputfp32 [batch_size, seq_len, hidden_size]
Axes and behavior
seq_lenvariable
head_dimconstant = 128
batch_sizevariable
q_proj_outq_proj_out = num_attention_heads * head_dim
hidden_sizeconstant = 4096
kv_proj_outkv_proj_out = num_key_value_heads * head_dim
num_attention_headsconstant = 32
num_key_value_headsconstant = 8
num_key_value_groupsconstant = 4
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias067_flash_attention_gqa_ultralongmodelllama-3-1-nemotron-8b-ultralong-1m-instructsha256d528eca47bd7…
Sources: NVIDIA SOL-ExecBench (2026-08-20)last observed 2026-08-20How records are decidedJSON