Attention QKV with QK norm single kernel backward

65 eligible runs

rmsnorm

Backward pass for Gemma3 fused QKV projection with Q/K RMSNorm normalization. Computes gradients for hidden_states, Q/K/V projection weights, and Q/K normalization weights.

Fastest reported

2.35µsmean · 100% of SOL

jonathanc.netLicense unknown · unknown

Reported evidence · last observed 2026-07-11. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 256 · hidden_size = 640 · q_proj_size = 1024 · kv_proj_size = 256 · num_attention_heads = 4 · num_key_value_heads = 116 cases

seq_len

batch_size

dtype

1282bf16/fp32 1284bf16/fp32 1288bf16/fp32 12832bf16/fp32 12864bf16/fp32 1311bf16/fp32 2564bf16/fp32 2568bf16/fp32 3734bf16/fp32 5121bf16/fp32 54116bf16/fp32 9971bf16/fp32 10241bf16/fp32 10244bf16/fp32 40961bf16/fp32 81921bf16/fp32

scrolls · 16 cases total

Aggregatessuite of 16 cases · mean latency

HardwareBest knownImplementationRuns

NVIDIA B200 · env 12.35 µsjonathanc.net52 NVIDIA B200 · env 224.5 µsRecursive13

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 52 results · last observed 2026-08-24Record history →

#

Implementation

Latency

vs #1

Trust

Observed

1

2.35µs100% SOL

1.00×

Reported · license unknown · no source

2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

2

Amir M. Mir | SF Tensor

25.0µs85% SOL

10.64×

Reported · license unknown · no source

2026-08-02

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

3

25.4µs85% SOL

10.81×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

4

Amir M. Mir | SF Tensor

26.1µs84% SOL

11.09×

Reported · license unknown · no source

2026-08-02

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

5

27.4µs84% SOL

11.66×

Reported · license unknown · no source

2026-08-03

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

6

27.4µs84% SOL

11.68×

Reported · license unknown · no source

2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

7

27.7µs84% SOL

11.78×

Reported · license unknown · no source

2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

8

28.1µs83% SOL

11.97×

Reported · license unknown · no source

2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

9

28.7µs83% SOL

12.23×

Reported · license unknown · no source

2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

10

28.9µs83% SOL

12.29×

Reported · license unknown · no source

2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

11

29.2µs83% SOL

12.44×

Reported · license unknown · no source

2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

12

29.3µs82% SOL

12.48×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

13

31.4µs82% SOL

13.36×

Reported · license unknown · no source

2026-08-21

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

14

31.4µs82% SOL

13.38×

Reported · license unknown · no source

2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

15

31.5µs82% SOL

13.42×

Reported · license unknown · no source

2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

16

31.6µs81% SOL

13.44×

Reported · license unknown · no source

2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

17

Amir M. Mir | SF Tensor

32.3µs81% SOL

13.77×

Reported · license unknown · no source

2026-07-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

18

33.6µs81% SOL

14.30×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

19

34.0µs80% SOL

14.49×

Reported · license unknown · no source

2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

20

Electric Cheetah

34.1µs80% SOL

14.50×

Reported · license unknown · no source

2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

21

34.1µs80% SOL

14.51×

Reported · license unknown · no source

2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

22

34.8µs80% SOL

14.79×

Reported · license unknown · no source

2026-07-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

23

Gleaming Coyote

34.8µs80% SOL

14.80×

Reported · license unknown · no source

2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

24

34.8µs80% SOL

14.83×

Reported · license unknown · no source

2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

25

34.9µs80% SOL

14.86×

Reported · license unknown · no source

2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

26

35.0µs80% SOL

14.89×

Reported · license unknown · no source

2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

27

37.2µs78% SOL

15.82×

Reported · license unknown · no source

2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

28

TH3CHARLie | TileGym

37.4µs78% SOL

15.90×

Reported · license unknown · no source

2026-07-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

29

37.6µs78% SOL

15.99×

Reported · license unknown · no source

2026-08-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

30

38.5µs78% SOL

16.38×

Reported · license unknown · no source

2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

31

39.0µs78% SOL

16.59×

Reported · license unknown · no source

2026-08-05

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

32

39.3µs77% SOL

16.73×

Reported · license unknown · no source

2026-08-09

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

33

40.0µs77% SOL

17.02×

Reported · license unknown · no source

2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

34

44.6µs75% SOL

18.99×

Reported · license unknown · no source

2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

35

Amir M. Mir | SF Tensor

50.6µs72% SOL

21.53×

Reported · license unknown · no source

2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

36

50.7µs72% SOL

21.59×

Reported · license unknown · no source

2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

37

52.5µs72% SOL

22.36×

Reported · license unknown · no source

2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

38

53.0µs71% SOL

22.58×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

39

54.9µs71% SOL

23.37×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

40

Amir M. Mir | SF Tensor

58.6µs69% SOL

24.95×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

41

60.4µs68% SOL

25.73×

Reported · license unknown · no source

2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

42

63.4µs67% SOL

26.98×

Reported · license unknown · no source

2026-07-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

43

65.4µs66% SOL

27.86×

Reported · license unknown · no source

2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

44

73.0µs63% SOL

31.09×

Reported · license unknown · no source

2026-07-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

45

112.0µs53% SOL

47.67×

Reported · license unknown · no source

2026-07-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

46

203.8µs38% SOL

86.78×

Reported · license unknown · no source

2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

47

209.7µs37% SOL

89.28×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

48

220.3µs36% SOL

93.79×

Reported · license unknown · no source

2026-07-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

49

312.1µs30% SOL

132.87×

Reported · license unknown · no source

2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

50

333.6µs29% SOL

142.03×

Reported · license unknown · no source

2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

51

836.2µs16% SOL

356.00×

Reported · license unknown · no source

2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

52

15.3ms1% SOL

6496.09×

Reported · license unknown · no source

2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation

Runtime

Best latency

Evidence

Availability

unknown

2.35µs

1.00×

Reported

license unknown · no source

unknown

24.5µs

10.41×

Reported

license unknown · no source

unknown

27.4µs

11.68×

Reported

license unknown · no source

unknown

28.0µs

11.93×

Reported

license unknown · no source

unknown

29.2µs

12.44×

Reported

license unknown · no source

unknown

32.9µs

14.02×

Reported

license unknown · no source

unknown

34.0µs

14.49×

Reported

license unknown · no source

Electric Cheetah

unknown

34.1µs

14.50×

Reported

license unknown · no source

Show all 65 implementations ›

Gleaming Coyote

unknown

34.8µs

14.80×

Reported

license unknown · no source

TH3CHARLie | TileGym

unknown

37.4µs

15.90×

Reported

license unknown · no source

unknown

80.8µs

34.38×

Reported

license unknown · no source

unknown

165.4µs

70.42×

Reported

license unknown · no source

unknown

426.6µs

181.62×

Reported

license unknown · no source

unknown

25.4µs

10.81×

Reported

license unknown · no source

unknown

27.7µs

11.78×

Reported

license unknown · no source

unknown

28.1µs

11.97×

Reported

license unknown · no source

unknown

28.7µs

12.23×

Reported

license unknown · no source

unknown

28.9µs

12.29×

Reported

license unknown · no source

unknown

29.3µs

12.48×

Reported

license unknown · no source

unknown

31.4µs

13.36×

Reported

license unknown · no source

unknown

31.4µs

13.38×

Reported

license unknown · no source

unknown

31.5µs

13.42×

Reported

license unknown · no source

unknown

31.6µs

13.44×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

32.3µs

13.77×

Reported

license unknown · no source

unknown

33.6µs

14.30×

Reported

license unknown · no source

unknown

34.1µs

14.51×

Reported

license unknown · no source

unknown

33.9µs

14.45×

Reported

license unknown · no source

unknown

34.9µs

14.86×

Reported

license unknown · no source

unknown

34.8µs

14.79×

Reported

license unknown · no source

unknown

35.0µs

14.89×

Reported

license unknown · no source

unknown

34.8µs

14.83×

Reported

license unknown · no source

unknown

36.1µs

15.37×

Reported

license unknown · no source

unknown

37.6µs

15.99×

Reported

license unknown · no source

unknown

37.2µs

15.82×

Reported

license unknown · no source

unknown

38.5µs

16.38×

Reported

license unknown · no source

unknown

39.0µs

16.59×

Reported

license unknown · no source

unknown

39.3µs

16.73×

Reported

license unknown · no source

unknown

40.0µs

17.02×

Reported

license unknown · no source

unknown

44.6µs

18.99×

Reported

license unknown · no source

unknown

50.7µs

21.59×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

50.6µs

21.53×

Reported

license unknown · no source

unknown

52.5µs

22.36×

Reported

license unknown · no source

unknown

53.0µs

22.58×

Reported

license unknown · no source

unknown

54.9µs

23.37×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

57.1µs

24.29×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

58.6µs

24.95×

Reported

license unknown · no source

unknown

60.4µs

25.73×

Reported

license unknown · no source

unknown

63.4µs

26.98×

Reported

license unknown · no source

unknown

65.4µs

27.86×

Reported

license unknown · no source

unknown

73.0µs

31.09×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

77.7µs

33.06×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

103.4µs

44.01×

Reported

license unknown · no source

unknown

112.0µs

47.67×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

121.9µs

51.91×

Reported

license unknown · no source

unknown

203.8µs

86.78×

Reported

license unknown · no source

unknown

209.7µs

89.28×

Reported

license unknown · no source

unknown

220.3µs

93.79×

Reported

license unknown · no source

unknown

312.1µs

132.87×

Reported

license unknown · no source

unknown

333.6µs

142.03×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

341.9µs

145.57×

Reported

license unknown · no source

unknown

836.2µs

356.00×

Reported

license unknown · no source

unknown

15.3ms

6496.09×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

25.0µs

10.64×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

26.1µs

11.09×

Reported

license unknown · no source

unknown

27.4µs

11.66×

Reported

license unknown · no source

Semantics

Inputs and outputs

grad_querybf16 [batch_size, num_attention_heads, seq_len, head_dim]

grad_keybf16 [batch_size, num_key_value_heads, seq_len, head_dim]

grad_valuebf16 [batch_size, num_key_value_heads, seq_len, head_dim]

hidden_statesbf16 [batch_size, seq_len, hidden_size]

q_weightbf16 [q_proj_size, hidden_size]

k_weightbf16 [kv_proj_size, hidden_size]

v_weightbf16 [kv_proj_size, hidden_size]

q_norm_weightbf16 [head_dim]

k_norm_weightbf16 [head_dim]

query_transposedbf16 [batch_size, num_attention_heads, seq_len, head_dim]

key_transposedbf16 [batch_size, num_key_value_heads, seq_len, head_dim]

q_rstdfp32 [batch_size, num_attention_heads, seq_len, 1]

k_rstdfp32 [batch_size, num_key_value_heads, seq_len, 1]

q_normedbf16 [batch_size, num_attention_heads, seq_len, head_dim]

k_normedbf16 [batch_size, num_key_value_heads, seq_len, head_dim]

rms_norm_epsfp32 scalar

grad_hidden_statesbf16 [batch_size, seq_len, hidden_size]

grad_q_weightbf16 [q_proj_size, hidden_size]

grad_k_weightbf16 [kv_proj_size, hidden_size]

grad_v_weightbf16 [kv_proj_size, hidden_size]

grad_q_norm_weightfp32 [head_dim]

grad_k_norm_weightfp32 [head_dim]

Axes and behavior

seq_lenvariable

head_dimconstant = 256

batch_sizevariable

hidden_sizeconstant = 640

q_proj_sizeq_proj_size = num_attention_heads * head_dim

kv_proj_sizekv_proj_size = num_key_value_heads * head_dim

num_attention_headsconstant = 4

num_key_value_headsconstant = 1

determinismunspecified

constraintsNo mutation or aliasing

Identity

alias051_attention_qkv_with_qk_norm_single_kernel_backwardmodelgemma-3-270msha2562ed74f1d0e5c…

Sources: NVIDIA SOL-ExecBench (2026-08-24)last observed 2026-08-24How records are decided JSON