Skip to content
KernelIndex
Search⌘K
Sign in

Fused vision cu seqlens attention with 2d RoPE backward

23 eligible runs
attention

Backward pass for fused vision attention with cu_seqlens and 2D rotary position embeddings. Computes gradients for hidden_states, QKV projection weights/biases, and output projection weights/biases.

Fastest reported
516.7µsmean · 81% of SOL
HyraLicense unknown · unknown

Reported evidence · last observed 2026-07-17. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhead_dim = 128 · num_heads = 28 · hidden_size = 3584 · qkv_out_size = 1075216 cases
total_seq_len
num_sequences
dtype
1311fp321961fp322112fp323733fp323922fp326134fp327735fp329196fp3210244fp3211766fp3220488fp32294015fp32409616fp32470424fp32588030fp32819232fp32
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 1516.7 µsHyra16NVIDIA B200 · env 2555.2 µsdoubleAI7

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 16 results · last observed 2026-08-24Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Hyra
516.7µs81% SOL
1.00×
Reported · license unknown · no source
2026-07-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Amir M. Mir | SF Tensor
522.2µs80% SOL
1.01×
Reported · license unknown · no source
2026-07-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Recursive
570.9µs78% SOL
1.10×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
doubleAI
593.6µs78% SOL
1.15×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
ac4k
637.8µs76% SOL
1.23×
Reported · license unknown · no source
2026-08-03

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Geometric
739.3µs72% SOL
1.43×
Reported · license unknown · no source
2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Geometric
759.6µs71% SOL
1.47×
Reported · license unknown · no source
2026-08-15

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Amir M. Mir | SF Tensor
776.1µs70% SOL
1.50×
Reported · license unknown · no source
2026-07-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Geometric
788.9µs70% SOL
1.53×
Reported · license unknown · no source
2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Amir M. Mir | SF Tensor
828.1µs69% SOL
1.60×
Reported · license unknown · no source
2026-07-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Proud Wren @ NVIDIA
864.5µs67% SOL
1.67×
Reported · license unknown · no source
2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
879.8µs67% SOL
1.70×
Reported · license unknown · no source
2026-08-12

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Geometric
921.5µs66% SOL
1.78×
Reported · license unknown · no source
2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Amir M. Mir | SF Tensor
1.35ms54% SOL
2.61×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
1.59ms50% SOL
3.08×
Reported · license unknown · no source
2026-08-08

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Cursor
2.64ms36% SOL
5.11×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Recursive
unknown
557.3µs
1.08×
Reported
license unknown · no source
ac4k
unknown
637.8µs
1.23×
Reported
license unknown · no source
Geometric
unknown
739.3µs
1.43×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
864.5µs
1.67×
Reported
license unknown · no source
Cursor
unknown
2.51ms
4.86×
Reported
license unknown · no source
Recursive
unknown
570.9µs
1.10×
Reported
license unknown · no source
doubleAI
unknown
593.6µs
1.15×
Reported
license unknown · no source
Geometric
unknown
759.6µs
1.47×
Reported
license unknown · no source
Show all 23 implementations ›
Geometric
unknown
788.9µs
1.53×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
776.1µs
1.50×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
828.1µs
1.60×
Reported
license unknown · no source
Geometric
unknown
879.8µs
1.70×
Reported
license unknown · no source
doubleAI
unknown
859.3µs
1.66×
Reported
license unknown · no source
Geometric
unknown
921.5µs
1.78×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.23ms
2.38×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.30ms
2.51×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
1.35ms
2.61×
Reported
license unknown · no source
Geometric
unknown
1.59ms
3.08×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
2.31ms
4.47×
Reported
license unknown · no source
Cursor
unknown
2.64ms
5.11×
Reported
license unknown · no source
Hyra
unknown
516.7µs
1.00×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
522.2µs
1.01×
Reported
license unknown · no source
doubleAI
unknown
555.2µs
1.07×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputfp32 [total_seq_len, hidden_size]
hidden_statesfp32 [total_seq_len, hidden_size]
qkv_weightfp32 [qkv_out_size, hidden_size]
qkv_biasfp32 [qkv_out_size]
proj_weightfp32 [hidden_size, hidden_size]
proj_biasfp32 [hidden_size]
cosfp32 [total_seq_len, head_dim]
sinfp32 [total_seq_len, head_dim]
cu_seqlensint32 [num_seqlens]
attention_dropoutfp32 scalar
scalingfp32 scalar
grad_hidden_statesfp32 [total_seq_len, hidden_size]
grad_qkv_weightfp32 [qkv_out_size, hidden_size]
grad_qkv_biasfp32 [qkv_out_size]
grad_proj_weightfp32 [hidden_size, hidden_size]
grad_proj_biasfp32 [hidden_size]
Axes and behavior
head_dimconstant = 128
num_headsconstant = 28
hidden_sizeconstant = 3584
num_seqlensnum_seqlens = num_sequences + 1
qkv_out_sizeqkv_out_size = hidden_size * 3
num_sequencesvariable
total_seq_lenvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias017_fused_vision_cu_seqlens_attention_with_2d_rope_backwardmodelqwen3-vl-4b-instructsha256f4c324fc8a90…
Sources: NVIDIA SOL-ExecBench (2026-08-24)last observed 2026-08-24How records are decidedJSON