KV shared attention with dual RoPE
KV-shared attention with dual RoPE (global and local), Q/K/V normalization, soft-capping, and grouped-query attention. Implements attention where later layers can reuse KV states from earlier layers, with dual rotary position embeddings.
Reported evidence · last observed 2026-08-06. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhead_dim = 96 · q_proj_dim = 3072 · hidden_size = 3072 · kv_proj_dim = 768 · num_attention_heads = 32 · num_key_value_heads = 8 · num_key_value_groups = 416 cases
Not measured on H100 for this workload. Challenges →
1174.6µs92% SOL1.00×Reported · license unknown · no source2026-08-06
Measured exactly what you asked. Reported by source; not independently reproduced.
2193.1µs91% SOL1.11×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
3222.3µs89% SOL1.27×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
4225.2µs89% SOL1.29×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
5268.8µs87% SOL1.54×Reported · license unknown · no source2026-07-19
Measured exactly what you asked. Reported by source; not independently reproduced.
6270.2µs87% SOL1.55×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
7299.3µs85% SOL1.71×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
8300.4µs85% SOL1.72×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
9301.7µs85% SOL1.73×Reported · license unknown · no source2026-07-19
Measured exactly what you asked. Reported by source; not independently reproduced.
10314.3µs84% SOL1.80×Reported · license unknown · no source2026-07-28
Measured exactly what you asked. Reported by source; not independently reproduced.
11322.7µs84% SOL1.85×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
12337.5µs83% SOL1.93×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
13445.7µs78% SOL2.55×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
141.16ms56% SOL6.62×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.