Hybrid attention mask preparation
Hybrid causal mask preparation system that creates separate attention masks for full attention and sliding window attention layers. Creates two distinct 4D causal masks: standard causal masking for full attention layers and sliding window causal masking for SWA layers.
Reported evidence · last observed 2026-06-25. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 12 cases · mean latencysliding_window = 128 · num_attention_heads = 64 · swa_num_attention_heads = 6412 cases
Not measured on H100 for this workload. Challenges →
1954ns94% SOL1.00×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
21.17µs92% SOL1.22×Reported · license unknown · no source2026-07-02
Measured exactly what you asked. Reported by source; not independently reproduced.
31.94µs88% SOL2.04×Reported · license unknown · no source2026-06-29
Measured exactly what you asked. Reported by source; not independently reproduced.
41.98µs87% SOL2.08×Reported · license unknown · no source2026-06-06
Measured exactly what you asked. Reported by source; not independently reproduced.
52.15µs86% SOL2.25×Reported · license unknown · no source2026-05-20
Measured exactly what you asked. Reported by source; not independently reproduced.
62.27µs85% SOL2.38×Reported · license unknown · no source2026-07-08
Measured exactly what you asked. Reported by source; not independently reproduced.
72.56µs83% SOL2.69×Reported · license unknown · no source2026-07-03
Measured exactly what you asked. Reported by source; not independently reproduced.
82.66µs82% SOL2.79×Reported · license unknown · no source2026-07-08
Measured exactly what you asked. Reported by source; not independently reproduced.
95.91µs64% SOL6.20×Reported · license unknown · no source2026-04-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1025.1µs30% SOL26.35×Reported · license unknown · no source2026-05-28
Measured exactly what you asked. Reported by source; not independently reproduced.
1173.2µs28% SOL76.69×Reported · license unknown · no source2026-05-30
Measured exactly what you asked. Reported by source; not independently reproduced.
1296.4µs27% SOL101.00×Reported · license unknown · no source2026-07-05
Measured exactly what you asked. Reported by source; not independently reproduced.
1399.3µs27% SOL104.09×Reported · license unknown · no source2026-05-28
Measured exactly what you asked. Reported by source; not independently reproduced.
14176.2µs20% SOL184.69×Reported · license unknown · no source2026-05-16
Measured exactly what you asked. Reported by source; not independently reproduced.
15184.9µs24% SOL193.80×Reported · license unknown · no source2026-05-16
Measured exactly what you asked. Reported by source; not independently reproduced.
16346.9µs12% SOL363.60×Reported · license unknown · no source2026-04-21
Measured exactly what you asked. Reported by source; not independently reproduced.