MoE group score aggregation and masking
Group-based score aggregation and masking for MoE routing. Reshapes 256 expert scores into 8 groups of 32 experts each, computes group scores by summing top-2 expert scores within each group, selects top 4 groups, creates and expands group mask to expert-level, and applies masked_fill to suppress non-selected experts with -inf. This reduces the expert search space from 256 to ~128 experts before final topk selection.
Reported evidence · last observed 2026-07-10. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyn_group = 8 · topk_group = 4 · num_experts = 256 · experts_per_group = 3216 cases
Not measured on H100 for this workload. Challenges →
13.51µs87% SOL1.00×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
23.71µs87% SOL1.06×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
33.73µs87% SOL1.06×Reported · license unknown · no source2026-07-18
Measured exactly what you asked. Reported by source; not independently reproduced.
43.82µs86% SOL1.09×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
53.86µs86% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
63.87µs86% SOL1.10×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
73.99µs85% SOL1.13×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
84.62µs83% SOL1.31×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
94.63µs83% SOL1.32×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
106.51µs75% SOL1.85×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.