Sparse expert dispatch and combine
Sparse expert dispatch and weighted combination for MoE training. Processes experts sequentially with sparse token extraction, expert computation (gate_up projection, gated GLU with clamping, down projection), and scattered accumulation using index_add_.
Reported evidence · last observed 2026-08-16. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 2880 · intermediate_size = 2880 · num_local_experts = 128 · num_experts_per_tok = 4 · intermediate_size_x2 = 576016 cases
Not measured on H100 for this workload. Challenges →
12.19ms71% SOL1.00×Reported · license unknown · no source2026-08-16
Measured exactly what you asked. Reported by source; not independently reproduced.
22.28ms70% SOL1.04×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
32.28ms70% SOL1.04×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
42.30ms69% SOL1.05×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
52.33ms69% SOL1.06×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
62.36ms68% SOL1.08×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
72.39ms68% SOL1.09×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
82.61ms65% SOL1.19×Reported · license unknown · no source2026-08-11
Measured exactly what you asked. Reported by source; not independently reproduced.
93.20ms58% SOL1.47×Reported · license unknown · no source2026-07-12
Measured exactly what you asked. Reported by source; not independently reproduced.
105.52ms41% SOL2.52×Reported · license unknown · no source2026-08-08
Measured exactly what you asked. Reported by source; not independently reproduced.