MoE training token repeat and expert computation
MoE training forward pass with token repetition and parallel expert computation. Each token is repeated num_experts_per_tok times, processed through experts using masked indexing with SwiGLU FFN, and routing weights are applied before summing across experts.
Reported evidence · last observed 2026-07-10. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 8192 · num_experts = 256 · num_experts_per_tok = 8 · moe_intermediate_size = 204816 cases
Not measured on H100 for this workload. Challenges →
14.47ms79% SOL1.00×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
24.48ms80% SOL1.00×Reported · license unknown · no source2026-08-05
Measured exactly what you asked. Reported by source; not independently reproduced.
34.61ms78% SOL1.03×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
44.80ms75% SOL1.07×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
55.03ms72% SOL1.12×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
65.05ms72% SOL1.13×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
75.14ms71% SOL1.15×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
85.15ms71% SOL1.15×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
95.16ms71% SOL1.15×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
105.17ms70% SOL1.16×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
115.17ms71% SOL1.16×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
125.35ms70% SOL1.20×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
135.55ms66% SOL1.24×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
146.53ms57% SOL1.46×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
158.72ms44% SOL1.95×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.