MoE expert inference batched dispatch
Optimized MoE expert inference dispatch that batches tokens by expert assignment, processes them through experts sequentially with SwiGLU activation, and reconstructs the output with weighted combinations. This is the critical inference path for a 256-expert MoE system that processes 8 experts per token.
Reported evidence · last observed 2026-08-11. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 4096 · num_experts = 256 · intermediate_size = 1024 · num_experts_per_tok = 816 cases
Not measured on H100 for this workload. Challenges →
11.21ms94% SOL1.00×Reported · license unknown · no source2026-08-11
Measured exactly what you asked. Reported by source; not independently reproduced.
21.25ms94% SOL1.03×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
31.29ms93% SOL1.07×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
41.31ms93% SOL1.08×Reported · license unknown · no source2026-08-16
Measured exactly what you asked. Reported by source; not independently reproduced.
51.32ms93% SOL1.09×Reported · license unknown · no source2026-08-06
Measured exactly what you asked. Reported by source; not independently reproduced.
61.34ms92% SOL1.11×Reported · license unknown · no source2026-08-04
Measured exactly what you asked. Reported by source; not independently reproduced.
71.37ms92% SOL1.13×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
81.41ms91% SOL1.17×Reported · license unknown · no source2026-07-31
Measured exactly what you asked. Reported by source; not independently reproduced.
91.45ms91% SOL1.20×Reported · license unknown · no source2026-07-31
Measured exactly what you asked. Reported by source; not independently reproduced.
101.48ms90% SOL1.22×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
111.59ms89% SOL1.31×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
121.61ms89% SOL1.33×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
131.71ms87% SOL1.41×Reported · license unknown · no source2026-08-08
Measured exactly what you asked. Reported by source; not independently reproduced.
141.77ms87% SOL1.46×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
151.77ms87% SOL1.46×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
162.00ms84% SOL1.65×Reported · license unknown · no source2026-08-06
Measured exactly what you asked. Reported by source; not independently reproduced.
174.41ms63% SOL3.65×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
186.52ms51% SOL5.38×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.