MoE expert computation
MoE expert computation with SwiGLU activation. Each expert is a small MLP with 2048 intermediate size that processes selected tokens. The expert uses gated activation (SwiGLU): down_proj(silu(gate_proj(x)) * up_proj(x)). Tokens are routed to experts based on expert_ids and weighted by expert_weights.
Reported evidence · last observed 2026-07-10. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 7168 · num_experts = 256 · intermediate_size = 2048 · num_experts_per_tok = 816 cases
Not measured on H100 for this workload. Challenges →
13.86ms84% SOL1.00×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
23.92ms83% SOL1.01×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
33.93ms83% SOL1.02×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
43.94ms83% SOL1.02×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
54.00ms82% SOL1.04×Reported · license unknown · no source2026-08-06
Measured exactly what you asked. Reported by source; not independently reproduced.
64.01ms82% SOL1.04×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
74.10ms81% SOL1.06×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
84.13ms80% SOL1.07×Reported · license unknown · no source2026-07-12
Measured exactly what you asked. Reported by source; not independently reproduced.
94.27ms79% SOL1.11×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
104.52ms75% SOL1.17×Reported · license unknown · no source2026-08-22
Measured exactly what you asked. Reported by source; not independently reproduced.
114.54ms75% SOL1.18×Reported · license unknown · no source2026-07-21
Measured exactly what you asked. Reported by source; not independently reproduced.
124.59ms75% SOL1.19×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
134.68ms74% SOL1.21×Reported · license unknown · no source2026-07-25
Measured exactly what you asked. Reported by source; not independently reproduced.
147.56ms51% SOL1.96×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.