Fused gate up projection with SwiGLU
Gemma3's fused gate and up projection followed by GELU-tanh activation and element-wise multiplication. Computes gelu_tanh(x @ gate_proj.T) * (x @ up_proj.T) where gate_proj and up_proj are weight matrices. This fuses two parallel linear projections from hidden_size=3072 to intermediate_size=24576, followed by GELU activation on gate output and element-wise multiplication with up output.
Reported evidence · last observed 2026-08-19. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 3072 · intermediate_size = 2457616 cases
Not measured on H100 for this workload. Challenges →
1317.9µs72% SOL1.00×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
2333.9µs70% SOL1.05×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
3336.7µs66% SOL1.06×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
4347.8µs64% SOL1.09×Reported · license unknown · no source2026-08-06
Measured exactly what you asked. Reported by source; not independently reproduced.
5349.7µs64% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
6350.8µs63% SOL1.10×Reported · license unknown · no source2026-08-24
Measured exactly what you asked. Reported by source; not independently reproduced.
7351.1µs63% SOL1.10×Reported · license unknown · no source2026-08-24
Measured exactly what you asked. Reported by source; not independently reproduced.
8351.6µs63% SOL1.11×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
9352.8µs63% SOL1.11×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
10355.4µs63% SOL1.12×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
11356.2µs63% SOL1.12×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
12357.7µs62% SOL1.13×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
13359.7µs62% SOL1.13×Reported · license unknown · no source2026-07-12
Measured exactly what you asked. Reported by source; not independently reproduced.
14363.7µs62% SOL1.14×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
15373.6µs58% SOL1.18×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
16382.9µs54% SOL1.20×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
17385.2µs54% SOL1.21×Reported · license unknown · no source2026-07-11
Measured exactly what you asked. Reported by source; not independently reproduced.
18387.4µs54% SOL1.22×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
19402.9µs51% SOL1.27×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
20404.1µs51% SOL1.27×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
21404.1µs51% SOL1.27×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
22404.9µs51% SOL1.27×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
23406.9µs50% SOL1.28×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
24407.1µs51% SOL1.28×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
25407.7µs50% SOL1.28×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
26417.4µs49% SOL1.31×Reported · license unknown · no source2026-07-29
Measured exactly what you asked. Reported by source; not independently reproduced.
27439.3µs44% SOL1.38×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
28463.8µs41% SOL1.46×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
29476.3µs40% SOL1.50×Reported · license unknown · no source2026-08-04
Measured exactly what you asked. Reported by source; not independently reproduced.
30522.5µs40% SOL1.64×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.