Audio encoder to language model multimodal fusion
Complete end-to-end multimodal fusion pipeline that processes raw audio features through a CTC encoder, applies windowed Q-Former downsampling with learnable queries, and scatters the resulting audio embeddings into language model token embeddings at audio token positions. This is the critical bottleneck that bridges audio and language modalities.
Reported evidence · last observed 2026-08-10. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencynum_queries = 40 · window_size = 15 · qformer_head_dim = 64 · text_hidden_size = 4096 · encoder_input_dim = 80 · qformer_num_heads = 16 · encoder_hidden_dim = 512 · encoder_output_dim = 1024 · projector_hidden_size = 102416 cases
Not measured on H100 for this workload. Challenges →
171.0µs99% SOL1.00×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
281.3µs98% SOL1.15×Reported · license unknown · no source2026-08-09
Measured exactly what you asked. Reported by source; not independently reproduced.
386.3µs98% SOL1.22×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
494.4µs94% SOL1.33×Reported · license unknown · no source2026-07-14
Measured exactly what you asked. Reported by source; not independently reproduced.
5115.8µs92% SOL1.63×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
6123.9µs94% SOL1.75×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
7127.3µs93% SOL1.79×Reported · license unknown · no source2026-08-04
Measured exactly what you asked. Reported by source; not independently reproduced.
8133.0µs92% SOL1.87×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
9135.8µs92% SOL1.91×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
10173.1µs85% SOL2.44×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
11176.9µs82% SOL2.49×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
12238.0µs73% SOL3.35×Reported · license unknown · no source2026-08-05
Measured exactly what you asked. Reported by source; not independently reproduced.
13316.9µs62% SOL4.47×Reported · license unknown · no source2026-07-26
Measured exactly what you asked. Reported by source; not independently reproduced.
14324.8µs66% SOL4.58×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
15441.2µs49% SOL6.22×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
16574.7µs40% SOL8.10×Reported · license unknown · no source2026-07-25
Measured exactly what you asked. Reported by source; not independently reproduced.
17607.1µs38% SOL8.56×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
18680.6µs35% SOL9.59×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
19690.9µs34% SOL9.74×Reported · license unknown · no source2026-07-25
Measured exactly what you asked. Reported by source; not independently reproduced.
20691.8µs35% SOL9.75×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.