Skip to content
KernelIndex
Search⌘K
Sign in

Embedding lookup

96 eligible runs
embedding

Embedding table lookup: batch×seq integer ids gathered from a vocab×hidden weight.

Best usable
42.6µsmedian · 1.07× vs fastest
LigerLiger-Kernel · BSD-2-Clause · triton

Reported evidence · last observed 2024-09-03. Reported by source; not independently reproduced.

pip install "liger-kernel==0.2.1"
View source →Run detail →
Fastest reported · not usable as-is
39.7µsmedian · 4.20× faster than baseline
torch.compilePyTorch · BSD-3-Clause · python

Reported evidence · last observed 2024-09-03. Reported by source; not independently reproduced.

No package · vendor the source →View source →Run detail →

Fastest by languagepython · torch.compile · 39.7 µs · #1triton · Liger · 42.6 µs · 1.07×

Current records

Workloadseq = 512 · batch = 32 · vocab = 1024 · hidden = 768 · fp3216 cases
vocab
batch
hidden
seq
dtype
1024840962048fp32102432768512fp322048840962048fp32204832768512fp324096840962048fp32409632768512fp328192840962048fp32819232768512fp3216384840962048fp321638432768512fp3232768840962048fp323276832768512fp3265536840962048fp326553632768512fp32131072840962048fp3213107232768512fp32
scrolls · 16 cases total
HardwareBest knownImplementationRuns
NVIDIA A100 · env 139.7 µstorch.compile3NVIDIA A100 · env 2300.1 µstorch.compile3

Not measured on H100, B200 for this workload. Challenges →

Source-native comparison · GPU NVIDIA A100 · Workload seq = 512 · batch = 32 · vocab = 1024 · hidden = 768 · fp32 · Protocol Liger-Kernel benchmark scripts · median · 3 results · last observed 2024-09-03Record history →
Estimated floor 1.61 µs · record 24.71× above itestimate, not evidence ›
DRAM 1.61 µs · bandwidth-bound on A100 80GB
every declared tensor crosses HBM exactly once (2,039 GB/s, A100 80GB datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1
torch.compilePyTorch
39.7µs
1.00×
Reported · BSD-3-Clause · source
2024-09-03stale

4.20× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-3-Clauseno install recipeView source →Run detail →
2
LigerLiger-Kernel
42.6µs
1.07×
Reported · BSD-2-Clause · source
2024-09-03stale

3.91× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →
3
TransformersbaselineHugging Face Transformers
166.7µs
4.20×
Reported · Apache-2.0 · source
2024-09-03stale

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredApache-2.0no install recipeView source →Run detail →

Scaling by vocab

50.0 µs100.0 µs150.0 µs10242048409616k32k64k128kvocab →39.7 µs46.5 µs54.6 µs60.2 µs64.1 µs66.9 µs69.0 µs69.2 µstorch.compile42.6 µs46.7 µs49.4 µs55.6 µs61.5 µs65.2 µs67.8 µs70.9 µsLiger166.7 µs144.1 µs153.9 µs162.7 µs166.7 µs168.8 µs169.2 µs170.3 µsTransformers
torch.compileLigerTransformersbest per workload · NVIDIA A100 · Liger-Kernel benchmark scripts held constant

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
TransformersHugging Face Transformers
python
144.1µs
3.63×
Reported
Apache-2.0 · source
torch.compilePyTorch
python
39.7µs
1.00×
Reported
BSD-3-Clause · source
LigerLiger-Kernel
triton
42.6µs
1.07×
Reported
BSD-2-Clause · source

Semantics

Inputs and outputs
input_idsint64 [batch, seq]
weightfloat [vocab, hidden]
yfloat [batch, seq, hidden]
Axes and behavior
seqvariable
batchvariable
vocabvariable
hiddenvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha25625fa6855b83f…
Sources: Liger-Kernel benchmarks (2024-09-03) · BSD-2-Clauselast observed 2024-09-03How records are decidedJSON