Skip to content
KernelIndex
Search⌘K
Sign in

Fused linear + cross-entropy

16 eligible runs
cross-entropy

LM-head projection of tokens×hidden activations through a vocab×hidden weight fused with cross-entropy against integer targets, without materializing the logits.

Best usable
119.5msmedian · 5.72× vs fastest
LigerLiger-Kernel · BSD-2-Clause · triton

Reported evidence · last observed 2024-11-22. Reported by source; not independently reproduced.

pip install "liger-kernel==0.4.2"
View source →Run detail →
Source baseline · unbeaten · not usable as-is
20.9msmedian
TransformersHugging Face Transformers · Apache-2.0 · python

Reported evidence · last observed 2024-11-22. The source's designated baseline implementation. Reported by source; not independently reproduced.

No package · vendor the source →View source →Run detail →

Fastest by languagepython · Transformers · 20.9 ms · #1triton · Liger · 119.5 ms · 5.72×

Current records

Workloadvocab = 128256 · hidden = 4096 · tokens = 4096 · bf16vocab = 128256 · hidden = 4096 · tokens = 8192 · bf16vocab = 128256 · hidden = 4096 · tokens = 16384 · bf16vocab = 128256 · hidden = 4096 · tokens = 32768 · bf16
HardwareBest knownImplementationRuns
NVIDIA A100 · env 120.9 msTransformers2NVIDIA A100 · env 256.0 msTransformers2

Not measured on H100, B200 for this workload. Challenges →

Source-native comparison · GPU NVIDIA A100 · Workload vocab = 128256 · hidden = 4096 · tokens = 4096 · bf16 · Protocol Liger-Kernel benchmark scripts · median · 2 results · last observed 2024-11-22Record history →
Estimated floor 531.8 µs · record 39.33× above itestimate, not evidence ›
DRAM 531.8 µs · bandwidth-bound on A100 80GB
every declared tensor crosses HBM exactly once (2,039 GB/s, A100 80GB datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1
TransformersbaselineHugging Face Transformers
20.9ms
1.00×
Reported · Apache-2.0 · source
2024-11-22stale

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredApache-2.0no install recipeView source →Run detail →
2
LigerLiger-Kernel
119.5ms
5.72×
Reported · BSD-2-Clause · source
2024-11-22stale

5.72× slower than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →

Scaling by tokens

100.0 ms40968k16k32ktokens →20.9 ms38.0 ms76.4 ms151.9 msTransformers119.5 ms168.1 ms274.1 ms508.5 msLiger
TransformersLigerbest per workload · NVIDIA A100 · Liger-Kernel benchmark scripts held constant · log scale

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
TransformersHugging Face Transformers
python
20.9ms
1.00×
Reported
Apache-2.0 · source
LigerLiger-Kernel
triton
119.5ms
5.72×
Reported
BSD-2-Clause · source

Semantics

Inputs and outputs
xfloat [tokens, hidden]
weightfloat [vocab, hidden]
targetint64 [tokens]
lossfloat [1]
Axes and behavior
vocabvariable
hiddenvariable
tokensvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha2564713ab4d824f…
Sources: Liger-Kernel benchmarks (2024-11-22) · BSD-2-Clauselast observed 2024-11-22How records are decidedJSON