Switch Transformer added parameters without adding compute

Route each token to one expert, not all of them

Standard transformers send every token through the same dense feed-forward layer. The Switch Transformer paper, submitted in January 2021, replaces that layer with many separate expert sub-networks and a router that sends each token to a single expert, rather than combining several as earlier mixture-of-experts designs did. Because each token still only activates one expert's worth of computation, the total parameter count can grow by adding more experts without a matching increase in the floating-point operations spent per token. The paper describes a model with an outrageous number of parameters but constant computational cost per token, echoing the framing of an earlier sparsely-gated mixture-of-experts paper from 2017 that first proposed routing tokens to a subset of thousands of expert sub-networks.

What the paper reported

Using T5-Base and T5-Large as the dense baselines, the paper reports up to a sevenfold pre-training speedup for equivalent computational resources, and a fourfold speedup when compared against T5-XXL. It reports training sparse models reaching approximately a trillion parameters on the Colossal Clean Crawled Corpus, improvements across all 101 languages tested against a multilingual T5 baseline, and the first published training of a large sparse model in bfloat16 rather than higher-precision formats. These are pre-training speed and multilingual quality comparisons under the paper's own training budgets; they are not evidence about inference cost, which is a separate question from training efficiency.

What sparse capacity costs elsewhere

A router that sends different tokens to different experts creates a model whose parameters must mostly stay resident in memory even though only a fraction compute on any given token, so total memory footprint scales with the full parameter count rather than the active count. Serving such a model also requires routing tokens to the right expert efficiently, often across multiple devices, which is a distinct engineering problem from serving a dense model of similar active size. The 2017 sparsely-gated paper had already reported this trade-off at smaller scale, describing over a thousandfold capacity increase with only a minor efficiency cost on the hardware it used; the Switch Transformer paper extends the same trade-off to language model pretraining specifically. A later paper on a smaller sparse mixture-of-experts model shows the same one-expert-per-token routing pattern used in a released, served model, though it does not itself revisit the Switch Transformer's training comparisons.

• Is a quoted speedup measured against a dense model of equivalent active or equivalent total parameters?
• Does the deployment budget account for holding all experts in memory, not just the active ones?
• What load-balancing mechanism did the paper use, and does it apply to your training setup?

The paper's evidence supports a training-compute argument for sparse routing under its own budget and corpus; it does not by itself establish inference cost or serving simplicity, which depend on separate engineering choices.