Mixtral activated a fraction of its parameters per token

Two parameter counts for one model

Mistral AI's announcement, dated 11 December 2023, introduces Mixtral 8x7B as a sparse mixture-of-experts model with '46.7B' total parameters but only '12.9B active parameters per token'. The paper, submitted 8 January 2024, states the figures slightly differently as '47B parameters' of which 'only uses 13B active parameters during inference', and describes the mechanism: each layer contains 8 feedforward blocks, called experts, and a router network selects two of them per token, with the selected pair varying at each timestep and each layer.

Why the active-parameter figure is the operative one

The paper reports that Mixtral 'outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks', and 'vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks', while an instruction-tuned variant is said to surpass 'GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B chat' on the human benchmark the authors used. These are the authors' own reported comparisons. The architectural point behind them is that inference cost tracks the 13-billion active-parameter figure, not the 47-billion total, since only two of the eight experts run for any given token; a dense 47-billion-parameter model would carry a materially higher inference cost than Mixtral for the same total parameter count, because every parameter in a dense model runs on every token.

What sparsity changes, and what it does not resolve

Mixture-of-experts routing is not new: the Switch Transformer paper, submitted in January 2021, had already reported 'up to 7x increases in pre-training speed' from a related sparse-routing approach at much larger scale. What Mixtral's paper demonstrates is that the same principle, applied at a scale several labs could plausibly train and serve, produces benchmark results competitive with a dense 70-billion-parameter model. It does not establish that parameter count is now irrelevant to capability; it establishes that total parameter count alone stopped being a reliable proxy for either serving cost or benchmark performance once routing could activate only a subset of parameters per token. A reader comparing 'model size' across a mixture-of-experts model and a dense model is comparing two different kinds of number unless both the total and active figures are stated.

Questions to carry into your own evaluation

• When a mixture-of-experts model's size is quoted, is the figure the total parameter count, the active count per token, or unclear which?
• Does an inference-cost comparison between two models use the active-parameter figure, or mistakenly use total parameters for one and active for the other?
• Are the benchmark comparisons in the paper run under the same prompting conditions as the model you intend to compare against?

Mixtral's contribution, on its own paper's terms, is architectural rather than purely a scale claim: it reports dense-model-competitive benchmark results while running a fraction of its total parameters per token. That is a reason to ask which parameter figure any other size comparison is actually using.