Mistral 7B claimed to beat larger models under Apache 2.0

A small model with a permissive licence

Mistral AI's announcement, dated 27 September 2023, released Mistral 7B under the Apache 2.0 licence, stating plainly that 'it can be used without restrictions'. That is a meaningfully different choice from the custom, condition-laden licences some other labs attached to open weights the same year: Apache 2.0 is a standard open-source licence with no field-of-use carve-outs. The company's own paper, submitted on 10 October 2023, describes the same 7-billion-parameter model as 'engineered for superior performance and efficiency' and documents the architecture choices behind that claim.

What was measured, and the mechanism behind the claim

The paper reports that Mistral 7B 'outperforms Llama 2 13B on all benchmarks' it tested and 'outperforms Llama 1 34B in reasoning, mathematics, and code generation'; an instruction-tuned variant is said to beat Llama 2 13B-Chat on both human and automated evaluation. These are the authors' own selected benchmark suites, not an independent leaderboard result. The efficiency argument rests on two named architectural choices: grouped-query attention, which the paper says accelerates inference, and sliding-window attention, which processes long sequences at lower computational cost than full attention over the whole context. Both are described as mechanisms for cost and speed, not as sources of the reported accuracy gains, which the paper attributes to training rather than architecture alone.

The limit of a same-size comparison

A model beating a larger one on selected benchmarks says something about training data and method, but it does not establish that the smaller model matches the larger one on tasks outside the tested suite, nor that the comparison models were tuned to the same standard. Mistral's own announcement is explicit that the reference points are Llama 2 13B and Llama 1 34B specifically, not a general 'best small model' claim, and the paper does not report results against every contemporary model of comparable size. Sliding-window attention also changes what 'context' means internally: each layer can attend only to a fixed window of recent tokens rather than the entire input, which is a design trade-off for speed that a task requiring reference to very distant tokens could expose.

Questions to carry into your own evaluation

• Are the benchmarks in a 'beats a larger model' claim the same ones your task resembles, or a different suite chosen by the paper's authors?
• Does sliding-window attention's fixed context per layer matter for a task that depends on information near the start of a long input?
• Is the Apache 2.0 licence still attached to the checkpoint you are downloading, or has a fine-tuned derivative changed the terms?

Mistral 7B's contribution, on the paper's own terms, is a training and architecture recipe that let a 7-billion-parameter model win selected comparisons against larger ones, released without the licensing conditions that accompanied some contemporaries. Both claims are narrower than 'best 7B model available', which the paper does not assert.