BIG-bench showed some skills appear abruptly with scale

Two hundred tasks from 450 authors

The Beyond the Imitation Game paper, submitted to arXiv on 9 June 2022, describes BIG-bench as a collaborative benchmark assembled from 204 tasks contributed by 450 authors across 132 institutions. Contributors were asked to submit tasks they believed current language models could not yet do well, covering domains from linguistics and mathematics to biology, physics and software development. The project repository frames the goal as probing models and extrapolating their future capabilities rather than measuring a single skill, a different aim from a benchmark like MMLU that samples existing exam questions.

What the paper measured, and what it found about scale

Every task was also given to human expert raters, so the paper could compare model scores against a human baseline rather than only against chance. It reports that model performance and calibration both improved with scale but remained poor in absolute terms next to rater performance, and that the pattern was broadly similar across different model families, with some benefit from sparse architectures. The paper separates two kinds of task behaviour: most tasks improved gradually as models grew, in line with what scaling curves generally predict, while a smaller set showed sudden breakthrough performance at a particular scale, concentrated in tasks that required multiple steps or used metrics sensitive to small wording changes. The paper is explicit that this is a description of an observed pattern in its own task set, not a general law about all future tasks.

What BIG-bench Hard later selected, and the limit that implies

A follow-up paper, Challenging BIG-Bench Tasks, submitted 17 October 2022, picked out 23 tasks specifically because prior model evaluations had not beaten the average human rater on them, naming this subset BIG-bench Hard. Applying chain-of-thought prompting rather than direct answering let PaLM exceed the human-rater average on 10 of those 23 tasks, and let a code-focused Codex model exceed it on 17. That result says as much about prompting technique as about the underlying model: the same weights scored differently depending on how the question was asked, which limits how far a flat BIG-bench or BBH score can be read as a fixed property of a model.

Questions to carry into your own evaluation

• Is a cited BIG-bench result from the full 204-task set, or from the BBH subset selected for being hard for prior models?
• Was the score produced with direct prompting or chain-of-thought, given how much that changed results on BBH?
• Does the task in question resemble the kind of multi-step, brittle-metric task where the paper found abrupt rather than gradual improvement?

BIG-bench's value was in crowd-sourcing difficulty and comparing models against human raters at scale, but its own authors and the BBH follow-up both show that a headline score depends heavily on which tasks are included and how the model is prompted.