BERT made pretrain-then-fine-tune the default recipe

A bidirectional model prepared once, then adapted

Google researchers posted the BERT paper on 11 October 2018 and open-sourced the code and checkpoints on 2 November. The paper's proposal is a pretraining objective that conditions on both left and right context in every layer, rather than reading a sentence in only one direction. The stated benefit is that the same pretrained network can then be adapted to a new task by adding one output layer and fine-tuning, without redesigning the architecture for each task.

What the paper reports

On the GLUE collection of language-understanding tasks the paper reports 80.5% average accuracy, a 7.7 percentage-point improvement over the best prior result it cites, with a 4.6-point gain on MultiNLI. On the Stanford Question Answering Dataset it reports 93.2 F1 on version 1.1 and 83.1 F1 on version 2.0, improvements of 1.5 and 5.1 points over the systems it compares against. The open-sourcing post repeats the SQuAD 1.1 figure, noting it surpassed a prior score of 91.6, and adds a practical claim: fine-tuning a state-of-the-art question-answering system from the released checkpoint takes about 30 minutes on a single Cloud TPU, or a few hours on one GPU. The code repository, as retrieved on 16 September 2026, backs this with checkpoints under an Apache 2.0 licence, including smaller distilled-size models the project kept adding as late as March 2020.

What changed after this result

BERT's numbers are fine-tuning results: each benchmark score reflects further training of the pretrained model on labelled examples for that specific task, not one frozen model answering every task unaided. That distinction matters because practice later moved away from fine-tuning separate copies of a model, toward prompting a single frozen model at inference time. The repository's later additions of smaller checkpoints are themselves evidence the original release did not settle the size-versus-accessibility question; researchers kept adding cheaper variants for years afterward. Nothing in the 2018 paper measures prompting, few-shot behaviour or open-ended generation, since none of those were how the model was evaluated.

Questions to carry into your own evaluation

• Is a reported BERT-era score the result of task-specific fine-tuning, or of the frozen model alone?
• Which GLUE or SQuAD version and split does a comparison actually use?
• Does a claimed speed or cost figure specify the hardware it was measured on?

BERT demonstrated that one bidirectionally pretrained network, fine-tuned per task, could beat systems built and tuned separately for each benchmark. It is a fine-tuning result, not evidence about prompting or zero-shot use, and its GLUE and SQuAD numbers should be read against the exact task setup the paper describes.