LoRA fine-tuned large models by training small added matrices

A frozen model plus a small trained detour

In June 2021, researchers proposed a way to adapt a large pretrained model without touching most of its weights. The LoRA paper freezes the original weight matrices and inserts a pair of small, low-rank matrices alongside selected layers, typically the attention projections. Only these added matrices are trained; the original weights never move. Because the added matrices are far smaller than the layer they sit beside, the number of parameters that require gradients and optimiser state falls sharply. This is a training-time change to the weights, not a change to the model's runtime plumbing: once training is done, the low-rank update can be merged back into the frozen weights so inference uses the original architecture unchanged.

What the paper reported

The paper describes testing this approach on RoBERTa, DeBERTa, GPT-2 and GPT-3, and reports that the adapted models performed on par with or better than full fine-tuning across the benchmarks it used, while training far fewer parameters. For GPT-3 175B specifically, the paper reports roughly ten-thousand-fold fewer trainable parameters than full fine-tuning with Adam, and a threefold reduction in the GPU memory required during training, with no added inference latency because the update merges back into the base weights. The reference implementation documents the same pattern on smaller models: on GLUE tasks with RoBERTa, the released code trains roughly 0.8 million parameters against 125 million for full fine-tuning while remaining competitive with it. These are the paper's own comparisons; the tasks, model sizes and rank chosen for the adapter all shape the outcome, so the same ratio should not be assumed for a different model or task.

What the adapter cannot change

A low-rank adapter only adds capacity in the directions the training data pushes it toward; it does not give a model new knowledge it never saw, and a rank chosen too small can cap how much the adaptation can shift behaviour on a difficult task. The later QLoRA paper extended the idea by training the same kind of adapters on top of a model whose base weights are quantised to four bits, reporting that a 65-billion-parameter model could be tuned on a single 48GB GPU. That result depends on the quantisation scheme holding up as well as the adapter mechanism itself, which is a separate claim from the original LoRA paper and should be checked against its own evidence, not folded into LoRA's.

• Which layers were adapted, and does that match where the target task actually needs new behaviour?
• What rank was used, and is there evidence the task saturates it?
• Was the comparison against a full fine-tune of the same base model on the same data split?

LoRA's contribution is a training-cost claim backed by the paper's own benchmarks, not a general guarantee that any adapter will match full fine-tuning on any task.