Retrieval beat fine-tuning at adding new facts in one test

Two ways to give a model new facts

Submitted 10 December 2023, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs set up a controlled comparison between two ways of adding knowledge to a language model after pretraining: unsupervised fine-tuning on new documents, and retrieval-augmented generation, which supplies relevant passages at query time instead of updating the model's weights. The retrieval approach it compares against traces back to the 2020 RAG paper, which paired a sequence-to-sequence generator with a dense retriever over a passage index rather than relying only on what the model had memorised during training. The 2023 paper's stated aim was to test which approach injects knowledge more reliably, both for facts a base model had already encountered during training and for facts entirely new to it.

Retrieval won on both kinds of knowledge

The paper reports that retrieval-augmented generation consistently outperformed unsupervised fine-tuning, both for existing knowledge the base model had already been exposed to and for knowledge new to it, across a range of knowledge-intensive tasks spanning different topics. It also reports that fine-tuning struggled specifically to teach a model new facts it had not encountered before, and that this could be partly addressed by exposing the model to multiple paraphrased variations of the same fact during training rather than a single statement of it. This is narrower than a claim that fine-tuning does not work: the difficulty is with injecting new factual content through unsupervised fine-tuning specifically, not with fine-tuning as a technique for adjusting style, format, or task behaviour, which is a different use of the same method.

What the comparison does not settle

The paper's result describes its own tasks, its own fine-tuning setup, and models available at the time it was run; a different fine-tuning method, such as the low-rank adaptation described in the LoRA paper, updates a much smaller set of parameters than full fine-tuning and was not the specific approach under test. The decision the paper supports is narrower than a blanket rule: when the requirement is that a model answer correctly about a fact that can change or that did not exist during training, the tested evidence favours retrieving it at query time over trying to write it into the weights. Retrieval also carries costs the paper does not measure in full, including retriever quality, index freshness, and added latency per query, all of which affect whether the advantage holds in a specific deployment.

Questions to carry into your own evaluation

• Is the knowledge you need to inject static at training time, or does it change after the model is deployed?
• Are you trying to change what the model knows, or how it responds, since the paper's finding concerns the former?
• Have you measured retrieval quality on your own corpus, rather than assuming the paper's retrieval setup transfers unchanged?

The paper gives a tested reason to prefer retrieval specifically for new or changing facts, not a general verdict against fine-tuning, and the two techniques answer different questions about what a model should know versus how it should behave.