A cheaper embedding model made vector search a default tool

A cheaper, unified embedding model

On 15 December 2022, OpenAI announced text-embedding-ada-002, replacing five separate first-generation embedding models with one. The announcement claimed the new model outperforms the old embedding models on text search, code search and sentence similarity tasks, citing results on the BEIR retrieval benchmarks and the SentEval classification suite, while matching prior performance on text classification. It also cut the published price by 90% compared with earlier models of the same size and by 99.8% compared with the older Davinci embedding, quadrupled the usable context to 8,192 tokens, and shrank the output vector to 1,536 dimensions. These are vendor-reported benchmark results under OpenAI's stated conditions, not an independent measurement, and a benchmark average does not describe performance on any particular collection of documents.

What an index does with the vectors

An embedding on its own is a list of numbers; retrieval requires comparing a query vector against many stored vectors and returning the closest ones. At web scale that comparison cannot be brute force, which is the problem Johnson, Douze and Jégou's FAISS paper addresses: it describes GPU designs for exact search, approximate search, and compressed-domain search using product quantization, reporting a k-nearest-neighbour graph built over 95 million images in 35 minutes and 1 billion vectors indexed in under 12 hours on four GPUs. The open-source FAISS library that resulted from this work states plainly that its compressed-vector methods trade search quality for memory: storing billions of vectors in a modest RAM budget means accepting a less precise search than an exhaustive comparison would give.

What similarity does not guarantee

Closest in embedding space is a geometric statement about the training objective that produced the vectors, not a guarantee that a retrieved passage is relevant, current or correct. Two passages can sit close together because they share vocabulary or topic while disagreeing on the fact a query actually needs, and a retrieval system built on cosine distance alone has no separate mechanism for checking that. Cost and index parameters compound this: a smaller, faster index sacrifices recall, and a benchmark score measured on BEIR's fixed tasks does not transfer automatically to a differently structured internal document set. Treating a retrieved neighbour as evidence rather than as an answer is what the rest of a retrieval pipeline exists to do.

Questions to carry into your own evaluation

• Was the embedding model's reported benchmark built from tasks that resemble your documents and queries?
• What recall is being traded away by the index's speed or memory settings?
• Does anything downstream check a retrieved passage's relevance before it reaches the model as context?

Cheap embeddings and fast approximate search made retrieval a default component of a system rather than a research luxury. They did not make the returned neighbours true.