Word2vec made word similarity a matter of vector arithmetic

Two papers, one line of work

On 16 January 2013 a Google team posted a paper proposing two small neural architectures, continuous bag-of-words and skip-gram, for learning a vector for every word in a vocabulary from plain unlabelled text. The stated result is that vectors trained this way score well on a word-similarity test and do so at much lower computational cost than the neural language models the paper compares against, learning from a 1.6-billion-word corpus in under a day. A follow-up paper posted on 16 October 2013 refined the skip-gram model with negative sampling as a cheaper alternative to the original softmax output, and with subsampling of frequent words, which the authors report both speeds training and produces more useful vectors.

What the vectors were shown to do

The first paper's claim is about a similarity task: vectors placed geometrically close to words judged similar by the test set, and the paper reports this outperformed the best previously published technique on that specific test. The second paper, Distributed Representations of Words and Phrases and their Compositionality, adds a distinct claim about composition: individual word vectors, simply summed, could approximate the meaning of some two-word phrases, but the authors identify idiomatic phrases such as 'Air Canada' where summing word vectors fails to capture the meaning of the whole, and propose a separate method for finding and vectorising such phrases directly.

What an embedding is not

Both papers describe a static lookup table: one vector per word or phrase, fixed after training, with no mechanism for a word's vector to change according to the sentence it appears in. The papers do not claim the vectors encode a word's meaning in every context, only that they perform well on the specific similarity and analogy tests used. Later contextual models, which the transformer and BERT lines of work made practical, address exactly this limit by producing a different vector for the same word depending on its surrounding text, a capability neither 2013 paper claims or tests for.

Questions to carry into your own evaluation

• Is an embedding being used as a fixed per-word lookup, or does it vary by context, and does the claim match which kind it is?
• What similarity or analogy test produced a quoted embedding-quality number, and does that test resemble the task at hand?
• Does a phrase-level claim rely on summing word vectors, a case the 2013 papers themselves show can fail?

Word2vec's contribution was to make word similarity computable from raw text at low cost, verified on specific similarity and analogy benchmarks. It is not a claim about contextual meaning, and describing a static, one-vector-per-word representation as capturing what a word means goes beyond what either paper measured.