A 2017 paper replaced recurrence with attention alone

An architecture built only on attention

On 12 June 2017 a team primarily from Google published the paper that introduced the transformer, a sequence model that dispenses with the recurrent and convolutional layers that had dominated translation systems. Earlier architectures processed a sentence one token after another, or convolved over a fixed window; the transformer instead lets every position attend to every other position in a single mechanism the authors call self-attention. The paper's own claim is narrow and specific: on the machine translation task it studied, an attention-only network reached higher quality with far less computation.

What the paper measured

The authors trained the model on the WMT 2014 English-to-German and English-to-French translation tasks and reported 28.4 BLEU on the German pair, more than two points above the best prior result the paper cites, and 41.8 BLEU on French. The larger configuration trained in three and a half days on eight GPUs, a fraction of the training cost reported for the strongest earlier models. A companion post from Google Research frames the same result differently, emphasising that self-attention needs only a constant number of sequential steps regardless of sentence length, which is what made the large training-time reduction possible. The paper also reports a secondary test on English constituency parsing, where the same architecture, with almost no task-specific tuning, outperformed all but one previously proposed approach.

What the evidence does not establish

The paper is a translation and parsing result, not a general claim about language understanding. It says nothing about pretraining on unlabelled text, about the scale at which the architecture would later be used, or about how it would behave on reasoning or generation tasks far removed from sentence-pair translation. The blog post's framing of self-attention as shortening the path between distant words is a claim about the mechanism, not a measured property of downstream capability. Treating this paper as evidence for anything beyond its stated tasks and BLEU scores over-reads a result the authors themselves scoped to translation and parsing.

Questions to carry into your own evaluation

• Does a cited transformer result come from the original translation setting, or from a later pretraining regime the 2017 paper never tested?
• Is a claimed training-time saving measured against a comparable hardware and model-size baseline?
• What task, dataset and metric is a 'state of the art' claim actually attached to?

The transformer's contribution was a mechanism and a training-time result on two translation benchmarks. Everything that followed, from pretrained encoders to large generative models, is a separate body of evidence built on top of that mechanism, and each later claim needs its own source.