Eight authors, one Google lab, fifteen pages. The contribution is a subtraction: the machinery everybody had been building on — recurrence, convolution — is taken out, and what is left works better and trains faster.

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Vaswani et al., “Attention Is All You Need”, 2017 — abstract

The operation it keeps is one line long. Every position in a sequence is compared against every other position. The comparisons become weights. The position is then replaced by a blend of everything it was compared against, in the proportions those weights give.

Attention(Q, K, V) =
    softmax( Q K^T / sqrt(d_k) ) V

Q   what this position is looking for
K   what each position offers
V   what each position would contribute

The softmax is the part to stand in front of. It normalises the weights so that they sum to exactly one. Whatever a position is looking for, it must distribute the whole of its attention across the material actually present, in proportions, with nothing held back and nothing left unassigned. There is no weight reserved for not here.

So the mechanism cannot return the finding that the thing it was looking for is absent from the sequence. Not because the finding is suppressed — because it has no representation. The arithmetic has no symbol for it.

Order is in the same position. A thing compared against everything at once has no idea what came first; sequence is supplied afterwards, added to the input as a separate signal, because the operation has no native sense of before and after and never acquires one.

The weights sum to one. There is no reading in which the referent is missing.