Left: one draw. A query and four keys, each 512 standard-normal numbers, fixed
until you press new draw; only the first
d_k of them enter the dot product,
so the slider changes how much of the same vectors the score sees. Right: the
same measurement averaged over 160 independent draws, which is where the claim
actually lives — a single draw is noisy, and pressing new draw a few
times at d_k = 64 shows how much
it varies. Predict what the red curve does as
d_k grows, and whether the gold
one follows it. The callout above proves
Var(q·k) = d_k; this is what that
variance does to the softmax, and what dividing it back out undoes.