The sequence-to-sequence bottleneck
A model reads a string, then tries to copy it. Its encoder must pass the entire input to its decoder through one fixed-size vector. As the string grows, when will copying start to fail?
The decoder receives only this vector, then generates from its own previous output.
Recorded outputs for the same input. Highlighted, underlined characters differ from the original; ∎ is an early stop symbol. Scores here describe this example.
No recorded example at this length. Choose 10, 25 or 40 above; the chart has measurements at every length.
Each output is generated from the model’s previous output. Accuracy is measured on its first L outputs, where L is the input length. “Entire string correct” requires all L positions to match; it does not test whether the model stops correctly afterwards.
The no-input baseline is a separately trained width-128 decoder given a zero vector. Its score measures what it achieves without input information. The accuracy gain over this baseline is not a measurement of information carried. Random digits can still match by chance. English is predictable, but its alphabet is larger: it does not necessarily score better at every width or length.
These are measured results from trained models, not a theoretical capacity limit. One number in the context vector does not correspond to one stored character. Changing memory size selects another trained model. The attention comparison always uses width 16 and accesses all encoder states.
Loading measured results…