Original paper
Universal Transformers
A reading note on the work cited above. Explanations and reading questions are identified separately from the paper’s reported findings.
Two questions about computation
A useful reading lens separates communication from computation: which representations exchange information, and how often are they updated? The original Transformer organizes attention within a layered architecture. These two questions provide a starting point for understanding shared computation. Parameter count describes only one resource used by a model.
A shared update, applied repeatedly
UT repeats shared attention and transition modules across depth, updating sequence positions in parallel. Optional adaptive halting gives different positions different iteration counts.
Illustration: applying the same “add one” rule four times changes a state without introducing four independently parameterized rules. This simple example separates the number of stored rules from the number of executed operations. Reusing a block still requires running it each time.
Evidence and its scope
The reported algorithmic and language experiments improve on the tested Transformer baselines. Adaptive halting slightly hurts machine translation. The computational-universality argument permits sufficient memory and computation that can grow with input length.
Reading implication: experimental performance, computational expressivity, and generalization under a fixed budget are distinct questions. Evidence answering one should be identified precisely when discussing another.
Questions for a new comparison
For a new experiment, ask what is held equal: trainable parameters, executed updates, total training work, or inference latency? Then ask which modules share weights and whether stopping rules are tuned using held-out data. These are proposed reading questions, not additional results established by this paper.
Sources & further reading
Universal Transformers · paper and publication historyUniversal Transformers · full text, §§2–4 and Appendix BAttention Is All You Need · original Transformer paperAdaptive Computation Time for Recurrent Neural NetworksContinue reading