Original paper

Deep Equilibrium Models

Shaojie Bai · J. Zico Kolter · Vladlen Koltun

NeurIPS 2019 · 2019Read the paper ↗

A reading note on the work cited above. Explanations and reading questions are identified separately from the paper’s reported findings.

What must a training run remember?

Consider a long calculation in which each step changes a hidden state. A useful design question is whether learning requires keeping the whole path, reconstructing it later, or differentiating a relationship satisfied by the final state. These possibilities lead to different memory and computation costs.

Solve the representation equation

DEQ defines a representation by z* = fθ(z*; x). A root finder approximates this equilibrium, and implicit differentiation computes gradients without retaining every intermediate activation.

Interpretation: the equation specifies the target; a solver specifies how to approach it. Two solvers can target the same equation while using different numbers of steps or stopping at different residual errors. A comparison should therefore describe both the transformation and the numerical procedure.

Memory savings have a precise meaning

The sequence experiments achieve competitive accuracy with activation memory independent of effective depth. The reported DEQs are generally slower than the compared explicit networks. Solver tolerance affects accuracy. Constant memory here refers to depth, not sequence length, width, or batch size.

What to record in an evaluation

Suggested evaluation questions: how small is the residual when the solver stops, what happens at the iteration cap, and how variable is inference time across inputs? Record peak memory and latency separately. An attractive point on one resource axis need not dominate every alternative on all axes.

Sources & further reading

Deep Equilibrium Models · NeurIPS 2019Deep Equilibrium Models · full paper, §§3 and 5The Reversible Residual Network: Backpropagation Without Storing ActivationsStabilizing Equilibrium Models by Jacobian Regularization

Continue reading

Reusing a Transformer block