series · 5 parts · in progress
Networks as Integrators
Numerical analysis as an architecture catalog: skip connections as an Euler step, momentum nets as half of Newton, and conservation laws as testable predictions about hidden states.
Start reading → Your Skip Connection Is Half of Newton- 01 Your Skip Connection Is Half of Newton A residual block x + F(x) is one forward-Euler step: depth is time, the block is a velocity, position moves directly. That is half of Newtonian mechanics. A planet does not update position from force; force updates velocity, velocity updates position, and that split is why orbits are stable. So what does the missing half cost a deep network? We let the physics make three predictions about trained networks, then check all three live in the page. One of them comes back stranger than we wrote it. JAX companion Skip Connections With Inertia, in JAX/Flax NNX
- 02 Transformers With a Velocity Ledger A pre-norm Transformer's residual stream is forward Euler: x += Attn(norm x); x += MLP(norm x). So the whole integrator dictionary transfers, and the same question follows: does a velocity ledger in the residual stream do for a Transformer what it did for a ResNet? The answer splits. On quality, four variants tie. On dynamics, the ledger changes everything: the residual-stream path through depth gets dramatically shorter and straighter, reaching the same answer by a calmer journey. Same destination, gentler road. JAX companion A Velocity Ledger for Transformers, in JAX/Flax NNX
- 03 A Network Built from Hamiltonian Steps A pendulum organizes its motion around energy, while an ordinary residual network has no comparable scalar. This post builds a residual block as a symplectic step of a learned Hamiltonian. The continuous field conserves that Hamiltonian exactly; the discrete network keeps it in the bounded oscillatory band predicted by symplectic integration. The learned pendulum stays within 0.6% where a plain field model drifts 36%, and the classifier survives being run at four times its training depth. JAX companion Building the Hamiltonian-Step Net in JAX/Flax NNX
- 04 Backprop Without the Memory Training memory is a tax nobody chose: backprop must hold every activation of the forward pass hostage until the backward pass consumes it, so depth costs memory even when it costs little compute. This post spends the invertibility of the momentum residual block: it can be run backward, so the backward pass can recompute the past instead of storing it. Measured on the same network, standard backprop's activation memory grows from 13 MB to 674 MB as depth goes 8 to 512; the reversible pass holds flat at 3.2 MB, pays 24% in step time, and returns the same gradient, until a one-line arithmetic of friction and float noise says it cannot. JAX companion Reversible Backprop as a custom_vjp in JAX
- 05 Depth on Demand The last post made depth a resolution: layers are time steps of a learned flow, and running more of them just renders the same trajectory finer. But every camera knows not to spend equal film on empty sky. This post gives a trained network the integrator's next tool, an error controller that chooses its own step size per input, with no retraining: the same weights, rendered to tolerance. The controller reproduces the reference verdicts at a fraction of the steps, its cost follows the integrator's textbook one-third power law, and the map of where it spends is a genuine surprise: effort tracks the stiffness of the learned flow, not the difficulty of the classification. JAX companion An Error Controller for a Trained Net, in JAX
This series is still being written. New parts land in the archive and the RSS feed as they publish.