A Network Built from Hamiltonian Steps

A pendulum in a vacuum swings around one organizing quantity: its energy. An ordinary deep network has no comparable scalar; a hundred layers in, the hidden state may have drifted anywhere the updates send it. But if a residual block is one step of a flow, its update can inherit Hamiltonian structure. The continuous field then conserves a learned energy exactly, while a symplectic discretization keeps the numerical energy error bounded instead of letting it drift. This post builds both objects and asks what that structure buys a classifier that never requested any physics.

The fit that should have been enough#

Every modeling instinct says the pendulum problem is already solved. You want to learn a dynamical system, so you collect states $(q, p)$, angle and momentum, and you train a network to predict how each state moves: $F_\theta(q, p) \approx (\dot q, \dot p)$. This is supervised learning at its most comfortable. Our plain MLP drives the field error down to a mean squared error of 4.4e-05 on held-out states. The arrows it draws are, to the eye and to three decimal places, the pendulum’s arrows.

So integrate it. Start a pendulum at a healthy swing, step it forward with the learned arrows, and let it run for sixty seconds of pendulum time. The trajectory it produces is not the pendulum. It is a spiral wearing a pendulum costume: each loop is slightly wrong, the loops do not close, and by the end of the rollout the model’s swing carries 36% more energy than it started with, on its way to more. A model that was right to three decimal places everywhere has become confidently wrong, with the error compounding at every step.

Where did the error come from, if every arrow was almost perfect? From the word almost, and from the fact that nothing polices the sum. Each step’s error is tiny and essentially random, but random errors in a free model have a full plane to wander in, and a random walk does not stay home. The energy of the learned trajectory is a number nothing defends, so it drifts, and every drift compounds into the next loop’s geometry. The real pendulum faces the same buffeting from truncation and floating point, and does not care. Which should stop you: what does the real system have that the learned one lacks?

Not smaller errors. A missing dimension. The true pendulum’s motion is chained to a level set of its energy: the state can slide along the set, but the physics offers no direction that leaves it. An error can only push you where movement is possible, and off the level set is not a place movement exists. The free model has no such chain. Its two outputs can point anywhere, so its errors can too.

Learning one number instead of two#

That diagnosis suggests a strange repair. If the disease is freedom, remove it: do not let the network draw arrows at all. Let it learn a single scalar $H_\theta(q, p)$, one number per point of phase space, and define the motion as the rotation of that scalar’s gradient by ninety degrees:

This is Greydanus, Dzamba, and Sprague’s Hamiltonian Neural Network (2019). Count what it gives up. A free field on 2-D phase space is two functions’ worth of expressive freedom; this is one function, and not even that function’s own gradient, but its gradient turned sideways. The network cannot decide where states go. It can only decide what the terrain is, and the terrain decides where states go: along its contour lines, always, because a rotated gradient is by definition perpendicular to the direction in which the scalar changes.

You would expect to pay for a constraint like that in fit, and this is the first surprise the run hands us: the constrained model fits the arrows better, 3.1e-05 against the free model’s 4.4e-05. In hindsight it is not magic. The pendulum’s true field actually is the rotated gradient of a scalar, so the constraint is not a handicap, it is information: the HNN searches a smaller space that happens to contain the answer. But nothing in the fit numbers prepares you for what happens dynamically. Watch both models run, live, from the same push.

Over the full rollout the free model drifts 36%; the HNN stays within 0.6%. The mechanism begins in the vector field: its instantaneous direction is tangent to a contour of the learned surface, so the continuous dynamics cannot cross that contour. Numerical integration adds a second question. RK4 does not preserve the learned energy exactly, but here its discretization error remains small enough that the structural advantage is visible across the rollout.

The one-line proof is worth seeing after the picture has made it obvious. Along the learned motion,

identically, for every $\theta$, trained or random. The identity belongs to the continuous vector field before training starts: gradient descent chooses the scalar, while the symplectic-gradient wiring makes its directional derivative zero. A numerical rollout approximates that field, so its energy follows a narrow error band rather than an exact constant. The architecture supplies the invariant; the integrator determines how faithfully the discrete trajectory tracks it.

The terrain is the knowledge#

So the HNN does not store the pendulum’s motion at all. It stores a surface, and the motion is a consequence. That surface is sitting in the trained weights right now, and since phase space here is honestly two-dimensional, no projection tricks are needed to look at it, we can just paint it.

This is what it means for the scalar to be the model. Ask the HNN what motions are possible and it hands back a topographic map rather than a rollout: every stripe is an orbit, the closed stripes around the center are swings, and the model’s entire dynamical repertoire fits in one picture. The plain field model has no such picture. Its knowledge is a thousand local arrows with no global object tying them together, which is why nothing ties its rollout together either.

A fair objection lands here: this is a pendulum, two dimensions, a physics toy. Networks that classify do not live on maps. The rest of this post is about how far that objection survives.

A classifier that swings#

The objection is weaker than it looks. If depth is time and the residual block is a field, then a classifier already is a dynamical system: the encoder sets an initial condition, the stack flows it, the head reads where it landed. The velocity ledger upgraded that flow to second order. Nothing in the dictionary says the flow cannot be Hamiltonian. So build it.

Split the hidden state into two halves, position and momentum, $z = (q, p)$ with each half in $\mathbb{R}^4$, and let the only learned object in the block be a scalar potential $V_\theta(q)$ over the position half. One layer is one leapfrog step of size $h = T/L$, the workhorse discrete integrator of Hamiltonian mechanics:

Kick, drift, kick: half a push from the terrain, a glide, half a push. The same shared block repeats at every layer, an encoder lifts each input onto the terrain, a linear head reads the final state, and the whole thing trains by ordinary gradient descent on cross-entropy. Every abstract piece here has a physical name, and the translation is the same dictionary as before:

Landscapes, wells, things that roll: the physical world here is the one the editing post already put you in. What has changed is that the landscape is no longer a reading of the model, it is the model. Training a classifier now means sculpting a terrain such that marbles thrown from different classes swing to separable places. And unlike the pendulum’s phase plane, this state space is eight-dimensional, so when the panels below show you a plane of it, that plane is a window, not the room.

The test of a real metaphor is whether it predicts. First: leapfrog should keep $E_\theta$ inside a bounded oscillatory band, and shrinking the step should shrink that band at the method’s second-order rate. Second: “sixteen ticks” and “sixty-four finer ticks” should approximate the same total-time journey. A leapfrog classifier trained at depth 16 should therefore survive being run at depth 64, where repeating an ordinary sixteen-layer network may change the computation entirely. The run checks both predictions.

The scoreboard, and the trap in it#

What does the constraint cost? On three 2-D benchmarks against a parameter-matched plain residual net, three seeds each: moons 100.0% versus 99.9%, rings 100.0% versus 99.9%, spirals 98.8% versus 92.9%, that last with a ±5.4-point seed spread on the constrained side. Read the scoreboard cold and the conclusion writes itself: the conservation law is free on easy tasks and costs six points on the hard one. If accuracy is the only column you keep, the constraint looks like a tax, and there is no argument to make with the table; the claim it supports is the existence proof, a network with a hard conservation law in its hidden state trains by plain gradient descent and lands in the pack.

Accuracy is the wrong instrument for the rest of it. A conservation law is a claim about what stays true of the computation between the encoder and the head, and a test-set percentage is only ever looking at the head. So look between.

The first prediction lands at the expected scale. At depth 16, where each layer is a coarse $h = 0.375$ tick, the learned energy oscillates in a band of 0.05 to 0.19 across the datasets. Refine to depth 64, four times smaller $h$, and the band contracts to 0.003 to 0.011, a factor of sixteen to twenty-two. Sixteen is $4^2$: the measured change matches the $h^2$ error law of a second-order symplectic integrator and the shadow-Hamiltonian picture. The block carries that numerical structure into the trained classifier.

Two boundaries of the law, measured by the same run. The state norm grows through depth in both networks, by similar factors; the law conserves $E_\theta$, not $\lVert z\rVert$, and anyone hoping conservation means bounded activations is hoping for the wrong theorem. And at real step sizes the conservation is the discrete, banded kind; the exactly-zero identity belongs to the continuous flow. The plain network is not missing a tighter band, it is missing the row: with no scalar in the wiring there is nothing whose band you could report.

The journey, rendered finer#

The second prediction is the one with consequences you can spend. Both networks were trained at depth 16 with total time $T$ fixed, so depth is not “how much network” but “how finely the same journey is rendered”. Run the trained weights at depth 64 and a marble takes the same roll in smaller ticks. If the leapfrog net really learned a terrain, its answers should barely move. If the plain net learned sixteen specific compositions, at sixty-four it is a different function wearing the same weights.

At four times its training depth the plain net’s spirals accuracy falls from 98.8% to 68.0%, thirty points of the scoreboard gone, and on rings it drops from 100.0% to 86.0%. The leapfrog net goes from 92.9% to 92.4% on spirals and from 99.9% to 99.8% on rings. The six-point tax buys a network whose depth axis can be turned after training, and it reveals why the two accuracy columns were never comparable in the first place: the two networks were never playing the same game. One learned a function of sixteen specific steps. The other learned a flow, and sixteen was just the resolution it happened to be rendered at.

There is a rhyme here with the weight-tied fixed-point network, which extrapolated by iterating longer because its depth was repetition of one operator. The rhyme sharpens the point rather than repeating it. Both networks here share their block across depth, the plain one too, and sharing alone did not save it. What survives the depth change is the integrator: fixed total time, steps that mean time, a block that is a physical tick rather than a free function. The extrapolation lives in the numerics, not the parameter count.

What a law is worth#

So what is the law worth, all told? Not accuracy: two tables tie and the free net takes the third by six points at the training depth. Not a bound on the state norm either. What it gives is structural, and structure compounds. A named, watchable invariant where the plain net has undefined. A depth axis that means resolution, cheap to raise after training, with the failure mode of composition removed. And one more property, paid for by the same geometry, that this post has not spent: a leapfrog step, like the velocity ledger’s step, is algebraically invertible. Kick, drift, kick can be undone: subtract the second kick, glide back, subtract the first. A network whose every layer can be run backwards does not need to store what its layers computed; the backward pass of training can reconstruct activations instead of memorizing them, and the memory bill of depth collapses. That is the next post.

The shelf this block came from is long. Numerical analysis spent a century building integrators that preserve energy, volume, reversibility, time symmetry, each one debugged, each one now a candidate residual block, each one a testable claim about what a trained hidden state does. A pendulum swings for a thousand years on one conserved number, and it took one afternoon of gradient descent for a classifier to learn a number of its own. Which physics a network should be handed has become a design decision, and the shelf is fully stocked.

Provenance: one run in JAX and optax on Kaggle (bundle kgl_blog-hamiltonian-v2) produced the pendulum field fits, the RK4 rollouts, and the three-dataset, three-seed network comparison. The panels above step the real exported weights as you watch them. The runnable companion builds both networks in Flax NNX.

@misc{bouhsine2026anetworkthatconservesenergy,
  author       = {Bouhsine, Taha},
  title        = {A Network Built from Hamiltonian Steps},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/a-network-that-conserves-energy/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}