Your Network Is a Stack of Layers. It Could Be a Fixed Point.

Before a deep network has seen a single example, you have already made its biggest decision for it: how deep. Six layers, twelve, ninety-six. Each one gets its own slab of weights, and you commit to the whole tower up front, the same height for a trivial input and a hard one, because the architecture is fixed and the input has no say. If you guess too shallow the network cannot think long enough; too deep and every input pays for depth it never needed. Depth is a hyperparameter you tune in the dark.

There is a different way to build the thing, and it starts by refusing to choose. Take one layer and apply it again, and again, feeding its output back into itself. Now “depth” is just how many times you went around, and you no longer have a stack of distinct layers, you have a single operation and a loop. The network stops being a tower of weights and becomes something stranger and cleaner: an equation, whose answer is the state that no longer changes when you apply the operation once more. Not computed forward through layers, solved for.

This post builds that network out of the one piece the rest of this series has been about. Swap a neuron’s activation for the Yat kernel and it becomes a prototype, a point in space you can point at; a network of them is a list you can edit by hand. Share one such operator across all of depth and something new falls out: the answer becomes a fixed point, depth stops being your decision at all, and it stays as legible as it was. Every number below is from a real run (scripts/yat_deq.py), 98.2% on two interleaving moons.

One operator, applied until it stops#

What would it even mean for a network to have no layers? Start from a Yat block, the same one from the last two posts: a bank of $M$ prototypes $W$, and a similarity of the running state $z$ to each of them,

An ordinary deep net would stack $L$ of these, each with its own fresh prototypes. Refuse that. Keep one set of prototypes and one mixing matrix $A$, and wire the block to read its own output. Injecting the input $x$ at every turn so it is never forgotten, one step of the network is

That is the whole network. Not a layer of it, all of it. To see why that is a recursion and not a stack, unroll it. A depth-$L$ network is the same map applied $L$ times,

with the same $W$ and the same mixing $A$ at every step. An ordinary deep network is the same shape but with the sharing switched off: every layer gets its own weights,

a different $A_\ell$ (a different alpha) and different prototypes per layer, $L$ separate transforms welded in a row. Our network is the special case $A_\ell \equiv A$, $W_\ell \equiv W$: one operator, reused. That is the only change, and it is the load-bearing one, because a finite stack of different maps has nothing to converge to, while a single map applied over and over has a limit, and the limit is a fixed point. A twelve-deep version and a fifty-deep version become the same three matrices $(W, A, U)$; depth is only how many times you go around. Keep going until the state stops moving and you have reached a point $z^\*$ that the operator maps to itself,

The reading-off at the end is an ordinary linear layer on the settled state. Everything interesting is in that fixed-point equation. The network’s answer is not the output of a last layer, because there is no last layer. It is the state the operator can no longer change, and to get it you drop $z$ in at zero and apply $F$ until it comes to rest.

How different are a loop and a stack, really? Drawn side by side, they are the same picture with one wire changed. In the top row below is this network, one operator box with its output wired straight back to its input, and the coloured strip is the real state after each turn. The bottom row is the ordinary alternative, the same computation with the sharing switched off: a fixed row of separate operators, each with its own alpha, that you pass through once. The loop reuses a single $W$ and $A$ and can turn as many times as the input needs; the stack has exactly as many weights as it has layers, and stops when it runs out of them.

Watch one input do exactly that. The state starts at the origin and each turn of the operator moves it; the left panel is a flat window onto the 32 dimensions it actually lives in, with every input’s resting point drawn faintly so you can see where this one is heading. The right panel is the size of each step. When the steps vanish, the state has arrived, and that resting point is the answer.

Is that resting point trustworthy?#

A resting point is only a good answer if it is the answer. Two worries should nag at you. Does the state settle at all, or could it wander forever, so that “run until it stops” never stops? And if it does settle, does it settle in the same place no matter where you dropped it in, or does the answer secretly depend on the arbitrary choice to start at zero? An equation is only worth solving if it has one solution the solving reliably finds.

Both worries are the same property in disguise, and it has a name. The operator is a contraction if applying it to two states always brings them closer together: $\lVert F(z) - F(z') \rVert \le \rho \lVert z - z' \rVert$ for some $\rho < 1$. When that holds, Banach’s theorem hands you everything at once. There is exactly one fixed point, and iterating from any starting state marches straight to it, the distance to it shrinking by a factor of at least $\rho$ every step. No wandering, no dependence on where you began.

This is the same physical picture the editing post built, seen from a new angle. There, prototypes were masses and an input fell into the deepest basin of their field. Here the iteration is the falling: $F$ defines a flow on state space, and a contraction means that flow has a single basin with a single point at the bottom. Drop the state in anywhere and it rolls to the same rest. The contraction factor $\rho$ is how steep the basin is, how fast the rolling converges.

Nothing forces a network to be a contraction, though, and the raw operator is not one. It has to be trained into it, by penalizing the operator’s stretch (its largest local slope $\lVert J_F\rVert$) until it drops below one. After training, the measured slope at the resting point averages 0.66 and tops out at 0.92 across the test set: a genuine contraction. So the picture below is not a hope, it is what the trained operator does. Fix one input, start the state from four wildly scattered places, and watch.

The straight line on the right is the whole argument. Every step multiplies the gap between the starts by roughly the same factor, the signature of a contraction, and a geometric decay reaches zero fast. That is also the answer to the first worry: the residual on each input drives all the way down to about $4 \times 10^{-7}$, the state truly stops, it does not merely slow down.

One thing should still bother you, though: we trained this operator into a contraction, which means gradients flowed through it. Through what, exactly? There is no stack of layers to backpropagate through, and no fixed number of turns to unroll. Hold that question; the depth story has to pay out first.

Then how deep is the network?#

The question we opened with has not gone away. A stack made you answer “how deep?” before the data had a say, and the loop seems to dodge the question, but it has only moved it: some number of turns still gets spent on every input, and something is choosing that number. So what does each turn actually buy? The dial below applies the shared operator a chosen number of times to every point of the input plane at once, and colors each point with the decision the network would give if it stopped there. For the first few turns the boundary is still forming, the two moons bleeding into each other. Then watch what happens once it snaps into place.

Past the snap, more turns change nothing. The boundary freezes, the accuracy goes flat: the classification is settled by about the sixth turn, long before the state has microscopically stopped, and it never changes afterward. Further depth is wasted motion. So a fixed depth was always the wrong question. Ask instead: has the answer stopped moving? When it has, halt. A twelve-layer stack cannot ask that, its depth is welded in. This network can, because its depth is a loop with a natural exit condition, the residual falling below a threshold.

Who decides when it stops?#

If the network halts when its answer stops moving, who decides when that happens? Not the architecture, and not you. Some inputs are easy: they sit deep inside a class, far from any ambiguity, and the state drops into the basin in a handful of turns. Others are genuinely hard, caught between the two moons where the answer is a close call, and the state takes longer to commit. Which hints at something a stack could never offer: a different depth for every input.

Give the solver a depth budget and see who has finished. As the budget grows, the interiors of the two classes fill in first, the confident inputs, and the last points still iterating are precisely the ones hugging the boundary. The histogram on the right is the real distribution of turns-to-settle across the test set: most inputs are done in around twenty, the easiest in fourteen, and a lone ambiguous straggler needs sixty-five.

That is adaptive computation with no extra machinery. A standard network runs every input through all ninety-six layers whether it is a blank wall or a hard corner case. This one gives the blank wall six turns and the hard corner as many as it needs, because depth was never a fixed cost, it was iterations to a fixed point, and easy points reach it sooner. Nobody had to schedule any of it.

Two moons never needed the depth#

Here is the uncomfortable thing about everything so far. Two interleaving moons are an easy problem, and the network settled them in about six turns. If six turns is all it ever takes, why build a machine that can go a thousand? The equilibrium view is elegant, but elegance is not a reason to go deep, and no toy blob will give you one. You only believe in depth when a problem makes you earn it, so take one that does.

Reachability on a grid is a problem that makes you earn it. Mark a goal cell; a cell counts as reachable if a free neighbour is reachable, which holds if one of its free neighbours is, and so on back to the goal. The answer spreads outward one ring per step, so a cell thirty hops away cannot possibly be decided in fewer than thirty turns. Depth stops being a knob and becomes the propagation distance itself. We train the very same shared Yat operator on small $11\times11$ grids, reading each cell’s five-cell neighbourhood, blocked by walls, and let it iterate to its fixed point, which is the finished reachable map.

Now for the hard question. The operator only ever saw $11\times11$ grids while training, unrolled for just 30 turns, so no front it ever followed traveled more than thirty steps. A $27\times27$ maze needs the front to travel forty cells and more, into territory no training run ever reached. Everything we know about generalization says the network should fall apart out there. So put it there. Pick a grid it could never have seen, and watch it think.

It does not. The front crosses the thirty-step line as if nothing were there, keeps flooding, and solves the mazes to 99.5% of cells, as long as you let it iterate long enough for the front to cross. That is the thing a fixed stack cannot do. A twelve-layer network can propagate information exactly twelve hops and then it is out of layers, so any maze wider than twelve is beyond it forever. A recursion has no such ceiling: it learned a local rule and applies it as many times as the instance demands. This is why sharing the operator across depth is not a parameter-saving trick, it is what lets the network solve problems larger than the ones it was trained on.

Read the curves and the whole case for depth is in them. On a short budget of 22 turns a $27\times27$ grid sits near 54% of cells correct. Even at the full training length of 30 turns, the most thinking it was ever trained to do, it has only climbed to 76%. Neither number means the network is broken; it is unfinished, the front simply has not arrived. Let it keep turning past anything training ever did and every size climbs to near-perfect, the largest last, exactly in the order of how far their fronts must travel. The scatter beside it closes the circle with the earlier adaptive-depth story: the turns each maze actually needs track its true propagation distance almost perfectly, $r = 0.98$. The network never picks a depth. The problem hands it one.

Trained through what, exactly?#

The question parked back in the contraction section is now due. Everything above, the tuned contraction, the moons boundary, the maze rule, was trained, and training means gradients. In a stack, gradients have an obvious road: backpropagate through the layers, storing every activation on the way forward so the backward pass can retrace the path. Here there is no road. The forward pass is “iterate until it stops”, fourteen turns for one input and sixty-five for another, and unrolling whatever it happened to take, storing every intermediate state, would cost memory that grows with every turn. Worse, it would betray the framing: we said the answer was the solution of an equation, not the path to it. So stop differentiating the path.

Differentiate the equation instead. The fixed point satisfies $z^\* = F(z^\*; x)$, an identity, and taking its derivative makes the chain rule fold into itself, $dz^\* = J_F\,dz^\* + \partial_\theta F$, so the gradient is one linear solve in the operator’s Jacobian $J_F = \partial F/\partial z$ at $z^\*$,

That is the implicit function theorem, and it costs constant memory no matter how many turns the forward pass took. You never store a hundred layers of activations to backpropagate through, you differentiate the equation, not the path to it. And the answer closes on itself with a small piece of elegance: the same contraction ($\lVert J_F\rVert < 1$) that made the forward iteration converge makes this backward solve converge too, by the same argument.

What you actually get#

So what did collapsing the stack into an equation actually buy? The whole network, at any depth, is 1700 numbers. There is no per-layer weight budget because there are no distinct layers, only the one operator $(W, A, U, C)$ reused for every turn. Depth is free in parameters; it costs iterations, not memory.

None of it cost accuracy, either. The contraction penalty, the shared weights, the self-imposed halting, and it still reaches 98.2% on the interleaving moons, from those 1700 shared numbers, and near-perfect reachability on mazes far larger than any it trained on.

And it stays legible, which is the through-line of this whole series. The same 24 prototypes describe the computation at every turn, not a fresh unnamed bank per layer, so the entire network reads as one small, fixed vocabulary applied repeatedly. And at the fixed point the answer is literally a vote over those prototypes, the representer form the series keeps returning to, now the solution of an equation rather than the output of a pass. A recurrent network you could not read has become a kernel machine you can.

There is one thing this legibility used to buy that we have not dared check. The last network this readable was one you could edit by hand, and every proof that an edit touched nothing else rested on a pasted row entering the score once, as one more term in one sum. In this network there is no once: whatever you paste into the operator is applied at every depth and fed back into its own input. Whether the editability survived the melting is the next post’s question.

A network, in the end, does not have to be a tower you commit to and march through. It can be an operator you apply until the answer stops changing, an equation you solve. Choose the operator well, a Yat kernel that keeps it legible and can be tuned into a contraction, and depth stops being a guess you make in the dark. The input tells you how deep to go, and the network tells you when to stop.

@misc{bouhsine2026yournetworkisafixedpoint,
  author       = {Bouhsine, Taha},
  title        = {Your Network Is a Stack of Layers. It Could Be a Fixed Point.},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/your-network-is-a-fixed-point/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}