Edit One Operator, Edit Every Depth

Two posts ago, a network was a list you could edit. Its whole trick was that knowledge had an address: a class was a handful of rows you could point at, so teaching was pasting rows in, forgetting was deleting them, and you could diff the weights afterward to prove nothing else moved. Then the last post took the stack of layers those rows lived in and melted it down: one shared operator, applied over and over until the state stops moving, the answer a fixed point rather than a final layer. It was a good trade. But look at what it did to the address book.

In the list network, a pasted row entered the score once, as one more term in a sum, and the proof that old classes could not feel it was a single line. Here there is no “once.” Whatever you paste into the one shared operator is applied at every depth, and its output is fed back into its own input, around and around until the equation settles. An edit does not add a term; it changes the map whose fixed point is the answer, and every observed equilibrium is free to drift. The solver’s local stability also has to be measured again after the map moves. So the question this post exists to answer is sharper: did melting the stack melt the editability too?

The short version: it survives, but it splits in two, and which guarantee you keep depends on which kind of row you touched. The rest of this post is finding that split and testing both halves of it. Every number comes from one run, and every panel below solves the real operator’s fixed points as you drag it.

Where would an edit even go?#

The machine under edit is the one the last post built, retrained with a third class in the mix, so one paragraph of recap and no more. A bank of $M = 24$ shared prototypes and one mixing matrix define a single operator on a 32-dimensional state,

iterated from $z = 0$ until it stops at $z^\* = F(z^\*;x)$, then read out linearly, $\hat y = C z^\* + c$. Ours is trained on three classes, two interleaving moons and one Gaussian blob, and lands at 97.9%, 98.1%, and 100% per-class recall from 1,733 parameters shared across all depth. At the test equilibria, the measured spectral norm $\lVert J_F\rVert_2$ has mean 0.629 and max 0.904. That is a local stability audit over the tested states. It is not the uniform state-space bound a global contraction certificate would require.

Now suppose a fourth class shows up, a second blob the network has never seen, and you refuse to train. In the list network the move was obvious because everything was rows. Here the anatomy offers exactly two places to put new rows, and the entire post hangs on the difference between them. There are the rows that read the settled state: the readout is a matrix, and a matrix accepts new rows. And there are the rows that steer the state: the shared bank $W$ inside $F$ itself, the wells the state rolls through on its way to rest. Editing the first leaves the equation alone. Editing the second edits the equation, at every depth at once.

Teach it without touching the flow#

The timid door first, because it is the one that comes with a proof. To teach the new class, take eight examples of it, drop each into the frozen network, and let the operator do what it always does: iterate them to their equilibria. Those eight settled states $c_1, \dots, c_8$ are where the flow already parks the new class. Paste them in as readout rows, scored through the very same Yat kernel the operator uses, and give the new class the score

with $\alpha$ one calibration constant that puts the kernel on the scale the trained logits speak at. No gradient steps. The new class is recognized at 85.8% recall.

And the old classes? Here the proof from the editing post survives verbatim, because $F$ was never touched. Same operator, same fixed points, same linear readout of them: the old scores are not approximately preserved, they are the identical computation. The run asserts it: max $|\Delta\,\text{old logit}| = 0.0$, exactly, and zero old test points get captured by the new class. Teaching by readout is the editing post’s append-a-row, alive and well inside the recursion.

A new mass, and whose landscape it lands in#

Why does the timid edit feel timid? Put it back in the landscape of wells, which the last post moved into state space: the operator is the flow, the contraction gives it one basin, and the answer is the rest point at the bottom.

The readout edit hangs a new instrument beside that landscape without touching it. The state rolls exactly where it always rolled; we just read its resting place against eight new reference points. That is why the invariance is exact and why the edit cannot break anything: a gauge cannot alter the thing it measures. But it also cannot help the thing it measures. The new class’s inputs settle wherever the old flow happens to drop them, and the flow was never carved for them. You can see the shortfall in two numbers from the run: the new class’s margin over the old scores is a thin 0.55, and its test inputs settle a mean distance of 0.159 from their nearest anchor, a loose cloud around the pasted rows rather than a tight one. Recall 85.8% is what reading an uncarved landscape buys.

To do better, the mass has to go into the landscape. Drop the eight anchors into the shared bank, $W \leftarrow [W; c_k]$, with matching mixing columns $\gamma\,\hat c_k$, and the operator itself now has wells at the new class: the flow bends toward them, at every depth, because every depth is the same operator. That is the strong door. One paste, present everywhere. And it is the one door with no proof behind it.

Through the strong door#

What could go wrong with deepening a well? In the list network, an appended unit added one bounded term to one class’s score, and the story ended there. In a recursion the pasted rows feed their own output back into their own input. Make the well too deep and the local slope of $F$ can cross one around the equilibria the solver visits. That removes the local contraction evidence and often predicts slower or unstable iteration. The right response is direct: remeasure the slope, rerun the solver from multiple starts, and audit the fixed-point drift.

And the natural paste breaks that local stability margin. Choosing $\gamma$ so each anchor gets a healthy pre-activation bump sends the measured slope to $\lVert J_F\rVert_2$ max 1.438 over the test probes. The measurement no longer supports locally contractive behavior around every probed equilibrium, so the raw edit is too aggressive for this solver.

The repair is one number. The edit has a single gain knob, so bisect it: find the largest fraction of the raw $\gamma$ whose measured slope stays under a ceiling of 0.98 on the test probes. The run lands on $0.462\,\gamma$; the local audit reads mean 0.646, max 0.980. The tempered paste still does its job: new-class recall goes from 85.8% to 100%, the margin grows from 0.55 to 3.83, and the settle distance to the nearest anchor falls from 0.159 to 0.097. The flow now carries the new class’s inputs home instead of merely labeling where they stall.

What did that cost everyone else?#

The readout edit’s invariance was a proof. The dynamics edit’s cannot be, the map moved, so what remains is a measurement, the one this whole post has been building toward: over all 520 old-class test inputs, solve every fixed point under the base operator and under the edited one, and subtract.

The drift $\lVert \Delta z^\* \rVert$ has median 0.020 and 90th percentile 0.071, on settled states whose norms run around 3.7: for the typical old input, the basin bottom moved by about half a percent of the state’s scale. But the tail is real and deserves its numbers: 29 of 520 inputs moved more than 0.1, the worst by 1.47, and the score changes follow the same shape, median 0.029, max 1.47. Three predictions flipped, all near boundaries, taking the old classes from 97.9/98.1/100 to 96.8/97.6/100. Settle times barely notice: the old classes’ median goes 18 to 19 turns.

So say it precisely, because this is the difference between this post and its parent. The recursion downgraded the dynamics edit from provably untouched to measurably barely touched: a median drift of 0.02, a worst case of 1.47, three flips in 520. Nothing here is exact; what remains is that the disturbance is small, concentrated near the new class, and, unlike any fine-tuning run, fully auditable, because the same solver that computes the answer computes exactly what the edit did to it. If you need the proof back, the readout edit still has it.

And what did all of it buy, against the alternative of just training? One comparison settles it, and it closes the loop with the editing post’s incremental-learning result. There, teaching classes one at a time landed exactly where training on everything from scratch did, 79.4% equals 79.4%, because superposed fields cannot remember arrival order. Here the from-scratch four-class model reaches 98.75% overall; the incremental network, base plus constructed class, reaches 96.1% with the readout edit and 98.3% with the stability-tempered dynamics edit. Not equal this time, and the reason is plain: the from-scratch model’s flow was trained for all four classes, while ours had the fourth carved in by hand. Half a point is the measured price of never taking a gradient step on the new class. The recursion turned an identity into a near-tie: exactness traded for a small, reported gap.

The third door#

There is a tempting third door this whole time, and it is worth walking through it just to watch it slam. The old instinct says: edit a layer. The recursion even seems to allow it: apply the edited operator for the first few turns, then let the original take over, the equilibrium version of splicing a modified layer into a stack. What survives at the fixed point?

Nothing measurable survives. Run the hybrid with the edit steering for the first 3, 10, or 30 turns and the base operator afterward, and the state lands back on the base equilibrium to within $3.3 \times 10^{-7}$, while the solver’s own noise floor, the discrepancy with no edited turns at all, is already $2.4 \times 10^{-7}$. On these inputs the base iteration erases the excursion geometrically fast. A finite prefix cannot change the limiting point when the subsequent base iteration returns to the same observed attractor.

Which sharpens the title of this post into a statement about necessity. In an equilibrium model there is no such thing as editing a layer, because a fixed point only remembers what acts on it at every depth. Weight-tying does not merely permit the everywhere-edit, it is the only edit there is. One paste into the shared operator is automatically that everywhere-edit; anything less evaporates.

What a stack would charge for the same edit#

The evaporation result says an edit must remain active in the limiting operator to affect its fixed point. The tied network gets that for free; what does it cost when the weights are not shared? The run trains the control: the same architecture with the tying switched off, twelve untied layers, which needs 18,981 parameters to the loop’s 1,733 before anyone edits anything.

Reading is cheap everywhere: pasting anchor rows onto the stack’s final state teaches the class at full recall with zero dynamics edits, the readout route again, depth-free in any architecture. But putting the class into the dynamics, the thing that carved the wells and increased the margin above, has no single place to go. The paste must be repeated at each layer where the edit should exist: 512 numbers per layer, 6,144 to cover all twelve. Each pasted layer moves old inputs’ final states further, from 0.46 with one layer to 3.86 with all twelve, past the tied edit’s worst case of 1.47. A stack has no single equilibrium operator to audit, so the comparable check is end-to-end sensitivity across its layers. One operator makes the tied edit cheaper to apply and easier to measure.

Forgetting splits into two ledgers#

Now run the whole thing backwards, because unlearning was the editing post’s sharpest result, delete the rows and the class is provably gone, and the recursion does something more interesting than preserving it: it splits it in two.

Forget the class we just taught, and the old guarantee returns in full. Its knowledge sits in eight appended rows of $W$ and eight columns of $A$; slice them off and the run checks the result against the pre-edit operator: bit-for-bit identical, every array equal. What construction pasted in, deletion removes exactly. The original fixed points and every original stability measurement return because the operator itself is restored.

Now try to forget a class the network was trained on, and the address book fails for the first time. The blob class has one readout row, but its knowledge does not live there; gradient descent wove it into the shared bank that every class’s flow runs through. Mask the row and the surface result looks like exact unlearning: recall 1.00 to 0.00, and the other classes read literally the same numbers, 97.9/98.1 before, 97.9/98.6 after, the tiny gain being one moon point the blob had been capturing, handed back to its owner. But the flow still carries the class. Paste eight anchor rows, the same construction that taught class 3, and the “forgotten” class comes back at 100% recall, no training. Masking took away the reading and left the knowledge exactly where it was.

That is the boundary, and it is the finding of this post: inside a fixed point, what you constructed, you can delete exactly; what you trained, you can only silence. True erasure of trained knowledge would mean editing the dynamics it is woven into, and the evaporation section already told you such an edit must reach every depth, which is retraining by another name. The editing post’s exact unlearning was never a property of the architecture alone; it was a property of knowledge stored as rows. The equilibrium model keeps the guarantee precisely for the knowledge that lives that way.

So, did melting the stack melt the editability? No, but it split the contract. Readout rows keep every proof from the list network: exact invariance, exact undo, zero gradient steps. Dynamics rows are more powerful—one paste reshapes the operator at every turn—but they replace invariance with an audit: re-solve the fixed points, report the drift, test multiple starts, and temper the gain when local slopes rise. An equation you can edit is an equation you can measure after the edit. This post has done both.

@misc{bouhsine2026editafixedpoint,
  author       = {Bouhsine, Taha},
  title        = {Edit One Operator, Edit Every Depth},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/edit-a-fixed-point/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}