Interpretability
Mechanistic and white-box interpretability: reading attention heads, MLP prototypes, induction circuits, and the geometry of learned representations.
-
A Network Made of Parts
Cut the image into patches, run one shared kernel bank over every patch, average the results, and classify. Linearity makes the output exactly decomposable into per-patch score contributions. The architecture was also supposed to fix three failures of the whole-image network. It fixed none: softening falls with distance, and concepts remain distributed across most of the bank at every tested granularity.
-
How to Interrogate a Kernel Network
A network whose hidden units are kernel prototypes is supposed to be legible. Legible claims are cheap unless someone can check them, so this post builds the checking: five instruments that put a trained Yat network under oath, each one asking a question that only this kernel makes askable. The first instrument finds that the softening constant in the formula sits ten thousand times below the distances it is supposed to soften, so the trained network never uses it at all: a term can be load-bearing in the theory and idle in the artifact, and only an audit tells you which.
-
The Concept That Would Not Die
An empirical feature covariance gives a trained kernel network ranked orthogonal axes. Deleting one axis is an exact algebraic intervention; the experiment asks whether it is also a semantic one. It is not: the damage spreads broadly and a small probe recovers the targeted distinction.
-
The Trained Network, Under Mercer's Microscope
Every hidden representation induces an empirical kernel. Decompose it into ranked modes, audit their stability and semantic evidence, then repeat the measurement on grayscale CIFAR-100 where one hundred classes leave room for a genuine concept-count test.
-
Running the Survival Trial, in JAX/Flax NNX
Build the Yat DeepSurv trunk and Cox loss in Flax NNX, recover exact prototype contributions and explicit row edits, then run the LR-fair five-dataset benchmark with concordance, calibration, Brier, AUC, and classical baselines.
-
Solving It and Descending It, in JAX/Flax NNX
A runnable companion to the solve-vs-descend post: the Yat kernel and its Gram matrix, the exact kernel ridge solve via Cholesky, the same kernel as a Flax NNX module trained by AdamW with LR sweeps and best-epoch selection, the measured timing wall, minibatching through 511k rows, and the conv trunk the solve can never train. Every number is from the real Kaggle runs.
-
The White-Box Survival Model on Trial
Build a survival network from learned prototype patients, derive its exact risk decomposition, and benchmark it on five datasets against Cox, penalized Cox, Random Survival Forest, and ReLU DeepSurv. Calibration, editing, and shift detection are measured separately.
-
One Kernel Family, Fitted Two Ways
A dense kernel-ridge solve and a learned-center Yat expansion use the same kernel family but optimize different hypothesis classes. Their predictions correlate at 0.95 on housing; the compressed model then scales through datasets the dense baseline cannot hold.
-
Building the Second Layer by Hand, in JAX/Flax NNX
A runnable companion: build a whole second feature layer by hand in JAX, on top of the hand-built first. Named min-AND combinations of layer-1 edges (junctions, continuations, bends, stripes) feed the same constructed Yat head, no training anywhere. It reproduces the flat rung: 83.3% at layer 1, 82.9% with both, 78.8% from relations alone, and counts the combinatorial wall of 224 pairwise and 4,630 three-way types where construction stops.
-
How Far Down Can You Build?
One hand-built feature layer matched a trained backbone at 83.3% on Fashion-MNIST, and real networks are deep. Conveniently, the recipe for a second layer has been on the shelf for half a century: vision science says edges assemble into junctions, continuations, bends and stripes. This post takes the recipe down and follows it, builds layer 2 entirely by hand with every dimension still nameable in one sentence, and measures exactly where construction stops, and why.
-
Edit One Operator, Edit Every Depth
One post taught and forgot classes by editing rows of a Yat network, with proofs that nothing else moved. Another melted the stack of layers into a single operator iterated to a fixed point. This is the collision. Every one of those editing proofs rested on a pasted row entering the score once, as one term in one sum, and in an equilibrium network there is no once: whatever you paste is applied at every depth and fed back into its own input, and every fixed point is free to drift. So did melting the stack melt the editability? This post pastes, deletes, and measures: every guarantee that survives is either proved inside the recursion or measured against the real run, fixed point by fixed point.
-
Your Network Is a Stack of Layers. It Could Be a Fixed Point.
A deep network makes you choose its depth before you have seen the problem, and gives every layer its own weights. Share one Yat-kernel operator across depth and the stack becomes a single equation: the answer is the fixed point reached by iteration. On the measured test trajectories, the solver converges from widely separated starts and the local Jacobian norm stays below one. The same twenty-four prototypes describe every step, reaching 98.2% on two moons from 1,700 shared parameters.
-
A White-Box Kernel FFN in JAX/Flax NNX
A runnable companion: build a transformer whose feed-forward block is a finite learned-center kernel expansion. Train it on tinyshakespeare, then read each memory slot, attribute outputs exactly, edit one slot, and test peak kernel response as an abstention score.
-
The MLP Block Can Be a Kernel Memory
Replace an MLP activation with a kernel and its feed-forward block becomes an explicit learned-center expansion. Its slots can be read, attributed, and edited, but this architectural parameterization is not the classical representer theorem.
-
Where Does a Weight Live?
A standard neuron's weight and its input never actually meet: one is a point you can see, the other an arrow off in its own space, joined only by a shadow. This is what a reproducing kernel Hilbert space fixes: it gives input and weight one shared address, where the optimal weight is built from the data itself and sits right next to it. Four interactive panels.
-
How Much of a Fashion-MNIST Network Can You Build by Hand?
Construct the prototype head on random and learned features, then replace the backbone with named edge and corner measurements. On Fashion-MNIST the zero-training pipeline reaches 83.3%, versus 85.7% for the matched trained model.
-
Editing a Network by Hand, in JAX/Flax NNX
A runnable companion: build the prototype Yat-MLP in Flax NNX, then add a class by concatenating a few prototype rows and forget a class by masking them out, with no gradient steps. Class-incremental learning that matches a from-scratch build, and exact machine unlearning, both as array edits you can read. Every number is from a real run on Fashion-MNIST.
-
Your Network Is a List of Pictures. You Can Edit It.
If a neuron is a labelled picture, a classifier is a list of them, and a list is something you edit. Add a class to a trained-free Yat-kernel network by placing twenty pictures, and it recognizes that class at 95% with zero gradient steps. Delete a class by removing its pictures, and it is forgotten exactly, the other classes untouched. Class-incremental learning with no penalty and machine unlearning that is instant and exact, both falling out of the architecture rather than bolted on.
-
The Yat-Kernel MLP in JAX/Flax NNX
Build a finite bank of Yat-kernel sections in JAX/Flax NNX, verify the kernel, train it on two moons and Fashion-MNIST, inspect exact prototype contributions, and test the initialization control that separates visible centers from noisy ones.
-
What a Finite Kernel Buys an MLP
Replace the activation with a finite bank of learned kernel sections. The resulting MLP exposes prototypes, exact layer-local contributions, measurable geometry, and the conditions those claims require, then tests the construction on arithmetic and Fashion-MNIST.
-
Why Attention Needs Q and K Projections
The dot product in attention is not enough by itself. Without learned query and key projections, attention can only compare tokens in the residual stream’s native geometry. With a shared projection it learns a symmetric metric. With separate Q and K projections, the score becomes a learned bilinear form x_iᵀW_QW_Kᵀx_j: directional, role-aware, low-rank, and different per head. That bilinearity is what lets attention ask one kind of question and let tokens advertise another kind of answer.
-
The Prototype Readout in JAX/Flax NNX
A runnable companion to The Readout is a Convex Combination of Prototypes: read the columns of W_out as output prototypes in Flax NNX, measure the convex/conic/affine/linear regimes numerically, then build a Nadaraya–Watson kernel readout that is convex by construction (nonnegative weights that sum to one, a point that never leaves the prototype hull), with the nonnegativity-vs-positive-definiteness distinction checked in code.
-
The Readout is a Convex Combination of Prototypes
The second linear map in a transformer MLP is a dictionary of output prototypes, one per hidden unit. If the hidden activations are nonnegative and normalized, W_out reads the active neurons as a convex combination of output prototypes. Two independent constraints, nonnegativity and summing to one, sort the readout into four regimes: convex, conic, affine, and linear. This reframes the MLP readout as the same object that makes attention legible (a weighted sum over named basis elements), connects it to feed-forward key-value memories and modern Hopfield retrieval, and shows when a kernel makes it convex by construction.
-
Untangling the Moons: A Visual History of Contrastive Learning
Eight contrastive losses, twenty years of history, and one geometric audit. Watch the losses organize the same 2D points while separating opposition, orthogonality, simplex packing, and statistical independence.
-
Self-Attention as Kernel Regression in JAX/Flax NNX
A runnable companion to Attention is Explainable Because it is a Kernel: build scaled dot-product attention from scratch in Flax NNX, prove in code that it is exactly a Nadaraya–Watson kernel smoother, watch the separate q/k projections break positive-definiteness numerically, swap the exp-dot-product kernel for Gaussian, Yat, and linear kernels to see which keep the weights a convex partition of unity, read the temperature as a kernel bandwidth, and train a single head end-to-end to route to a marked token.
-
What Attention Weights Can Explain
Self-attention has the normalized weighted-average form of a compatibility smoother. That exposes exact routing arithmetic, but it does not make attention weights causal explanations or guarantee a Mercer kernel on tokens.
-
What Activations Do to Geometry
ReLU, GELU, and their relatives enter a layer's Jacobian as an input-dependent row scaling. Here is when that scaling erases directions, when it merely distorts them, and what the usual repairs actually guarantee.