series · 4 parts
Weights in Kernel Space
Once everything is a kernel, what is a weight? An interlude on RKHS foundations: where a weight lives, what it can be, and why the MLP block is a representer theorem.
Start reading → The Readout is a Convex Combination of Prototypes- 01 The Readout is a Convex Combination of Prototypes The second linear map in a transformer MLP is a dictionary of output prototypes, one per hidden unit. If the hidden activations are nonnegative and normalized, W_out reads the active neurons as a convex combination of output prototypes. Two independent constraints, nonnegativity and summing to one, sort the readout into four regimes: convex, conic, affine, and linear. This reframes the MLP readout as the same object that makes attention legible (a weighted sum over named basis elements), connects it to feed-forward key-value memories and modern Hopfield retrieval, and shows when a kernel makes it convex by construction. JAX companion The Prototype Readout in JAX/Flax NNX
- 02 Where Does a Weight Live? A standard neuron's weight and its input never actually meet: one is a point you can see, the other an arrow off in its own space, joined only by a shadow. This is what a reproducing kernel Hilbert space fixes: it gives input and weight one shared address, where the optimal weight is built from the data itself and sits right next to it. Four interactive panels. JAX companion Where a Weight Lives, in JAX/Flax NNX
- 03 What Can a Weight Be? A kernel is a spectral price list: it decides which functions are affordable, and regularization sets the budget. Compare Sobolev, Gaussian, spherical, and finite examples, then connect their eigenvalues to kernel ridge shrinkage and effective dimension. JAX companion A Kernel's Price List, in JAX
- 04 The MLP Block Can Be a Kernel Memory Replace an MLP activation with a kernel and its feed-forward block becomes an explicit learned-center expansion. Its slots can be read, attributed, and edited, but this architectural parameterization is not the classical representer theorem. JAX companion A White-Box Kernel FFN in JAX/Flax NNX