Lazy Training
2 long-form posts on Lazy Training: machine-learning research by Taha Bouhsine, each built around live, in-browser interactive visualizations.
-
Lazy Training a Yat Network in JAX/Flax NNX
A runnable companion to the lazy-training post: the Yat layer with its frozen softplus scalars, the one-line NNX filter that trains a readout while the bank stays frozen, the per-arm learning-rate bracketing, and the movement telemetry that caught the anti-lazy power law. With the run's own prototype trajectories: a Gram matrix crystallizing as neurons accumulate, two readouts racing on frozen banks, and eight random prototypes drifting through training without ever becoming pictures.
-
How Many Random Neurons Buy a Trained One?
Freeze a bank of Yat units at random initialization, noise prototypes that will never move, and train nothing but the linear readout on top. This should not work, and the reason it does is one of the oldest facts in the kernel literature wearing a new activation: a frozen random layer is a Monte Carlo estimate of a kernel, one neuron per sample, converging at the square-root rate. This post watches the readout learn on frozen noise live in the page, counts how many frozen units buy each rung of the accuracy ladder this series has built by other means, measures the Monte Carlo exponent in the curve, and then lets the features go to see exactly what training them is worth, including the one kernel whose centers famously refuse to move.