You Only Have to Train the Features

Which part of a network actually needs to be trained? The obvious answer is all of it, and the obvious answer is wrong. Take a convolutional network and leave every weight at its random starting value; it sorts Fashion-MNIST at chance, ten percent, the way a pile of random numbers should. Now throw away its untrained last layer and build the classifier by hand instead, twenty prototype pictures per class, no gradient steps anywhere, and the same random network suddenly sorts most of the test set correctly. Whatever the training was for, it was not for the classifier. This post is about where that line falls, between what you have to optimize and what you can simply place, and it turns out to fall somewhere sharp: you train the features, and you build everything above them.

What a feature actually is#

That promise leans on two words, features and head, and neither is as obvious as it sounds, so pin them down first. A network does its job in two stages. First it boils the raw image down to a short list of numbers that summarize it; those numbers are the features, and the stack of layers that produces them is the backbone. Then it draws boundaries between the classes using only those numbers; that second part is the head, also called the readout or the classifier. Ordinary training moves both stages at once, which is exactly why it is hard to say which one is pulling its weight.

So pull them apart. And to see why the features matter at all, ask what a good one even buys you.

Why does an image need re-describing at all, when its pixels are sitting right there? Because pixels are a representation too, just a poor one. Lay the raw images out by similarity and the two clothes the network confuses most, a pullover and a coat, fall into one overlapping cloud: there is no line you could draw through raw pixels that keeps them apart. A feature is a learned measurement, a number the backbone is trained to compute precisely so that this stops being true. Train the backbone and the same two classes pull apart into separate clusters. The measurements were chosen, by gradient descent, to make the classes separable.

That is what “train the features” means: you train the backbone until its features lay the classes out cleanly. Everything after that, the part that finally says coat or pullover, is the head, and the rest of this post is about how little that head needs you.

Split the bill#

So what is each stage worth on its own? Pay for them separately and find out. The features cost a full training run: a small convolutional backbone, two conv-and-pool blocks down to 64 features, trained the ordinary way with a linear head and cross-entropy, reaching 85.7%. The classifier we refuse to pay for. Freeze the backbone, throw its trained head away, and on the frozen features build a new head by hand: twenty prototype points per class from k-means, each wired to vote for its own class, an image classified by the nearest one. No gradient step ever touches it. It reaches 83.2%. (Both numbers, and every learned-feature number in this post, come from one run of scripts/construct_vs_optimize.py in the repo.)

Training the head, on the very same features, bought 2.5 points. The other seventy-five, everything that separates this from a coin flip, came from the backbone. (The same built head on raw pixels, from the previous post, capped out at 79%, which is the whole reason we wanted features in the first place.)

Write it down and the split is one line. The network is $f = h \circ g$: a backbone $g$ that maps an image to features $z = g(x)\in\mathbb R^{64}$, and a head $h$ that scores classes from $z$. Training minimizes a loss over both. Building the head replaces $h$ with the kernel vote from the editing post, the same landscape of softened inverse-square wells with a prototype mass at each k-means centroid $\mu_u$, except the wells now sit in feature space: each class is scored by its nearest prototype, and the deepest well claims the image,

It has no free parameters to fit, so its accuracy is a function of $g$ and nothing else. The two heads differ in exactly one freedom: the trained head moves $g$ and $h$ together, the built head rides whatever $g$ the backbone settled into.

The network that never trained#

How far does the head’s independence go? Take the experiment to its limit and train nothing at all. Leave the backbone at its random initialization, every weight a fresh random draw, and build the same head on the garbage features that fall out.

You would expect nothing, and on this backbone the trained head delivers exactly that: it sorts the test set into ten bins that are each a random rainbow, dead at chance. But the built head, on those same random features, sorts them into mostly-pure bins and lands at 74%.

Seventy-four percent, from a network that never saw a gradient. Random convolutional features are a known free lunch: even an untuned stack of filters scrambles the image in a way that keeps some of its class structure intact, and a head that only has to read structure does not care that nobody trained it.

The whole trade-off, in one curve#

Two endpoints make a story but not a law, so watch the whole way across. At every epoch, freeze the backbone, read its trained head, then build a fresh head on the current features and read that too.

The two curves climb together. The trained head starts at chance and the built head starts at 74%, and from the moment the backbone has any real features the two run nearly parallel, a couple of points apart, all the way up. The gap between them is the entire value of training the classifier. Everything else is the backbone lifting both at once.

The number says the representation does the work. The geometry says why.

What was the backbone doing all that time?#

The curve says the backbone did the lifting; the features themselves can show how. Laid out in two dimensions at each stage of training, the climb has a shape. An untrained backbone gives a single tangled blob, and the built head, which labels a point by the nearest prototype, has nothing to grab: points and prototypes are all piled together. Train the backbone and the blob resolves into clusters, one per class. The head never changed. The space under it did, and suddenly nearest-prototype is enough.

That is the whole mechanism. The built head is a fixed rule, which stored point is this closest to, and a fixed rule is only ever as good as the layout it runs on. Training the features is the act of arranging the layout until the fixed rule works.

The head you built is one you can read#

There is a bonus hiding in having built the head instead of training it: you can still read it, the way the earlier posts read a network of pixel prototypes. A prototype is a stored point, and a stored point has a nearest real image, the training example whose features land closest to it. On raw pixels that stand-in was a blurry k-means average. In feature space, where a class clusters tightly, the nearest real image is one sharp, specific example, the this looks like that of prototype networks.

And the reading survives the move even though the backbone does not. The trained backbone is a genuine black box, sixty-four numbers with no individual meaning. The head riding on top is still a bank of pictures, so the classifier stays legible while the representation under it never was. The interpretability lives in the part you place, not the part you train.

And the edits, the teaching and forgetting that made the pixel-space head worth having, do they survive the move into feature space? Unchanged in form, because the head is still a list of prototypes with a one-hot readout. Forgetting a class is deleting its block: drop the Sandal prototypes and Sandal recall falls from 93% to 0% while the other nine classes hold at 82.2% to 82.5%, untouched, because their scores never referenced a Sandal prototype. Teaching a class is appending a block: build the head from eight classes, then add prototypes for Bag and Boot, and they are recognized at 95% each while the head reaches its full 83.2%, exactly as if all ten had been there from the start. Adding and forgetting were never properties of pixels. They are properties of storing a class as its own block of prototypes, which holds in whatever space the prototypes live in.

Where the line falls#

So where does the line between optimize and construct actually fall? At the top of the backbone, and the cut is sharp. A network is a representation with a classifier on top. The representation is the part you must learn: random features give 74%, trained features give 83 to 86, and that climb is the only place real accuracy comes from. The classifier is the part you can build: place prototypes, wire a one-hot readout, and you are within a couple of points of the trained head, with the teaching and forgetting thrown in for free.

Which leaves the greedy question. If the head comes for free, how much of the backbone can you get without training either? More than you would think. Swap the learned features for hand-designed ones, oriented-edge and corner detectors pooled over the image, the kind of thing computer vision built by hand for decades, and feed those to the same built head. With no training anywhere, not in the features and not in the classifier, it reaches 83.3% (a real run too, scripts/handbuilt_vision.py), matching the trained backbone. That is the next post: building the features by hand too, and seeing how far down the construction can reach before you finally have to learn.

The prototype-network idea (“this looks like that”) is from Chen et al. (2019); the surprising strength of random convolutional features is from Saxe et al. (2011); Fashion-MNIST from Xiao et al. (2017); the Yat kernel from Bouhsine (2026). This continues Your Network Is a List of Pictures. You Can Edit It.