How Much of a Fashion-MNIST Network Can You Build by Hand?

Which part of a network actually needs to be trained? The obvious answer is all of it, and the obvious answer is wrong. Take a convolutional network and leave every weight at its random starting value; it sorts Fashion-MNIST at chance, ten percent, the way a pile of random numbers should. Now throw away its untrained last layer and build the classifier by hand instead, twenty prototype pictures per class, no gradient steps anywhere, and the same random network suddenly sorts most of the test set correctly. Whatever the training was for, it was not for the classifier. This post is about where that line falls, between what you have to optimize and what you can simply place, and it turns out to fall somewhere sharp: you train the features, and you build everything above them.

What a feature actually is#

That promise leans on two words, features and head, and neither is as obvious as it sounds, so pin them down first. A network does its job in two stages. First it boils the raw image down to a short list of numbers that summarize it; those numbers are the features, and the stack of layers that produces them is the backbone. Then it draws boundaries between the classes using only those numbers; that second part is the head, also called the readout or the classifier. Ordinary training moves both stages at once, which is why it is hard to say which one is pulling its weight.

So pull them apart. And to see why the features matter at all, ask what a good one even buys you.

Why does an image need re-describing at all, when its pixels are sitting right there? Because pixels are a representation too, just a poor one. Lay the raw images out by similarity and the two clothes the network confuses most, a pullover and a coat, fall into one overlapping cloud: there is no line you could draw through raw pixels that keeps them apart. A feature is a learned measurement, a number the backbone is trained to compute precisely so that this stops being true. Train the backbone and the same two classes pull apart into separate clusters. The measurements were chosen, by gradient descent, to make the classes separable.

That is what “train the features” means: you train the backbone until its features lay the classes out cleanly. Everything after that, the part that finally says coat or pullover, is the head, and the rest of this post is about how little that head needs you.

Split the bill#

So what is each stage worth on its own? Pay for them separately and find out. The features cost a full training run: a small convolutional backbone, two conv-and-pool blocks down to 64 features, trained the ordinary way with a linear head and cross-entropy, reaching 85.7%. The classifier we refuse to pay for. Freeze the backbone, throw its trained head away, and on the frozen features build a new head by hand: twenty prototype points per class from k-means, each wired to vote for its own class, an image classified by the nearest one. No gradient step ever touches it. It reaches 83.2%. (Both numbers, and every learned-feature number in this post, come from one real run.)

Training the head, on the very same features, bought 2.5 points. The other seventy-five, everything that separates this from a coin flip, came from the backbone. (The same built head on raw pixels, from the previous post, capped out at 79%, which is the whole reason we wanted features in the first place.)

Write it down and the split is one line. The network is $f = h \circ g$: a backbone $g$ that maps an image to features $z = g(x)\in\mathbb R^{64}$, and a head $h$ that scores classes from $z$. Training minimizes a loss over both. Building the head replaces $h$ with the kernel vote from the editing post, the same landscape of softened inverse-square wells with a prototype mass at each k-means centroid $\mu_u$, except the wells now sit in feature space: each class is scored by its nearest prototype, and the deepest well claims the image,

It has no free parameters to fit, so its accuracy is a function of $g$ and nothing else. The two heads differ in exactly one freedom: the trained head moves $g$ and $h$ together, the built head rides whatever $g$ the backbone settled into.

The network that never trained#

How far does the head’s independence go? Take the experiment to its limit and train nothing at all. Leave the backbone at its random initialization, every weight a fresh random draw, and build the same head on the garbage features that fall out.

You would expect nothing, and the network as a whole delivers exactly that. Nothing in it has been trained, its own output layer is still a random draw, and it sorts the test set into ten bins that are each a random rainbow. But the built head, reading those same random features, sorts them into mostly-pure bins and lands at 74%.

Seventy-four percent, from a network that never saw a gradient. Random convolutional features are a known free lunch: even an untuned stack of filters scrambles the image in a way that keeps some of its class structure intact, and a head that only has to read structure does not care that nobody trained it. What this endpoint measures is narrow, though. It measures how much class structure survives a random stack, which is a lot. It does not weigh building a head against fitting one on those same frozen features, because this run never trains a head on them; the classical random-features probe would be the control for that, and the comparison drawn below is between building the head and training the whole network end to end.

The whole trade-off, in one curve#

Two endpoints make a story but not a law, so watch the whole way across. At every epoch, freeze the backbone, read its trained head, then build a fresh head on the current features and read that too.

The two curves climb together. The end-to-end curve starts at chance, where nothing has trained yet, and the built head starts at 74%; from the moment the backbone has any real features the two run nearly parallel, a couple of points apart, all the way up. That steady couple of points is what training the classifier is worth once the features exist. Everything else is the backbone lifting both at once.

The number says the representation does the work. The geometry says why.

What was the backbone doing all that time?#

The curve says the backbone did the lifting; the features themselves can show how. Laid out in two dimensions at each stage of training, the climb has a shape. An untrained backbone gives a single tangled blob, and the built head, which labels a point by the nearest prototype, has nothing to grab: points and prototypes are all piled together. Train the backbone and the blob resolves into clusters, one per class. The head never changed. The space under it did, and suddenly nearest-prototype is enough.

That is the whole mechanism. The built head is a fixed rule, which stored point is this closest to, and a fixed rule is only ever as good as the layout it runs on. Training the features is the act of arranging the layout until the fixed rule works.

The head you built is one you can read#

There is a bonus hiding in having built the head instead of training it: you can still read it, the way the earlier posts read a network of pixel prototypes. A prototype is a stored point, and a stored point has a nearest real image, the training example whose features land closest to it. On raw pixels that stand-in was a blurry k-means average. In feature space, where a class clusters tightly, the nearest real image is one sharp, specific example, the this looks like that of prototype networks.

And the reading survives the move even though the backbone does not. The trained backbone is a genuine black box, sixty-four numbers with no individual meaning. The head riding on top is still a bank of pictures, so the classifier stays legible while the representation under it never was. The interpretability lives in the part you place, not the part you train.

And the edits, the teaching and forgetting that made the pixel-space head worth having, do they survive the move into feature space? Unchanged in form, because the head is still a list of prototypes with a one-hot readout. Forgetting a class is deleting its block: drop the Sandal prototypes and Sandal recall falls from 93% to 0% while the other nine classes hold at 82.2% to 82.5%, untouched, because their scores never referenced a Sandal prototype. Teaching a class is appending a block: build the head from eight classes, then add prototypes for Bag and Boot, and they are recognized at 95% each while the head reaches its full 83.2%, exactly as if all ten had been there from the start. Adding and forgetting were never properties of pixels. They are properties of storing a class as its own block of prototypes, which holds in whatever space the prototypes live in.

Build the features too#

The trained backbone is not the only way to arrange a readable space. Before learned vision features, systems used measurements a person could name: oriented gradients, corners, and pooled spatial histograms. Here six fixed edge detectors and one corner detector form the entire vocabulary.

Two Sobel filters give the local gradient. Its magnitude and angle are

Each oriented channel uses a triangular angular window around one of six directions; the corner channel is $|g_x(p)|\,|g_y(p)|$. Pool every channel over a $7\times7$ grid and the representation has $7\times49=343$ coordinates. Each coordinate has a literal name such as “vertical-edge energy in the middle-left patch.”

The same prototype head now reads those 343 values. With twenty class-conditioned k-means centers per class and a nearest-prototype score,

the full pipeline reaches 83.3% on the Fashion-MNIST test set. No gradient step fits the detectors or the readout; k-means still estimates prototypes from data, so “hand-built” means constructed without gradient training, not data-free.

Where the line falls#

Where does the line between optimize and construct fall in this experiment? The random backbone plus constructed head reaches 74%; training the backbone raises the same construction to 83.2%; jointly optimizing the matched network reaches 85.7%. Most of the measured gain comes from the representation, while the final few points remain sensitive to optimizing the head and backbone together. The prototype head can still be placed directly, keeping its class blocks available for explicit teaching and deletion.

The hand-built and learned-feature experiments now give one answer. In this matched setup, generic vision measurements recover most of the score, while joint training buys the last 2.4 points. That is an existence result for Fashion-MNIST and its strong edge prior, not a claim that modern representation learning is generally dispensable. The next experiment keeps the construction explicit and asks what a second composed layer contributes.

The prototype-network idea (“this looks like that”) is from Chen et al. (2019); the strength of random convolutional features from Saxe et al. (2011); the fixed edge vocabulary follows the HOG lineage of Dalal and Triggs (2005); Fashion-MNIST from Xiao et al. (2017); and the Yat kernel from Bouhsine (2026). The combined runnable implementation is Constructing the Fashion-MNIST Network, in JAX/Flax NNX.