Latent on the Spectrum: Why Cats Sit Closer to Dogs Than to Cars
The regular simplex is the perfect codebook only when classes are strangers. Real labels carry a similarity structure, and a latent space is a lossy, finite-dimensional encoding of a kernel — its codebook is the top eigenmodes of the label-similarity matrix. But the prototypes are only the top of the spectrum: the information lives in the modes below them, the continuum between prototypes, and the Welch bound sets the geometry of that channel. A follow-up to the Welch-bound post, with live in-browser experiments — steer a codebook from simplex to taxonomy and watch its spectrum, spend a dimension budget, watch supervision erase structure, watch neural collapse grind the information spectrum to zero, read dark knowledge off a feature wandering the frame, and see a structured codebook make better mistakes.