What Can a Weight Be?

Pick a kernel and the weight it hosts can have a sharp corner, a kink, a sudden spike. Pick a different one and that corner is not merely discouraged, it is impossible: no weight in that space has one, ever. Same data, same picture of a weight sitting among the points that the last post arrived at. The kernel, quietly, decided what kinds of function the weight is permitted to be.

So these homes are not interchangeable, and it is worth knowing which one you just moved into. Once a weight has an address, what is it actually allowed to be?

A kernel is a price list for roughness#

What decides who gets in? Money, in effect. Picture every candidate weight arriving with a shopping list: it is assembled from a fixed catalog of basic shapes, and each shape has a posted price. Smooth, slow shapes are cheap; sharp, fast, wiggly ones are expensive, and how expensive depends on the kernel. The space admits exactly the weights that can afford their total bill. A kernel is, secretly, a price list for roughness.

Mercer’s theorem hands you that catalog outright. Every kernel splits into a set of modes $\phi_i$, fixed shapes, each with a non-negative weight $\lambda_i$:

The reproducing kernel Hilbert space built from $k$ is then nothing but the functions you can write in those modes, $f = \sum_i c_i \phi_i$, with a very particular notion of size:

That norm is the bill. To use mode $i$ with amplitude $c_i$ you pay $c_i^2/\lambda_i$. A mode with a large eigenvalue $\lambda_i$ is cheap; a mode with a tiny eigenvalue is ruinously expensive. The RKHS is exactly the set of functions whose total bill is finite, the functions you can afford. And since sharp features, corners and spikes, are built from the high-frequency modes, whether the weight can be rough comes down to one thing: how fast the eigenvalues decay. That single curve is the kernel’s whole personality.

Sobolev: the home with room for a corner#

So which kernels let the weight be sharp? The Matérn family and the Laplace kernel $e^{-\lVert x-y\rVert}$ have spectral densities with polynomial tails. Slow decay means high frequencies are merely expensive, not effectively forbidden. The inverse-multiquadric belongs on the other side of this comparison: its Fourier tail is exponential-type, so its RKHS is smoother than a finite-order Sobolev space.

That has an exact name. The RKHS of such a kernel is norm-equivalent to a Sobolev space $H^s$, the functions whose first $s$ derivatives are square-integrable:

For the Matérn kernel of smoothness $\nu$ in $d$ dimensions, $s = \nu + d/2$. A Sobolev space is the natural home of functions with a finite amount of smoothness. A function with a corner has a finite Sobolev norm for small enough $s$, so it sits comfortably inside. The weight is allowed to be sharp. This is the right-sized home: big enough to hold the rough functions real data needs, structured enough to still penalize nonsense.

The Gaussian: a home too small to bend#

Now the cautionary opposite, and it is the kernel almost everyone reaches for first. The Gaussian, or RBF, kernel $e^{-\lVert x-y\rVert^2 / 2\sigma^2}$ has eigenvalues that do not decay like a polynomial. They fall off like a Gaussian themselves, faster than any power of the frequency. So the high-frequency modes are not expensive, they are astronomically expensive, and the bill for any function with a sharp feature is not large but infinite.

The consequence is severe and not widely felt: the RKHS of the Gaussian kernel contains only analytic functions, the infinitely smooth ones, and it is a strict subset of every Sobolev space at once. A function with a corner is not in it. A bump that is exactly zero outside a small region is not in it. The Gaussian kernel does not just prefer smooth weights, it makes a non-smooth weight literally impossible to express. You feel this the moment you fit data with a real edge in it.

The sphere: the home of normalized data#

Both of those homes sit in flat space, but the data in this series does not. We L2-normalize every feature vector, which puts every point on a sphere, and on a sphere the natural kernels are the ones that depend only on the angle between two points, $k(x, y) = \kappa(\langle x, y\rangle)$. These are the zonal, or dot-product, kernels, and Schoenberg settled their structure in 1942: such a kernel is positive definite on the sphere exactly when it expands in Gegenbauer polynomials with non-negative coefficients.

What makes the sphere beautiful is that its modes are already famous. The eigenfunctions are the spherical harmonics $Y_{k,j}$, the standing-wave patterns of a sphere, graded by a degree $k$: degree zero is a constant, degree one is a single smooth swell, and the patterns ripple faster as $k$ grows. A zonal kernel assigns one price $b_k$ to an entire degree at once, and its RKHS is the functions on the sphere whose harmonic content it can afford:

Same skeleton as before, a price list over modes, but now the modes are the vibrations of a sphere and the smoothness of the weight is set by how fast the kernel lets the degrees fade.

The rest of the zoo, in one line each#

And the rest of the zoo? The same lens sorts every kernel you have met, one line each. A polynomial kernel keeps only finitely many modes alive, so its RKHS is finite-dimensional: the weight can only be a polynomial of bounded degree, and there is no room for anything finer. A bandlimited, or sinc, kernel sets its prices to a hard step, one below a cutoff frequency and zero above, so its RKHS is the bandlimited functions and the cutoff is an absolute wall. On $[0,1]$, the Brownian-motion kernel $\min(s,t)$ has the Cameron–Martin space: absolutely continuous functions $f$ with $f(0)=0$ and $f'\in L^2$, equipped with $\lVert f\rVert^2=\int_0^1|f'(t)|^2dt$. It is an anchored first-order Sobolev space, not all of $H^1$. Three kernels, three decay rates, three different guest lists—and the domain and boundary condition help define each list.

Regularization sets the budget#

The price list and regularization are one mechanism, not two stories. For fixed-kernel ridge regression,

the representer theorem gives $\hat f_\lambda(\cdot)=\sum_i\alpha_i k(x_i,\cdot)$ and the coefficients solve a linear system. In an eigenbasis of the declared kernel operator, regularization shrinks mode $j$ by a factor of the form $\mu_j/(\mu_j+\lambda)$, up to the normalization convention used for the empirical objective and Gram matrix. Large-eigenvalue modes survive; small-eigenvalue modes are suppressed.

That same filter defines an effective dimension,

This is a property of a specified operator or empirical Gram matrix at a specified regularization level. It is not a dataset-free capacity number. Nor does a small effective dimension, by itself, prove generalization: a rate also needs a sampling model, noise assumptions, target regularity, and a statement connecting the population and empirical spectra.

The point-space and mode-space views are equivalent for the fixed problem. Coefficients $\alpha_i$ say which sampled kernel sections assemble the predictor; eigen-coordinates say which spectral shapes survive the budget. Ridge shrinkage is graded rather than sparse, so a nonzero training point is not automatically a support vector in the SVM sense. If a sparse explanation is wanted, sparsity must be introduced and measured rather than read into the RKHS norm.

Where the prototype lives#

So when this series puts a prototype in the space, which home does it move into? The Yat kernel is a squared alignment $(\langle x,W\rangle+b)^2$ multiplied by an inverse-multiquadric distance gate $1/(\lVert x-W\rVert^2+\varepsilon)$. Locality is visible in that formula. Its full native-space smoothness is not: the answer depends on the complete product kernel, the domain, and the measure. A Fourier tail for the distance gate on $\mathbb R^d$ cannot by itself establish the spectrum of the full kernel on a normalized sphere.

What its home is instead is the more useful thing here. Under the domains and measures analyzed in the cited paper, its spectrum has exponential-type decay slower than the corresponding Gaussian spectrum, so the fixed RKHS is larger in that scoped comparison. Universality requires the constant alignment term, $b>0$, together with $\varepsilon>0$; at $b=0$ the even alignment factor loses odd functions on symmetric domains. On normalized inputs the kernel is zonal, so spherical harmonics supply the coordinates. The property that makes a prototype a picture is on a different axis from frequency smoothness: each section is centered at an input-space point. Locality and bounded response then depend on the bounded, normalized domain used by these experiments, not on PSD alone.

A weight can be as rough as the kernel’s eigenvalues let it afford to be, on the geometry and measure used to define the operator. Choosing a kernel is not merely choosing a similarity score; it is choosing a spectral cost structure and a shape of space. For the prototype networks in this series, the directly established source of visual legibility is simpler: each learned center lives in input coordinates and can therefore be rendered. Spectral smoothness is a different property and should be computed from the full kernel before it is named.

Positive-definite kernels on the sphere are Schoenberg (1942); native spaces and the Sobolev correspondence are in Wendland (2004) and Stein (1999); the Gaussian RKHS is treated in Steinwart and Christmann (2008); the polynomial-alignment and IMQ construction is Bouhsine (2026). The runnable companion constructs valid periodic spectra and audits truncated RKHS norms in JAX: A Kernel’s Price List, in JAX. The conceptual predecessor is Where Does a Weight Live?.