Activations Are Bad for Geometry

Somewhere today, someone is computing cosine similarities between embeddings, running $k$-nearest-neighbors in a representation space, clustering the activations of a hidden layer, and trusting what comes back. The trust rests on an assumption so default that it is rarely said out loud: the geometry of the representation space means something. Nearby points are near because the data was similar; angles measure relationships; distance is information. So here is the uncomfortable question this post exists to press: how much of that geometry is a property of your data, and how much is an artifact of what the network’s activation functions destroyed on the way up?

The destruction has a specific and inspectable mechanism. A neural network layer is a map, and whether that map preserves the geometry of the data manifold, distances, angles, volumes, is decided by its Jacobian. For the standard form $F(\mathbf{x}) = \phi(\mathbf{W}\mathbf{x} + \mathbf{b})$, with input $\mathbf{x} \in \mathbb{R}^d$, layer width $n$, and $\phi$ applied coordinatewise, the Jacobian factors as

where $\mathbf{D}_\phi(\mathbf{z}) = \mathrm{diag}(\phi'(z_1), \ldots, \phi'(z_n))$. Read the factorization as a division of labour. $\mathbf{W}$ does all the rotating and mixing; the activation contributes only $\mathbf{D}_\phi$, a diagonal matrix that can do nothing but rescale coordinates, by whatever values $\phi'$ happens to take at the $n$ pre-activations the input produces. Under almost every activation in current use those values are zero, small, or bounded, and none of that is accidental: it is exactly what gives the activation its selectivity. It is also, as we are about to see, what lets it quietly vandalise the geometry your cosine similarities assume is real.

The Jacobian, activation by activation#

How much damage can one diagonal matrix really do? The singular values of $\mathbf{J}_F$ are the singular values of $\mathbf{W}$ scaled coordinatewise by the entries of $\mathbf{D}_\phi$, so the answer turns entirely on what $\phi'(z)$ looks like at the $n$ pre-activations the layer actually sees.

The picture is uniform. ReLU gives $\phi'(z) \in \{0, 1\}$, a hard gate that zeroes rows of $\mathbf{J}_F$ on the negative half-plane. Sigmoid and tanh never zero exactly but saturate at both ends; small singular values multiply, and the Jacobian becomes ill-conditioned the moment any coordinate is far from zero. Leaky ReLU keeps $\phi'$ strictly positive but pins it into $\{\alpha, 1\}$ with $\alpha \ll 1$, taking a factor-of-$1/\alpha$ hit to the condition number in expectation. GELU and softplus are smooth and $\phi' > 0$ everywhere; their conditioning cost is mild, gradual, and never zero.

These differences are not stylistic. They are the difference between a layer that can collapse, a layer that can degenerate, and a layer that mostly behaves.

Rank collapse, made concrete#

What does the hard gate actually cost? For ReLU the answer is exact. With active set $S(\mathbf{x}) = \{ i : z_i > 0 \}$,

the rank of $\mathbf{W}$ restricted to its surviving rows. If the rows of $\mathbf{W}_S$ no longer span $\mathbb{R}^d$, the Jacobian loses column rank and a neighborhood of $\mathbf{x}$ is crushed onto a lower-dimensional subset of output space. Information that lived along the killed directions is gone in the strong sense, the next layer receives the same image regardless of where you were inside $\ker \mathbf{J}_F$.

Surely width saves you here. Make the layer wide, $n \gg d$, and even with half the rows gated off, the surviving submatrix is almost certainly still full column rank; the gate fires, and nothing is lost. That reasoning is correct, and it is exactly the reasoning that fails, because it is correct for one layer. A network is a product of layers,

and ranks of products can only shrink: rank lost at any layer, at any depth, is lost end-to-end, because no later layer can resurrect a direction that has already been crushed to a point. Width buys you a good chance of surviving each round; depth makes you play the round again and again, and every loss is permanent. Even short of exact collapse, the near-losses compound too, each layer’s small singular values multiply into the next layer’s, and the end-to-end Jacobian drifts toward effective rank one. Residual connections bias each layer toward $\mathbf{I}$ plus a small perturbation and mitigate the compounding, but they do not guarantee against it.

You do not have to take the multiplication argument on faith; you can watch it happen.

Play with it for a minute and the depth story stops being abstract. At depth one the ellipse is a healthy ellipse almost everywhere. By depth six or eight, whole regions of the input plane have gone needle-shaped: the effective rank hovers near one, and every input in such a region is being told about through a single surviving direction. Nothing pathological was chosen here, the weights are ordinary random matrices, which is the point: this is what stacked pointwise gating does by default.

When activations don’t break things#

The negative result has a positive sibling. If $\phi$ is strictly monotone and $\mathbf{W}$ has full column rank, then $F$ restricted to any compact data manifold $M \subset \mathbb{R}^d$ is a homeomorphism onto its image, distinct points stay distinct, topology is preserved, no rank collapse anywhere. If $\phi$ is also smooth, the pullback metric $g_M$ is a well-defined Riemannian metric (possibly ill-conditioned, but never singular).

The test sorts the standard activations cleanly. Sigmoid, tanh, GELU, softplus, and the identity are all strictly monotone; under a full-rank $\mathbf{W}$ they preserve topology, and their only sin is condition number. ReLU is not strictly monotone, it has a flat half-line, and that flat half-line is the source of every pathology above. Leaky ReLU with $\alpha > 0$ scrapes by: it is strictly monotone, so $g_M$ is well-defined, but $\phi'$ has a jump at zero, so $g_M$ is only piecewise smooth.

The corollary is that the activation question is largely the question of strict monotonicity. Lose it, and you lose homeomorphism on a measurable region of input space. Keep it, and the only thing left to manage is conditioning.

What the activation does to the metric#

The trouble with everything so far is that you cannot picture it. Rank, kernels, the singular values of a Jacobian in hundreds of dimensions: none of that is an image your intuition can hold. But there is a physical picture that carries all of it. Think of the data manifold as a printed sheet of graph paper and the layer as a hand that deforms it. A smooth, strictly monotone activation stretches the sheet, unevenly, but without tearing or creasing: grid lines stay grid lines. Saturation crushes whole regions of the grid toward a point. And ReLU folds the sheet, pressing everything on the wrong side of the crease flat onto the crease itself. Once folded, two different points of the sheet lie on top of each other, and no later hand can pull them apart again; it can only carry the folded stack along.

The correspondence is exact, not decorative:

And the picture predicts before the algebra confirms: a fold cannot be undone by any further stretching, which is exactly the rank-lost-is-lost-end-to-end rule you just watched compound across depth.

With the picture in hand, the equation can name it. The grid spacing the deformed sheet carries is the pullback metric induced by the layer on a submanifold $M \subset \mathbb{R}^d$,

This is the metric the network thinks the data lives in. $\mathbf{D}_\phi$ enters twice, squared, with two consequences worth naming.

Directional rescaling. Each row of $\mathbf{W}$ is weighted by $\phi'(z_i)^2$ in $g_M$. Sigmoid and tanh saturation, the leaky-ReLU $\alpha$ slope, every situation in which a $\phi'(z_i)$ goes small: all push the corresponding row’s contribution toward zero. The “learned distance” the layer imposes is dominated by the rows whose neurons haven’t saturated; the rest contribute almost nothing to perceived similarity.

Directional erasure. When $\phi'(z_i) = 0$ exactly, the row drops from $g_M$ entirely. The metric becomes singular along directions in $\ker \mathbf{J}_F \cap T_\mathbf{x} M$: distances collapse to zero. This is the manifold-side picture of rank collapse, the geometric statement that the layer has stopped being a homeomorphism at $\mathbf{x}$.

The sheet metaphor turns literal in the panel. Identity and GELU stretch the disk; sigmoid and tanh compress it without folding; ReLU folds the negative half-planes onto the axes and crushes entire wedges of the disk onto a 1D set. There is no separate metric tensor the network keeps somewhere, the post-activation grid spacing is the metric.

High dimensions make this worse, not better#

The intuition that “with a wide enough layer, ReLU sparsification is fine” survives in width but not in pressure. Under the simplest model, $\mathbf{z} \in \mathbb{R}^n$ with each $z_i$ independent and symmetric, the probability that at least one coordinate is zeroed is

By $n = 10$ this is $99.9\%$. In a transformer hidden layer of width $768$, every forward pass has approximately half its coordinates zeroed at every point. Whether this turns into rank collapse depends on the structure of $\mathbf{W}_{S(\mathbf{x}), :}$, but the pressure toward sparsification does not disappear in the limit, it becomes the operating regime, and the analysis above stops being worst-case and becomes typical.

The point is not that ReLU is bad in high dimensions. It is that high dimensions are exactly where the geometric pathologies of pointwise activations live, and that handwaving about width does not make them go away.

The expressivity–geometry tradeoff#

Why have an activation at all? Without one, a stack of $L$ layers is the single linear map $\mathbf{W}_L \cdots \mathbf{W}_1$, no nonlinear class boundary, no useful expressivity. The activation buys selectivity: $\phi'(z_i) \approx 1$ when a neuron’s prototype matches the input and the projection passes through, $\phi'(z_i) \approx 0$ when it doesn’t. Selectivity is what the activation is for.

But selectivity is exactly what damages geometry. Sharper activations, derivatives closer to $\{0, 1\}$, separate classes better and lose more geometry. Smoother, never-zero activations preserve more geometry but suppress selectivity, leaving the layer near its linear part. The choice of activation is the choice of where to sit on this axis. ReLU is a corner solution: maximum selectivity, maximum geometric damage. GELU and softplus are middle solutions. Identity is the other corner, perfect geometry, no expressivity gain over a single linear layer.

There is no escape from this tradeoff as long as the nonlinearity is pointwise. Every dimension spent on selectivity is taken from the metric.

Reading common tricks as Jacobian regularization#

If pointwise activations really are a standing threat to the Jacobian, you would expect the field to have been fighting them all along, without necessarily saying so. It has. Several standard practices, usually treated as separate phenomena, are all variations on a single intervention: keep $\mathbf{J}_F$ away from rank collapse and saturation.

Residual connections turn each layer into $\mathbf{x} + F(\mathbf{x})$, whose Jacobian is $\mathbf{I} + \mathbf{D}_\phi \mathbf{W}$ instead of $\mathbf{D}_\phi \mathbf{W}$. The identity term gives the Jacobian a floor, full rank by construction, well-conditioned as long as $\|\mathbf{D}_\phi \mathbf{W}\|$ stays modest. The cumulative rank decay that plagues stacked ReLU layers becomes a perturbation around $\mathbf{I}$ instead of a multiplicative product of degenerate matrices.

Batch and layer normalization rescale the pre-activation $\mathbf{z}$ to roughly zero mean and unit variance. This is exactly the regime in which sigmoid/tanh/GELU have their largest $\phi'$ and ReLU has its highest active fraction. Without normalization, $\mathbf{z}$ drifts during training; the saturation set grows; $\mathbf{D}_\phi$ shrinks. Normalization holds the input distribution in the activation’s live zone, $\mathbf{D}_\phi$ stays away from zero.

Weight and spectral normalization bound the singular values of $\mathbf{W}$. They have no direct effect on $\mathbf{D}_\phi$, but by keeping $\mathbf{W}$‘s spectrum tight they prevent the linear factor from compounding whatever damage $\mathbf{D}_\phi$ has already inflicted.

These are not activation replacements. They are stabilizers, they keep the architecture in the regime where $\mathbf{J}_F$ is least bad. The fact that the same analysis explains three different “tricks” is the content: each one holds a different piece of the Jacobian away from a different failure mode.

Why this matters for evaluation#

This is where the question from the top comes due. Cosine similarity, $k$-nearest neighbors, clustering, retrieval: every downstream operation on representations assumes the space carries the geometry it is reading. If the layer has collapsed rank, cosine similarity compares vectors whose angles are artifacts of the surviving directions rather than properties of the data manifold. If the layer has saturated, small distances in representation space correspond to entirely different scales of input distance depending on which coordinates were saturated where. The metric you evaluate with is not the metric the network actually exposed.

The conclusion is not “don’t use cosine similarity.” It is that cosine similarity (and every other downstream metric) is only meaningful when the network has preserved the geometric structure the metric is reading. Choose activations that preserve Jacobian rank where the task requires it. Control input magnitudes via normalization so the activation does not saturate. Match the evaluation metric to the geometry the architecture actually preserves. None of this is optional if the goal is to compare representations rather than collateral.

The kernel alternative#

The tradeoff exists because the architecture has separated geometry (carried by $\mathbf{W}$) from selectivity (provided by $\phi$), and each piece can do its job only at the other’s expense. A kernel-machine layer dissolves the separation. A symmetric positive-definite kernel $K(\mathbf{z}, \mathbf{z}')$ provides selectivity (sharper kernels are more selective) and geometry (a Gram matrix is a metric) at the same time. The function the layer computes is a finite expansion in kernel sections,

with a closed-form RKHS norm $\|f\|^2_{\mathcal{H}_K} = \boldsymbol{\alpha}^\top \mathbf{K} \boldsymbol{\alpha}$. There is no $\mathbf{D}_\phi$ sitting in the middle to collapse rank or saturate the metric. The primitive is the geometry; selectivity is implemented as geometry; kernel similarity is the score.

The pointwise activation is not the price of expressivity. It is the price of refusing to make the primitive a kernel. Pick activations with the same care you pick a loss. They are not there for nonlinearity, and they are not free.

@misc{bouhsine2026activationsarebadforgeometry,
  author       = {Bouhsine, Taha},
  title        = {Activations Are Bad for Geometry},
  year         = {2026},
  month        = {feb},
  howpublished = {\url{https://tahabouhsine.com/blog/activations-are-bad-for-geometry/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}

@unpublished{bouhsine2026manifoldsactivations,
  author = {Bouhsine, T.},
  title  = {Manifolds, Activations, and Lost Geometry: How Pointwise Nonlinearities Break the Map},
  year   = {2026},
  note   = {Unpublished manuscript}
}