#kernels
2 posts tagged #kernels.
-
Attention is Explainable Because it is a Kernel
A reading of self-attention through kernel smoothing and RKHS.
-
Activations Are Bad for Geometry
Pointwise activations factor into the layer's Jacobian as a diagonal modulation. The same modulation that buys selectivity destroys geometric structure on the data manifold.