Opposite Is Not Different: The Cosine-Similarity Bug in CLIP and Contrastive Learning

Suppose you are training a contrastive model, and you want two embeddings, a photo of a dog and a paragraph of tax law, to end up as different as possible. Where on the cosine axis do you send them? Every instinct says $-1$: the scale runs from $+1$ for identical down to $-1$ for opposite, so maximal difference should live at the bottom. That instinct is wired into the losses behind CLIP, SimCLR, and most of modern contrastive learning, whose gradients push every negative pair toward opposition.

But look at what a pair at cosine $-1$ actually is. Two unit vectors $\mathbf{u}, \mathbf{v}$ with $\cos\theta(\mathbf{u}, \mathbf{v}) = -1$ are antiparallel, $\mathbf{v} = -\mathbf{u}$, and antiparallel means linearly dependent. They span a single one-dimensional subspace; knowing $\mathbf{u}$ determines $\mathbf{v}$ exactly. In every algebraic, geometric, and information-theoretic sense they are the same direction with a sign flip. Two vectors at $\cos = -1$ are not different. They are redundant.

The geometry of difference is orthogonality. Vectors with $\cos\theta = 0$ are linearly independent: their span has dimension two, the projection of one onto the other is zero, and neither can be reconstructed from the other. Orthogonality is where genuinely new information lives.

If that is right, then some of the most influential losses of the past decade have been asking their embeddings for the wrong thing. The question worth chasing is what that mistake actually cost, and which losses quietly avoided it.

The cosine scale has three landmarks#

The standard mental model puts cosine similarity on a single axis from $+1$ (“most similar”) to $-1$ (“most different”). That model is missing the structure of vector spaces. Three points on the scale are qualitatively different:

The “difference” axis runs from $\pm 1$ (dependent) to $0$ (independent), not from $+1$ to $-1$.

The information argument is the cleanest way to see it. As a working measure for this post (not an established quantity, but a natural one), define the directional information $I_\text{dir}(\mathbf{u}; \mathbf{v}) = \cos^2 \theta$, the fraction of $\mathbf{v}$‘s variance explained by $\mathbf{u}$. Both parallel and antiparallel give $\cos^2 = 1$: each one is reconstructable from the other up to a sign. Only orthogonality gives $\cos^2 = 0$: neither carries any information about the other.

The thesis in one line:

Where does CLIP send its negatives?#

The natural first test is the loss behind the most famous embedding space in the field. OpenAI’s CLIP is trained with InfoNCE, a softmax contrastive loss over a batch of $N$ image-text pairs:

The gradient with respect to any negative similarity is strictly positive: the loss decreases monotonically as that similarity decreases. Cosine similarity is bounded below by $-1$, so the global minimum of every negative term is antiparallel alignment. CLIP wants every pair of unlike concepts to be linearly dependent on each other.

This is not merely hard at scale; it is unsatisfiable outright. Antiparallelism is not a resource a bigger space supplies more of: the Gram matrix of $k$ unit vectors with every pairwise cosine equal to $-1$ has a negative eigenvalue as soon as $k > 2$, so no more than two vectors can be pairwise opposite in any dimension. What a $d$-dimensional space does supply is $d$ mutually orthogonal antipodal pairs, one per axis of an orthonormal basis, which is why CLIP’s 512-dimensional embedding space carries at most 512 binary oppositions. ImageNet has 1,000 classes; the real world has millions of concepts. A loss demanding $\cos = -1$ for every negative is asking for a configuration that does not exist in the space it’s being computed in.

So does CLIP end up somewhere terrible? No, and the distinction matters. Separate where the loss lands from how it gets there. Because the all-pairs-at-$-1$ configuration is unreachable, what the softmax family actually converges to is the regular simplex, the near-optimal packing derived below, so the equilibrium is fine. The cost lives in the dynamics: the per-pair gradient keeps pointing at $-1$ with no notion of “far enough,” and the optimiser spends its steps, and its batches, pushing toward a target the geometry cannot honour. I trace this equilibrium-versus-dynamics split through the whole contrastive lineage, loss by loss on live points, in Untangling the Moons.

The computational bill of those dynamics is easy to itemise. Random unit vectors in $\mathbb{S}^{d-1}$ concentrate near cosine zero with variance $1/d$, for $d = 512$, the standard deviation of a random pair’s cosine is around $0.044$. The loss’s gradient is dominated by the rare negatives that happen to lie far enough from the equator to register; most of the batch contributes near-zero signal. To accumulate enough gradient, CLIP was trained with $N = 32{,}768$ pairs per step, an $N \times N$ similarity matrix at ${\sim}4$ GB per device, and a multi-year engineering effort in distributed training, gradient caching, and memory-efficient attention. The geometry is not the sole reason those batches help, large batches also surface hard negatives and tame gradient variance, but it is one steady drain among them, and the one this post can state exactly: part of that engineering paid for a target the space could never deliver.

What would a stopping point look like?#

If the damage is in the open-ended push, the fix should be a loss that knows when a negative is far enough, and one loss in the lineage has exactly that. Google’s SigLIP replaces the softmax with a pairwise sigmoid:

with $y_{ij} = \pm 1$ for matched/mismatched and $b$ a learnable bias. The sigmoid gradient on mismatched pairs vanishes once $\mathrm{sim}_{ij} < b$ by a few multiples of $\tau$. The loss does not push negatives toward $-1$; it requires only that they fall below the bias.

That single change aligns the objective with the geometry. Random embeddings on $\mathbb{S}^{d-1}$ already concentrate around cosine zero, exactly where SigLIP is willing to leave them. The loss isn’t fighting the spherical geometry to drag every negative across the equator. The pairwise structure eliminates the $N \times N$ softmax competition. The result is better zero-shot accuracy with smaller batches and less compute. The headline framing is “sigmoid beats softmax,” but the structural framing is sharper: targeting orthogonality lets the network use the geometry it is operating on instead of fighting against it.

Cross-entropy has always targeted orthogonality#

The cross-entropy loss has been doing the right thing the whole time. For discrete distributions $p, q$,

and the key fact is its singularity structure: if $\mathrm{supp}(p) \cap \mathrm{supp}(q) = \varnothing$, then $H(p, q) = +\infty$. Disjoint supports are the probabilistic analog of orthogonal vectors, distributions that share no mass, like vectors that share no projection. The cross-entropy singularity is the orthogonality condition lifted into probability.

The same structure shows up directly in classifier weights. For a softmax classifier with logits $z_k = \mathbf{w}_k^\top \mathbf{h}$ and class label $y$, the gradient on wrong-class logits $z_k$ pushes them toward $-\infty$. On the unit sphere the loss has competing pressures: $\mathbf{w}_y$ wants to be parallel to $\mathbf{h}$ to maximise $z_y$, and each $\mathbf{w}_k$ with $k \neq y$ wants to be antiparallel to $\mathbf{h}$ to minimise $z_k$. With $n$ classes sharing the same $\mathbf{h}$, the antiparallel target cannot be reached by all $n-1$ wrong-class weights simultaneously, they cannot all be $-\mathbf{h}$. The equilibrium is whatever configuration best balances those pressures subject to mutual diversity of the $\mathbf{w}_k$, and the next section shows that configuration is the regular simplex with $\langle\mathbf{w}_k, \mathbf{h}\rangle = -1/(n-1)$: approximately orthogonal for any non-trivial multi-class problem, exactly orthogonal in the $n \to \infty$ limit, and exactly antiparallel only in the binary case $n = 2$. The “well-separated representations” that plain cross-entropy classifiers produce without any contrastive auxiliary loss are not a happy accident. They are the simplex, sitting where the geometry says it should.

The simplex packing result#

What is the optimal arrangement of $n$ class representations on $\mathbb{S}^{d-1}$ for $n \le d + 1$? It is the regular simplex: $n$ unit vectors with all pairwise cosines equal to

For $n = 2$ this is $-1$, exactly antiparallel. The binary case is where the opposition-is-difference intuition came from, and where it is correct. The confusion starts when that intuition is generalized.

For $n \ge 3$ the optimal configuration moves rapidly toward orthogonality. $n = 3$: $-\tfrac{1}{2}$. $n = 10$: $-\tfrac{1}{9} \approx -0.11$. $n = 50$: ${\sim} -0.02$. As $n \to \infty$, $\cos\theta \to 0$. Beyond the smallest cases the simplex is orthogonal up to a vanishing correction, and any loss that insists on $\cos = -1$ for every negative is asking the geometry for something it cannot supply.

What changes if you get the target right#

So what does getting the target right buy? The methods that target orthogonality (SigLIP, plain cross-entropy) share two properties: they align with the natural concentration of measure on $\mathbb{S}^{d-1}$, and they reach comparable or better accuracy with substantially less compute. The methods whose gradients chase opposition (SimCLR, CLIP, SupCon) run with enormous batches, and while batch size earns its keep in other ways too, more hard negatives, lower gradient variance, part of what those batches pay for is the standing tension between the objective and the sphere it lives on.

The thesis, in one sentence: the geometry of difference is not opposition; it is orthogonality. The equilibrium was never the problem, the simplex these losses settle into is nearly optimal; the problem is a gradient that never knows where to stop. The most influential contrastive losses of the past five years spent real engineering effort compensating for that geometric mistake, one factor among several behind their appetite for compute, and the loss that did not make the mistake was sitting beside them the whole time, in the form of plain cross-entropy.

@misc{bouhsine2026oppositeisnotdifferent,
  author       = {Bouhsine, Taha},
  title        = {Opposite Is Not Different},
  year         = {2026},
  month        = {feb},
  howpublished = {\url{https://tahabouhsine.com/blog/opposite-is-not-different/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}

@unpublished{bouhsine2026opposite,
  author = {Bouhsine, T.},
  title  = {Opposite ≠ Different: The Orthogonality Thesis},
  year   = {2026},
  note   = {Unpublished manuscript}
}