Untangling the Moons: A Visual History of Contrastive Learning

Twenty years of contrastive learning, eight losses, ten datasets, and the question of when a loss should stop pushing.

Contrastive learning is a standard recipe for turning raw data into a usable embedding space, and the recipe has been rewritten repeatedly. Read as a conceptual lineage, each rewrite changes a different part of the optimization problem. Triplet loss makes separation relative to an anchor. InfoNCE replaces explicit mining with a softmax over candidates, though it arrived from predictive coding and mutual-information estimation rather than as a triplet patch. CLIP scales the construction to paired modalities; SupCon admits many positives; alignment-and-uniformity decomposes two aggregate forces; SigLIP replaces global softmax normalization with pairwise logistic terms.

The geometric question is joint, not pairwise. Holding the rest of an InfoNCE row fixed, lowering one negative similarity lowers the loss. But every embedding participates in many terms, and the Gram matrix must remain feasible, so training cannot minimize each pair independently. A centered regular simplex is one attainable class-level arrangement when dimension permits; other objectives and sampling schemes produce other compromises. Here the demonstration is mechanical: in 2D, with the points being the embeddings, you can watch the coupled forces evolve.

The 2D panels answer a deliberately narrow question: what do these loss formulas do when the embeddings themselves are optimized? They do not reproduce encoder training, semantic negatives, or high-dimensional class geometry. Their value is visibility. You can inspect which terms become exactly inactive, which decay smoothly, which couple a whole row, and which objectives remain frustrated because the requested arrangement does not fit in two dimensions.

The fastest way to feel any of this is to skip the encoder entirely. Take 60 points in 2D, label them by class, and treat the positions themselves as the embeddings. Run gradient descent on the loss of your choice. Eight losses, eight geometries, eight failure modes, each visible in under a minute.

This post walks the lineage by idea, not strictly by date (CLIP and SupCon are contemporaneous, 2020–21, and ordered here by the conceptual thread, not their arXiv stamps). Each loss gets its own explorer, and the rest of the presets surface that loss’s named pathologies on demand. At the end, all eight race on the same dataset, from the same initial points, at the same step counter.

The setup#

What should you watch as the points move? Two things, and every explorer shows both: the embedding space on the left, where the geometry unfolds, and the loss curve with a running nearest-centroid accuracy on the right, which tells you whether the motion is progress or churn. Losses that use cosine similarity project to the unit circle every step, the dashed reference circle appears for those. Losses that use Euclidean distance leave the points free.

Ten datasets are available in the dropdown:

• random (default), uniform positions, random labels, no spatial signal. The loss must impose everything.
• random (4 classes), same, four-way.
• two-moons, Hadsell’s original test case. Two interlocking half-circles.
• overlapping blobs, heavy class overlap.
• concentric rings, same angular distribution, different radii. Diagnoses cosine-based losses.
• four classes / eight classes, multi-class stress.
• two spirals, imbalanced moons, noisy moons, adversarial settings.

Every explorer opens on random positions and random labels, because a pre-organised dataset lets the loss get away with doing very little and the thing worth watching is the configuration it imposes when nothing is handed to it.

Before the history: three meanings of different#

Put two unit vectors on a circle. At cosine $+1$ they coincide. At cosine $-1$ they are antiparallel. At cosine $0$ neither projects onto the other. None of those facts establishes statistical independence, which belongs to random variables and a joint distribution rather than to one deterministic pair of arrows.

For one InfoNCE anchor,

Lowering one negative logit lowers this anchor’s loss while the other logits are frozen. That exact local derivative is not a global instruction to put every negative pair at cosine $-1$. Shared embeddings move many logits together, and their Gram matrix must remain positive semidefinite.

For $n$ unit vectors, feasibility gives

so the average pairwise inner product is at least $-1/(n-1)$. When $n\le d+1$, a centered regular simplex attains that bound with every off-diagonal cosine equal to $-1/(n-1)$. The binary case is the exceptional one where the value is $-1$; as the class count grows, the feasible centered value approaches zero.

SigLIP changes the coupling by replacing a row-wise softmax with pairwise logistic terms. Its negative gradients become small below a learned bias-relative transition but vanish only asymptotically; it does not install a universal equilibrium cosine. Softmax cross-entropy likewise has no finite minimizer on separable data without normalization or regularization. Simplex-like limits require their assumptions to be named. With that geometry fixed, the history below becomes easier to read: each loss changes a pressure profile or a coupling rule, not the meaning of cosine itself.

1. The original: Hadsell, Chopra, and LeCun, 2006#

The modern contrastive era opens with Hadsell, Chopra, and LeCun’s Dimensionality Reduction by Learning an Invariant Mapping, a siamese network for face verification with what is now the canonical pair loss. For every pair of points you know whether they share a label. Pull positives together with a quadratic penalty; push negatives apart until they reach a margin $m$, and then stop:

The margin does all the work. Without it the repulsion has no scale and the embedding blows up. With it the loss is satisfied as soon as negatives are far enough, and then the gradient vanishes. Pair contrastive knows when to stop, which is both its virtue and its problem.

The named failure modes:

• Margin too small. No reward for separating classes far. The loss drops to zero almost immediately while accuracy stays near chance, so the curve looks healthy and the embedding is useless. Hadsell’s paper already has a name for it: the trivial solution.
• Margin too large. Repulsion never saturates, so negatives keep being pushed further forever and the embedding norm grows without bound. In a real network this is where gradients explode.
• Class imbalance. Every pair weighs the same in the gradient, so an 80/20 split lets the majority drag the minority wherever it goes.

By 2014 the problem with pair contrastive was visible in production face-recognition systems: $m$ is a global hyperparameter that wants to be different for different parts of the data manifold. The fix would be to make the comparison relative, to ask not is this pair close enough? but is the positive closer than the negative, by how much? That is the next loss.

2. The relativisation: FaceNet, 2015#

So what does a relative margin look like? Instead of asking two separate questions about positives and negatives, ask one combined question: is the negative farther than the positive, by at least margin $m$? That reformulation is Schroff, Kalenichenko, and Philbin’s FaceNet.

Only triplets that violate the inequality contribute gradient. The hope was that this makes training focus naturally on the hardest examples, automatically. The reality, which the FaceNet paper itself discovered and the next decade of work spent fighting, is that most random triplets are easy, and a triplet that is easy contributes no gradient at all. In a batch of $B$ examples there are $O(B^3)$ candidate triplets and almost all of them are silent.

The named failure modes:

• Easy triplets. At initialisation, random triplets are usually satisfied trivially. The loss is near zero from step one and no learning happens. FaceNet’s own response was semi-hard mining: pick the negative that violates the triplet by the smallest positive amount. By 2017 there were online miners, batch-hard miners, miner schedules, and an industry of triplet-mining infrastructure, all of it compensating for a loss that, left alone, has no gradient on most of its input.
• Aggressive margin. Set $m$ bigger than the typical inter-class distance and no triplet is ever satisfied; the embedding wanders stochastically.
• Class imbalance, arriving this time through sampling rather than weighting: anchors are drawn at random, so most of them come from the majority class.

Pair contrastive and triplet together define what the literature later named the margin family, losses whose gradient vanishes once a margin condition is satisfied. The margin family’s appeal is that it knows when to stop. Its problem is that it stops too soon, and selecting which examples it should think about is its own engineering project. By 2018 the field was ready to give up on margins entirely.

3. The softmax turn: van den Oord and SimCLR, 2018–2020#

Van den Oord, Li, and Vinyals, working on contrastive predictive coding, replaced the margin with a softmax. For an anchor $a$ with one positive $p$ and a batch of $N$ negatives:

Three structural changes from triplet. All negatives at once: every other point participates as a negative, weighted by its similarity to the anchor; hard negatives get most of the gradient automatically, eliminating the explicit miner. No hard margin: the softmax has support everywhere, so the gradient never vanishes, there is always an incentive to spread the negatives a bit wider. Embeddings on a sphere: cosine similarity ignores the norm.

The 2020 wave, SimCLR, MoCo, CLIP, all run on variants of this loss. The name InfoNCE comes from the loss being a variational lower bound on mutual information $I(X; Y)$, though the bound is capped at $\log N$ and so is loose at high MI (Poole et al., 2019); the name NT-Xent (normalised-temperature cross-entropy) comes from the SimCLR paper. They are the same loss.

The named failure modes are almost all about the temperature $\tau$:

• τ too low. The softmax becomes peaky. Gradients are dominated by the single hardest negative each step, and any noisy step destabilises the embedding. The loss curve becomes saw-toothed. Wang & Isola (next section) prove this corresponds to over-emphasising uniformity at the expense of alignment.
• τ too high. The softmax flattens; all negatives contribute roughly equally; the gradient is weak. Convergence is glacial.
• Concentric rings. Cosine similarity throws away the norm, so two concentric rings, different radii, same angles, become identical after projection. Accuracy at chance. This is not a defect of InfoNCE; it is a defect of cosine, and any loss that lives on the sphere inherits it.

I argued at length in Attention is Explainable Because it is a Kernel that the softmax over similarities in attention is mathematically a Nadaraya–Watson smoother, the kernel operator. InfoNCE is the same operator, the same softmax, pointed at a different objective: optimize positions so the kernel-weighted distribution of labels around each anchor matches its own label.

By 2019 the softmax family had a different problem. It was designed for self-supervised learning where each anchor has exactly one positive: the augmented view of the same image. When labels exist, every other example with the same label is also a positive, and InfoNCE was throwing away that signal one sample at a time. Before the labelled fix appeared, the loss took a detour through multimodality.

4. Multimodal scaling: CLIP, 2021#

Radford et al.’s Learning Transferable Visual Models From Natural Language Supervision, the CLIP paper, was the loudest deployment of InfoNCE the field has seen. CLIP trains an image encoder and a text encoder jointly on $400$ million image-caption pairs, with a single objective: each image’s embedding should be closest, in cosine distance, to its caption’s embedding, out of a batch of $N$ candidates. The loss is symmetrised InfoNCE: image-as-anchor and text-as-anchor, averaged.

Each side is an InfoNCE term. The symmetrisation is what makes the trained embedding bidirectional: a query in either modality retrieves nearest neighbours in the other.

CLIP’s failure modes are InfoNCE’s failure modes, amplified by scale. CLIP doesn’t freeze the temperature, it learns the logit scale as a log-parameterised scalar (initialised around $\tau = 0.07$ and clipped to keep logits bounded), but that only relocates the sensitivity rather than removing it: the learned scale is still the main throttle on how hard negatives keep separating, and there is no target that says stop. The spherical geometry compounds it: the embedding dimension ($512$ in the public release) was chosen for matrix-multiplication throughput, not for the simplex bound that $\mathrm{ImageNet}$-scale class counts would need; and the literal compute cost, measured in GPU-years, is staggering, much of it spent on the gradient bookkeeping to keep a $32,768 \times 32,768$ similarity matrix on-device.

This is where coupling becomes computationally visible. Each CLIP row compares one match with every candidate caption in the batch, and the symmetric term repeats the comparison in the other direction. Larger batches provide more negatives and change the gradient estimator, while requiring more similarity computation and communication. The geometry constrains the joint arrangement; it does not by itself explain CLIP’s chosen batch size.

5. Reintroducing supervision: SupCon, 2020#

What happens to InfoNCE when labels return and every same-class example is a legitimate positive? Khosla et al.’s Supervised Contrastive Learning answered by generalising the loss to multiple positives, averaging the InfoNCE term over the positive set $P(a)$:

The effect is dramatic. Where InfoNCE pulls each anchor toward a positive each step, SupCon pulls it toward the centroid of all positives. Classes collapse to tight clusters on the sphere, much faster than InfoNCE, and the paper’s headline result was beating plain cross-entropy on ImageNet classification accuracy with a two-stage pre-train-then-fine-tune recipe.

The named failure modes of SupCon are not about under-fitting. They are about being too good a classifier:

• Representation collapse. With low $\tau$ and many positives, each class collapses to a single point on the circle. Linearly separable to perfection; but anything you cared about within a class, pose, style, lighting, the things downstream non-classification tasks need, is gone. SupCon is approximately optimal for downstream classification and approximately worst for downstream tasks that need within-class variation. This within-class collapse, and the simplex arrangement the class means settle into, is neural collapse, named by Papyan, Han & Donoho (2020) for cross-entropy and connected to SupCon by Graf et al. (2021). Whether you like it depends on what you want the embedding for.
• Imbalanced classes, for the third time in this post and by a third mechanism: $|P(a)|$ varies across classes, so the per-anchor average sits on a different scale for each one and the softmax denominator belongs to the majority. Every loss in the lineage breaks under imbalance; each one breaks in its own way, which is why no single fix travels between them.
• Label noise. SupCon trusts its labels completely. A flipped label becomes a positive pulled toward the wrong centroid, dragging the cluster boundary every step. Robust-SupCon variants exist; vanilla does not handle this.

By 2020 InfoNCE and SupCon between them dominated practical embedding work, and the field had a new conceptual problem. The losses worked, but it was unclear why. The softmax form bundles several things together, what is the right way to think about what these losses are doing in the limit?

6. The theoretical decomposition: Wang and Isola, 2020#

The answer arrived the same year. Wang and Isola’s Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere proved that InfoNCE, in the limit of infinite negatives, is doing two separable things:

with total loss $\mathcal{L}_{\text{align}} + \lambda \mathcal{L}_{\text{uniform}}$. The uniformity term is the log-MMD of a Gaussian kernel, minimising it is the same as maximising the entropy of the embedding distribution over the sphere.

The decomposition is more than aesthetic. It says the two jobs are independent. Writing them as separate terms means you can balance them by tuning $\lambda$, diagnose which one is misbehaving from the loss curve, and, in principle, replace either one with a different functional and retain interpretability.

The failure modes of Alignment + Uniformity are particularly clean because the loss has only one balancing knob:

• λ = 0 (alignment only). The loss is minimised by collapsing all points to a single location. Positives are perfectly aligned, but so is everything else.
• λ very high (uniformity only). Points spread evenly across the sphere with no regard for class structure. Accuracy at chance.

The cleanness is the contribution. Wang & Isola don’t claim a new SOTA; they claim that this decomposition is what InfoNCE was doing all along, and once it is written explicitly you can see why InfoNCE works (the two objectives are intrinsically separable on the sphere) and predict when it will fail (whenever one of them is dominating).

By the end of 2020 the contrastive lineage looked finished. The full pipeline, pair → triplet → InfoNCE → CLIP → SupCon → align-and-uniform, covered every modality of comparison the literature had needed. The remaining work was supposed to be parameter tuning and architectural search.

What the lineage missed was a question none of its members had asked: should the negatives target really be cosine $-1$? Two years later, the answer arrived from inside Google.

7. Questioning the target: SigLIP, 2023#

The answer was to abandon the softmax altogether. Zhai et al.’s Sigmoid Loss for Language Image Pre-Training, SigLIP, replaces CLIP’s softmax with a pairwise sigmoid:

with $y_{ij}=+1$ for matched pairs and $-1$ otherwise, $t>0$ a learned scale, and $b$ a learned bias. The structural change is twofold. Pairwise, not row-softmax: no denominator forces all candidates in a row to compete through one normalization. A movable logistic transition: $b$ and $t$ determine where a pair’s gradient changes most rapidly. For a mismatched pair the gradient becomes exponentially small when $t\,\mathrm{sim}_{ij}+b\ll0$; it does not vanish at a finite universal cosine.

The implications, for the geometric question the lineage had been ignoring:

1. SigLIP’s negative pressure has a learned transition region rather than a row-relative softmax probability.
2. The negative gradient decays smoothly as a pair moves below that region. The bias is therefore a throttle, not an equilibrium certificate.
3. Pairwise factorization removes global softmax normalization and changes distributed communication. Implementations may still evaluate many or all image–text pairs, so the pairwise compute does not disappear automatically.

The named regimes:

• Transition near orthogonality. Negative pressure becomes small as similarities move below zero, so the two-class animation can appear to settle near perpendicular directions. The exact result remains coupled to positives, scale, bias, and optimization time.
• Transition near opposition. Moving the logistic transition left keeps appreciable negative pressure active farther around the circle. This resembles a harder repulsion schedule; it is not mathematically equivalent to CLIP’s row-softmax coupling.
• Shallow target (cos = $-0.5$). A deliberately chosen intermediate stop. Nothing is special about $-0.5$; what is special is that the bias lets you pick the stopping cosine at all, which no other loss in the family exposes.

SigLIP makes the negative-pressure profile tunable through a learned bias and scale. Its reported small-batch behavior is an empirical property of that objective and training system; this 2D panel isolates one plausible contributor, the way pairwise gradients decay.

The remaining question is whether to hard-code the stop rather than learn it. The cleanest such objective, pull positives to cos $1$, push negatives to cos $0$, stop, is the next section.

8. Giving the loss a place to stop: cosine-to-zero#

Two objects need separating. Local pressure asks how the loss changes when one similarity moves while other logits are held fixed. Joint geometry asks where all shared embeddings can move together under the full objective. The first comes from a derivative; the second requires the coupled optimization problem.

Regular simplices arise in symmetric normalized class models and in particular neural-collapse limits. They are a useful reference geometry, not a universal convergence theorem for InfoNCE, CLIP, and SupCon on arbitrary data.

The dynamics differ cleanly. Hinge margins become exactly inactive once their inequalities are satisfied. Logistic negative terms decay toward zero smoothly. Softmax pressure is relative: a negative matters according to its probability within the current candidate set. Temperature rescales these profiles, while the encoder and Gram geometry couple their effects. The animations let those mechanisms be compared without assigning all of them one destination.

An explicit squared target supplies a different signal. The next objective minimizes negative-pair cosine at zero by construction. It is not a reconstruction of InfoNCE or SigLIP; it is a controlled geometric alternative whose feasibility can be read directly:

On random data with two classes in 2D the behaviour is as clean as it gets: the points land on perpendicular axes, every inter-class cosine hits zero, and the optimisation halts, because orthogonality is genuinely achievable whenever $k \le d$. Four classes in 2D cannot be, so the loss is frustrated from the start and settles at a compromise below perfect separability. Note the trade this makes against the simplex it approximates: the simplex packs $k$ classes into $k-1$ dimensions, while exact orthogonality needs a full $k$, so cos→0 gives up some dimension-efficiency for a target that never overshoots and never needs to know $k$.

The failure modes are revealing in a different way than the rest of the family:

• Too many classes for the dimension. With $k > d$ the objective is unsatisfiable, the loss compromises (eight classes in 2D settle near a uniform $\pi/4$ spacing), and accuracy plateaus below 100%. That ceiling is geometric rather than an optimisation failure, which makes cosine-to-zero double as a diagnostic that the embedding dimension is too small for the class count.
• Concentric rings. Same problem as InfoNCE. Cosine throws away the norm.

Its target is not universally better. Exact pairwise orthogonality costs one dimension per class direction, whereas a simplex fits $k$ centered classes in $k-1$ dimensions. What the objective buys is interpretability: the negative term is exactly minimized at zero cosine. SigLIP has a smooth logistic transition rather than this finite quadratic minimum; softmax objectives remain relative to the candidate set.

The race#

You have seen each loss work alone. Now watch them race. Same dataset, same initial points, same step counter, only the loss differs. Eight panels in two rows: the historical lineage on top (Hadsell → FaceNet → InfoNCE → CLIP), and the modern wave on the bottom (SupCon → SigLIP → Align+Uniform → Cosine→0).

A handful of patterns become visible only when the panels are running side by side:

Pair and triplet stop early. Once the margin is satisfied the gradient vanishes and the points freeze. This is the margin family’s signature behaviour: a clean, early termination, with whatever geometric capacity remains unused.

Softmax-based losses never stop. InfoNCE, CLIP, SupCon, and Alignment + Uniformity all have non-vanishing gradients. They keep optimising past the point of perfect classification, fine-tuning the angular spread of the negatives long after nearest-centroid accuracy hits 100%, with $\tau$ as the only throttle. CLIP and InfoNCE look near-identical in 2D, they are the same loss with a symmetrisation factor.

SupCon is the harshest collapser. Many positives means each point is pulled toward its class centroid every step. Classes become near-Dirac on the circle. Great for classification, terrible if you care about within-class variation.

SigLIP and cosine-to-zero flatten for different reasons. SigLIP’s pairwise gradient becomes exponentially small below its learned transition region. Cosine-to-zero has an exact finite minimum for each negative term at orthogonality. The animations show both curves flattening, but the mathematics behind the flattening is different.

Boundary of the result#

What about the losses that never appeared? Everything above acts on one primitive, pairwise distances or similarities, and two adjacent ideas step outside it in ways that don’t translate to 2D:

• Barlow Twins (Zbontar et al., 2021) and VICReg (Bardes et al., 2022) bypass the negative-sample problem by regularising the batch’s cross-correlation matrix toward identity. They don’t need negatives, but they need enough embedding dimensions for the diagonal and off-diagonal terms to make sense.
• DINO, MoCo, BYOL use a momentum encoder and a slow-moving teacher to generate targets, removing the need for explicit negatives entirely.

The eight losses racing here all act on the same primitive (pairwise similarities) and disagree about a single geometric question: how far should different-class pairs be? Once you watch the disagreement play out at 60 random points in 2D, with each loss’s named pathologies exposed one preset at a time, the trade-offs the literature has been arguing about for nearly two decades stop being abstract.

Eight losses can organize the same points while implementing different contracts. Margins switch off at finite inequalities. Softmax losses rank candidates through a shared denominator. Logistic terms fade smoothly. Alignment and uniformity act on aggregate geometry. Cosine-to-zero names a finite pairwise target. The useful question is not which loss chose the morally correct angle; it is which pressure profile, coupling, and feasible geometry fit the representation problem in front of you.

@misc{bouhsine2026untanglingthemoons,
  author       = {Bouhsine, Taha},
  title        = {Untangling the Moons: A Visual History of Contrastive Learning},
  year         = {2026},
  month        = {may},
  howpublished = {\url{https://tahabouhsine.com/blog/untangling-the-moons/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}