Modality Gap

The modality gap in CLIP-style models: separated cones for image and text, why it appears, and whether to separate or represent.

1 post tagged #modality-gap.

Feb 23, 2026

Not All Infinities Are Equal: The Cross-Entropy Asymmetry Behind Hallucination

The singularity structure of cross-entropy is asymmetric — and that asymmetry explains LLM hallucination, the CLIP modality gap, and why contrastive losses need 32K batches.