CLIP matched images to captions instead of fixed labels

A network trained to match captions to images

On 5 January 2021, OpenAI introduced CLIP (Contrastive Language-Image Pre-training) alongside code and a trained model, ahead of the paper posted to arXiv on 26 February 2021. The method trains on 400 million image-caption pairs collected from the internet, learning to match each image to its correct caption out of a batch rather than to a fixed list of category names. Because the categories a user wants are supplied as text at inference time, the same network can be pointed at a new classification task without retraining, a property the announcement calls zero-shot transfer, borrowing the term from GPT-2 and GPT-3.

What the released model demonstrated

The headline comparison in the paper is that zero-shot CLIP matches the accuracy of a fully supervised ResNet-50 on ImageNet without using any of that dataset's 1.28 million labelled training images. The released repository reproduces this pairing of an image encoder and a text encoder trained jointly, with an MIT-licensed model usable through a short Python snippet. The same evaluation approach surfaced a weakness: on handwritten digits from the MNIST dataset, zero-shot CLIP reaches only 88 percent accuracy against a human baseline the announcement puts near 99.75 percent, because photographs of typed or written digits are rare among the internet image-text pairs CLIP was trained on. The announcement also states that CLIP's outputs are sensitive to how a category is phrased, sometimes requiring trial-and-error prompt engineering to get a good result.

What the bias findings limit, not just what they show

OpenAI's own broader-impacts analysis found that letting a user define categories at inference time carries risk: adding race labels from the FairFace dataset alongside terms such as 'criminal' and 'animal' caused CLIP to place people aged 0 to 20 into the egregious category around 32.3 percent of the time, a rate that fell to about 8.7 percent once 'child' was offered as a competing label. On identifying public figures from photographs, CLIP reached 59.2 percent top-1 accuracy when choosing among 100 candidates and 43.3 percent among 1,000, results OpenAI calls notable for a task-agnostic model but not competitive with dedicated production systems. None of this shows that CLIP is safe or unsafe in a given deployment; it shows that classifier behaviour depends on the label set a developer chooses, and that removing task-specific training data does not remove the need to test for bias.

Questions to carry into your own evaluation

• What label set will the deployment actually expose to users, and has it been tested with the same care as the benchmark categories?
• Does the task resemble internet image-caption pairs, or a domain such as scanned documents where CLIP's training distribution is thin?
• Is a reported accuracy figure zero-shot, or does it depend on prompt phrasing that was tuned after seeing results?

CLIP's contribution was a training method and a public model, not a verdict on any single downstream use. Its own release materials are unusually explicit that performance and bias both hinge on choices made after training, which is where most deployment risk still lives.