← All field notes

Research Watch · Model architecture · Multimodal

Multimodal grounding: connecting pixels to the token stream

CLIP and LLaVA describe the two layers most production vision-language systems are built from — a shared image-text embedding space, and a thin projection that lets a language model attend over visual features as if they were tokens.

AI-assisted / research-based

This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.

Language models read tokens. Giving one the ability to usefully "see" an image requires connecting visual information to that same token stream — and two widely cited papers describe the two layers most production systems build that connection from.

The language model in a vision-language system never sees a pixel. It sees a vector that already learned what the pixel meant.

A shared space, built without labels

CLIP ↗, by Radford and colleagues at OpenAI, trains an image encoder and a text encoder jointly on 400 million image-caption pairs collected from the web. The training task is deliberately simple: given a batch of images and captions, predict which caption actually goes with which image. There is no fixed category list anywhere in this objective — matching pairs are pulled together in a shared embedding space, mismatched pairs are pushed apart, and the categories the model ends up representing emerge from whatever the captions described.

Architecture / contrastive pretraining

No labels — just which caption fits which image

FIG 01
CLIP contrastive image-text pretraining An image encoder and a text encoder independently produce embeddings for a batch of image-caption pairs, and training pulls matching pairs together in the shared embedding space while pushing mismatched pairs apart. IMAGEImage encoder400M pairs, web-scale TEXTText encoderraw captions, no labels SHARED SPACEMatching pairs pulled closemismatched pairs pushed apartpredict which caption fits which image ZERO-SHOT TRANSFERNew concepts via textno task-specific labelsmatches supervised ResNet-50 TRAINING SIGNALthe pretraining task is only ever "which caption goes with which image"
CLIP never sees a fixed category list during pretraining. The contrastive matching task alone produces a shared embedding space that later supports zero-shot transfer to new visual concepts.

The payoff is zero-shot transfer: because natural language can reference any visual concept, a CLIP model can be pointed at a new task using only a text description of it, with no task-specific labels. The paper reports this matching the accuracy of a fully supervised ResNet-50 on ImageNet zero-shot — without using any of the 1.28 million labeled training examples that supervised model required — and demonstrates non-trivial transfer across more than 30 benchmark datasets spanning OCR, action recognition, and fine-grained classification.

A projection layer is the entire bridge

Visual Instruction Tuning ↗, by Liu, Li, Wu, and Lee, introduces LLaVA and answers a different question: given a strong visual encoder like CLIP's, what is the smallest addition needed to give a language model genuine visual instruction-following ability? Their answer is a single trainable projection layer that maps the visual encoder's output features into the same vector space as the language model's token embeddings — no architectural surgery on either the vision encoder or the language model itself.

Architecture / multimodal instruction tuning

A projection layer is the entire bridge

FIG 02 - MOTION
LLaVA vision-language architecture An image passes through a frozen vision encoder, a trainable projection layer maps its features into the language model's token embedding space, and the language model attends over image and text tokens together to produce an instruction-following response. FROZENVision encoderimage → visual features TRAINABLEProjectionimage → token space MERGEImage + text tokensone shared sequence LANGUAGE MODELInstruction-tunedon GPT-4-generated data KEY DETAILthe language model never sees raw pixels — only projected feature vectors alongside ordinary text tokens
A frozen vision encoder and a small trainable projection are enough to give a language model visual grounding — the heavy lifting is instruction-tuning on image-grounded data, not a new architecture.

Training data mattered as much as architecture here: the paper describes the first use of a language-only GPT-4 to generate multimodal instruction-following data — image-grounded instructions and responses — which is what the projected features are instruction-tuned against. The paper reports LLaVA reaching an 85.1% relative score against GPT-4 on a synthetic multimodal instruction benchmark, and a new state of the art of 92.53% accuracy when fine-tuned on Science QA.

What grounding does and does not guarantee

A model that connects visual features to language fluently is not the same as a model that reads visual detail precisely. Both papers' own evaluation setups hint at this distinction: CLIP's zero-shot transfer is measured on concept-level classification, not pixel-level precision; LLaVA's instruction data was itself generated by a language-only model reasoning about image descriptions, not verified against the actual pixels by a human at scale. In production, this gap shows up specifically on small text, exact counts, fine spatial relationships, and unusual chart or diagram layouts — exactly the content where a wrong answer is delivered with the same fluent confidence as a right one.

What this changes in production

Document and screenshot understanding

Extracting structured fields from scanned forms, invoices, or UI screenshots becomes a single-pass vision-language task rather than a separate OCR-plus-parsing pipeline — for content large and clear enough for the visual encoder's effective resolution, which is usually lower than a dedicated OCR system's.

Visual quality and content review

Describing or classifying what is visible in an image — for inspection, moderation, or accessibility workflows — benefits directly from a shared embedding space that lets a team query by natural language description rather than a fixed label taxonomy defined in advance.

Accessibility and description generation

Generating descriptive alt text for images embedded in longer documents or interfaces is a natural fit for the LLaVA-style architecture, since the output is exactly the kind of fluent natural-language description these models were instruction-tuned to produce.

A production checklist

Before shipping a vision-language feature to production, the team should be able to answer:

  • Was accuracy measured at the actual image resolution and content type production will see, not a benchmark's curated images?
  • Has the system been tested specifically on small text, exact counts, and spatial relationships — the documented weak points of this architecture class?
  • If instruction-tuning data was generated by another model rather than verified by humans against real images, what is the error rate on cases that data never covered?
  • Does the application distinguish "the model described something plausible" from "the model verified what is actually in the image" in how results are presented to users?
  • Is there a fallback or human review path for high-stakes visual judgments — safety-critical inspection, medical, or legal contexts — rather than trusting model output alone?
  • What is the latency and cost impact of the added visual encoder and projection step on the existing serving path?

Research referenced

Continue readingReturn to field notes →