On this page
BLIP-2 lets a frozen language model describe images and answer questions on them. A frozen image encoder reads the picture, a small trained Q-Former compresses it into 32 vectors, and these precede the language model's text. The Q-Former trains in two pretraining stages on image-caption pairs, then runs once per image at inference, before the language model writes.
Cross-attention takes queries from one stream and keys and values from another; learned queries are vectors trained like weights, and a mask blocks chosen query and key pairs.
BLIP (Li et al. 2022) trains one multimodal mixture of encoder-decoder (MED) on image-caption pairs. A ViT (vision transformer, which cuts an image into patch tokens) reads the image, and one text Transformer, started from BERT-base (a 12-layer text encoder), runs in three modes.
The unimodal encoder reads the caption alone and trains with the image-text contrastive loss (ITC), which pulls a matched image vector and caption vector together and pushes the batch's other captions apart, as in Image-Text Contrastive Learning (CLIP) (Radford et al., 2021). The image-grounded encoder also reads the image and trains with the image-text matching loss (ITM), a head that says matched or not. The image-grounded decoder writes the caption under causal self-attention and trains with the language modeling loss (LM).
BLIP's three passes over one image and caption, with one self-attention and one cross-attention layer standing in for the text Transformer, plus the decoder's own causal self-attention.
import torch
import torch.nn as nn
D, N_PATCH, N_TXT, H = 768, 197, 12, 12
img = torch.randn(1, N_PATCH, D) # ViT patch features
txt = torch.randn(1, N_TXT, D) # caption token vectors
self_attn = nn.MultiheadAttention(D, H, batch_first=True)
causal_self_attn = nn.MultiheadAttention(D, H, batch_first=True)
cross = nn.MultiheadAttention(D, H, batch_first=True)
causal = torch.ones(N_TXT, N_TXT, dtype=torch.bool).triu(1)
itc, _ = self_attn(txt, txt, txt) # ITC, text only
itm, _ = cross(itc, img, img) # ITM, reads img
y, _ = causal_self_attn(txt, txt, txt, attn_mask=causal)
lm, _ = cross(y, img, img) # LM, causal, reads imgWeb captions (alt-text) often miss the picture. CapFilt (captioning and filtering) cleans them with two copies of the MED, each fine-tuned on COCO's human-written captions.
The captioner, the decoder fine-tuned with LM, writes one caption per web image by nucleus sampling (each word drawn from the likeliest words covering 90% of the probability; a higher share allows rarer words and more varied captions, a lower one stays with safe common words); the filter, the encoder fine-tuned with ITC and ITM, drops captions its ITM head calls unmatched. The gain is measured by text recall@1, how often an image's top-ranked caption is the right one.
BLIP-2 (Li et al. 2023) keeps the image encoder and the large language model (LLM) frozen, to save compute and avoid catastrophic forgetting (new training wiping out old knowledge), and trains a small module between them in two stages. Stage 1 pulls out the image features most relevant to the text with BLIP's three objectives, the captioning loss renamed ITG (image-grounded text generation).
The Q-Former (Querying Transformer) has 32 learned queries of size 768 and two submodules, an image transformer and a text transformer, that share only self-attention.
It starts from BERT-base weights, with cross-attention random, and has 188M parameters, queries included.
Its self-attention mask is one of the three masks of Section 5.
One Q-Former block in PyTorch, LayerNorms left out; queries and text go in and come out the same shape.
import torch
import torch.nn as nn
N_Q, D, D_IMG, D_FF, H = 32, 768, 1024, 3072, 12
MHA = nn.MultiheadAttention
self_attn = MHA(D, H, batch_first=True)
cross = MHA(D, H, kdim=D_IMG, vdim=D_IMG, batch_first=True)
def ffn():
up, down = nn.Linear(D, D_FF), nn.Linear(D_FF, D)
return nn.Sequential(up, nn.GELU(), down)
ffn_q, ffn_t = ffn(), ffn() # image and text transformers
def block(x, img, allowed): # x (B, N_Q + T, D)
mask = ~allowed # True = blocked
x = x + self_attn(x, x, x, attn_mask=mask)[0]
q, t = x[:, :N_Q], x[:, N_Q:]
q = q + cross(q, img, img)[0] # only queries read img
q, t = q + ffn_q(q), t + ffn_t(t)
return torch.cat([q, t], dim=1)The output Z is smaller than the frozen features, so the queries keep only what the text needs.
In Figure 6 a region’s patches share its key, all one length. A patch’s weight is the softmax of query times key, and the output is the weighted mix (Cross-Attention, Figure 3). Keys pointing the query’s way get the most weight; a longer query picks more sharply, and near the centre all get about the same. Drag query 1 to try the directions training could give it; training fixes one.
Figure 6. 1 Direction picks the patches; 2 any size, 32 vectors.
Stage 1 trains all three objectives at once, each with its own mask over queries and text, so the queries cannot copy the caption and must carry what it needs. Contrastive (ITC) keeps queries and text apart, so each side builds its own summary to compare. Matching (ITM) lets everything see everything, and a head says matched or not. Generation (ITG) lets text see the queries and earlier text, so the queries must carry what the caption needs.
In the contrastive score, t is the output of the text's first token, a summary token.
The generation mask for 32 queries and 6 text tokens.
import torch
N_Q, N_TXT = 32, 6
pos = torch.arange(N_Q + N_TXT)
is_query = pos < N_Q
row_q, col_q = is_query[:, None], is_query[None, :]
past = pos[None, :] <= pos[:, None] # column <= row
text_sees = col_q | past
allowed = (row_q & col_q) | (~row_q & text_sees)
print(allowed.shape) # torch.Size([38, 38])Figure 7. Rows look and columns are seen; blue cells are allowed.
In Stage 2, one fully connected (FC) layer projects Z's 32 vectors to the LLM's embedding size, and they precede the text as soft visual prompts (vectors that act like extra words); only the Q-Former and the FC layer train.
OPT (decoder-only) trains with the next-token loss L; FlanT5 (encoder-decoder) trains with a prefix loss, where the caption's first part goes into the encoder with the prompts and the decoder writes the rest.
Stage 2 with OPT-125M as the frozen LLM and random Z as the Q-Former output.
import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM
NAME, IGNORE = "facebook/opt-125m", -100
llm = AutoModelForCausalLM.from_pretrained(NAME)
llm.requires_grad_(False) # frozen, gradients still pass
proj = nn.Linear(768, llm.config.hidden_size) # FC layer
embed = llm.get_input_embeddings()
Z = torch.randn(2, 32, 768) # Q-Former output (B, N_Q, 768)
ids = torch.randint(100, 2000, (2, 8)) # caption token ids
B, N_Q = Z.shape[:2]
x = torch.cat([proj(Z), embed(ids)], dim=1)
skip = torch.full((B, N_Q), IGNORE)
labels = torch.cat([skip, ids], dim=1)
L = llm(inputs_embeds=x, labels=labels).loss # mean
L.backward() # reaches proj and, upstream, the Q-FormerFigure 9. Dashed red arrows are gradients; bars compare trainable and frozen parameters.
The test is zero-shot visual question answering (VQA), with no training on the test sets, against Flamingo80B, an earlier 80B image-and-language model.
| Part | What BLIP-2 uses |
|---|---|
| Data | BLIP's 129M images. BLIP-large writes captions for each. CLIP's ViT-L/14 ranks them with the web caption, and the top two stay. |
| Image encoders | ViT-L/14 from CLIP and ViT-g/14 from EVA-CLIP (Fang et al. 2023), frozen, second-to-last layer features |
| LLMs | OPT and FlanT5, several sizes, frozen |
| Steps | 250k in Stage 1, 80k in Stage 2 |
| Hardware | 16 A100 GPUs; under 6 days for Stage 1, under 3 for Stage 2, largest size |
The paper credits Flamingo80B's OK-VQA lead to its 70B language model. It counts 188M trainable parameters in Table 1 and 108M in Table 2.
On the released OPT-2.7B checkpoint, an image and a question go in and an answer comes out.
import torch
from PIL import Image
from transformers import Blip2Processor
from transformers import Blip2ForConditionalGeneration
CKPT = "Salesforce/blip2-opt-2.7b"
processor = Blip2Processor.from_pretrained(CKPT)
model = Blip2ForConditionalGeneration.from_pretrained(CKPT, dtype=torch.float16)
model = model.to("cuda")
print(model.config.num_query_tokens) # 32
image = Image.open("photo.jpg").convert("RGB")
question = "Question: what is in the picture? Answer:"
inputs = processor(image, question, return_tensors="pt")
inputs = inputs.to("cuda", torch.float16)
ids = model.generate(**inputs, max_new_tokens=20)
print(processor.batch_decode(ids, skip_special_tokens=True))| CapFilt | Once, before pretraining, over the web images. |
|---|---|
| Stage 1 | Pretraining only; image-caption pairs, three losses every step, image encoder frozen. |
| Stage 2 | Pretraining only; the frozen LLM's next-token or prefix loss, gradients passing through it. |
| Inference | One Q-Former pass per image; its 32 vectors precede the text for every word generated. |
| Queries | Rows of Q and of Z. More pass more of the image as more LLM tokens, fewer compress the image more; 1 is one vector; 257 matches the ViT's feature count at 224 px, so the count no longer shrinks, though each query still reads a weighted mix of all features. Default 32. |
| Code | LAVIS BLIP-2 with both stage scripts, Hugging Face BLIP-2. |