4.5 CLIP for text-to-image

Key points

1. CLIP learned a shared space for images and text. Its text encoder gives diffusion training a strong prompt signal. It stays frozen during SD training. CLIP means Contrastive Language-Image Pre-training. VAE means variational autoencoder.

   What NVIDIA says (1)

   “Stable diffusion has three main components: A U-Net, an image encoder(Variational Autoencoder, VAE) and a text-encoder(CLIP).”

   — NeMo Framework 24.09: Stable Diffusion

2. context_dim is the width of the text embeddings that the U-Net's cross-attention reads. A mismatch makes the shapes incompatible.

   What NVIDIA says (1)

   “context_dim : Must be adjusted to match the text encoder’s output dimension.”

   — NeMo Framework 24.09: Stable Diffusion

3. A Vision Transformer splits an image into patches and processes them like tokens. CLIP means Contrastive Language-Image Pre-training. ML means machine learning.

   What NVIDIA says (1)

   “CLIP’s vision model is based on the Vision Transformer (ViT) architecture.”

   — NeMo Framework 24.09: CLIP

4. CLIP embeddings are a general-purpose image representation. They feed vision-language models and data-quality tools. CLIP means Contrastive Language-Image Pre-training.

   What NVIDIA says (2)

   “NeVA harnesses the power of the pre-trained CLIP visual encoder, ViT-L/14”

   — NeMo Framework 24.09: NeVA

   “a linear classifier that takes OpenAI CLIP ViT-L/14 image embeddings as input.”

   — NeMo Curator (NeMo Framework 24.09): Aesthetic Classifier

5. Both encoders must output the same size so their embeddings can be compared. CLIP means Contrastive Language-Image Pre-training.

   What NVIDIA says (1)

   “output_dim : Represents the dimensionality of the output embeddings for both the text and vision models.”

   — NeMo Framework 24.09: CLIP

Key terms

• Vision encoder: A network that turns an image into feature vectors that other parts of a model can use.
• CLIP: A model with an image encoder and a text encoder trained together so matching images and captions land close in one vector space.
• Vision Transformer: A transformer that splits an image into patches and processes them like tokens.
• Stable Diffusion: A latent text-to-image diffusion model made of a U-Net, a VAE image encoder and a CLIP text encoder.
• Text encoder: A network that turns a prompt into embeddings that guide a generative model.

Try it

• CLIP lab: match images and captions

Sample question

When you train Stable Diffusion, what role does CLIP play?