3.4 Identifying data, hardware and software components

Key points

1. A text encoder turns a prompt into embeddings. Imagen uses a large language model encoder, typically T5, instead of CLIP. CLIP means Contrastive Language-Image Pre-training.

   What NVIDIA says (1)

   “Imagen employs a text encoder, typically T5, to encode textual features.”

   — NeMo Framework 24.09: Imagen

2. A latent space is a compressed representation of the data. Diffusing in that smaller space is cheaper than in pixel space. VAE means variational autoencoder.

   What NVIDIA says (2)

   “an input image is condensed from 512x512x3 dimensions to 64x64x4.”

   — NeMo Framework 24.09: Stable Diffusion

   “This compression results in decreased memory and computational requirements when compared to pixel-space diffusion models.”

   — NeMo Framework 24.09: Stable Diffusion

3. Automatic speech recognition (ASR) turns audio into text. Text-to-speech (TTS) turns text into audio. Riva provides both on GPUs. DALI means the NVIDIA Data Loading Library. SDK means software development kit.

   What NVIDIA says (2)

   “Automatic Speech Recognition (ASR) takes an audio stream or audio buffer as input and returns one or more text transcripts”

   — NVIDIA Riva: ASR Overview

   “based on a two-stage pipeline.”

   — NVIDIA Riva: TTS Overview

4. FP16 is half-precision floating point. Half the bits means roughly half the memory for the same values. FP16 means 16-bit floating point.

   What NVIDIA says (1)

   “Half-precision floating point format (FP16) uses 16 bits, compared to 32 bits for single precision (FP32). Lowering the required memory enables training of larger models or training with larger mini-batches.”

   — Train With Mixed Precision

Key terms

• Variational autoencoder: An encoder-decoder network that compresses images into a smaller latent space and rebuilds them.
• Latent space: A compressed representation of data in which a model can work more cheaply than on raw pixels.
• Text encoder: A network that turns a prompt into embeddings that guide a generative model.
• Imagen: A cascaded text-to-image diffusion model that generates at 64x64 and then upsamples to 256x256 and 1024x1024.
• Mixed precision: Training mostly in 16-bit formats while keeping key values in 32-bit, to save memory and time.
• NVIDIA Riva: An NVIDIA SDK for GPU-accelerated speech recognition and speech synthesis.
• Automatic speech recognition: Turning audio into text.
• Text-to-speech: Turning text into spoken audio.
• T5: A transformer text encoder that Imagen typically uses for prompts.

Sample question

Which text encoder does Imagen typically use?