2.6 Multimodal transfer learning

Key points

1. Transfer learning reuses a pretrained model for a new task. Freezing keeps pretrained parts fixed while new parts, such as a projection layer, learn.

   What NVIDIA says (1)

   “freeze : If set to True , the model parameters will not be updated during training.”

   — NeMo Framework 24.09: NeVA

2. Transfer learning applies a network trained on one task to another domain. It helps most when new data is scarce.

   What NVIDIA says (2)

   “This deep learning technique enables developers to harness a neural network used for one task and apply it to another domain.”

   — What Is Transfer Learning?

   “Transfer learning is useful when you have insufficient data for a new domain”

   — What Is Transfer Learning?

3. TAO is NVIDIA's toolkit for adapting pretrained models. ONNX (Open Neural Network Exchange) is a standard model file format.

   What NVIDIA says (2)

   “You can select from 100+ pre-trained vision AI models on NGC and fine-tune them on your own dataset”

   — NVIDIA TAO Toolkit: Overview

   “TAO outputs trained models in ONNX format”

   — NVIDIA TAO Toolkit: Overview

Key terms

• DreamBooth: A fine-tuning method that teaches a text-to-image model a specific subject from a few images, bound to a unique identifier.
• Transfer learning: Reusing a model trained on one task as the start for another task.
• Freezing: Keeping some model weights fixed during training so only other parts learn.
• NVIDIA TAO: An NVIDIA toolkit for fine-tuning pretrained vision models on your data and exporting them to ONNX.

Sample question

In a NeMo multimodal config, what does freeze: True do for a component?