1.4 Testing multimodal data quality

Key points

1. NeMo Curator is NVIDIA's data curation library. Its aesthetic classifier estimates the subjective quality of an image. A higher score means a more pleasing image. PII means personal identifiable information. NSFW means not safe for work.

   What NVIDIA says (2)

   “Aesthetic classifiers can be used to assess the subjective quality of an image.”

   — NeMo Curator (NeMo Framework 24.09): Aesthetic Classifier

   “outputs a score from 0-10 where 10 is aesthetically pleasing.”

   — NeMo Curator (NeMo Framework 24.09): Aesthetic Classifier

2. An embedding is a vector that represents an input. The aesthetic classifier is a small linear model on top of CLIP image embeddings. CLIP means Contrastive Language-Image Pre-training. PII means personal identifiable information.

   What NVIDIA says (1)

   “a linear classifier that takes OpenAI CLIP ViT-L/14 image embeddings as input.”

   — NeMo Curator (NeMo Framework 24.09): Aesthetic Classifier

3. NSFW means not safe for work. Removing unsafe content is a common step in generative AI data pipelines. PII means personal identifiable information.

   What NVIDIA says (2)

   “a value between 0 and 1 where 1 means the content is NSFW.”

   — NeMo Curator (NeMo Framework 24.09): NSFW Classifier

   “Removing unsafe content is common in most data processing pipelines to prevent your generative AI model from learning to produce unsafe material.”

   — NeMo Curator (NeMo Framework 24.09): NSFW Classifier

4. Semantic duplicates are pairs that mean almost the same thing without being identical. Semantic deduplication embeds each sample, clusters the embeddings with k-means, and compares pairs by cosine similarity.

   What NVIDIA says (2)

   “uses embeddings to identify and remove “semantic duplicates” - data pairs that are semantically similar but not exactly identical.”

   — NeMo Curator (NeMo Framework 24.09): Semantic Deduplication

   “The embeddings are clustered into k clusters using k-means clustering.”

   — NeMo Curator (NeMo Framework 24.09): Semantic Deduplication

5. Cosine similarity measures how closely two vectors point the same way. Pairs above the chosen threshold are treated as semantic duplicates.

   What NVIDIA says (1)

   “Data pairs with cosine similarity above a threshold are considered semantic duplicates.”

   — NeMo Curator (NeMo Framework 24.09): Semantic Deduplication

6. A heuristic is a simple rule of thumb. Heuristic filters compute simple statistics and drop documents that fail them. This also applies to the captions in image-text data.

   What NVIDIA says (1)

   “There are heuristics that measure quality by gathering simple statistics like how many punctutation marks a document has, how long is the document, and how repetitive is the document.”

   — NeMo Curator (NeMo Framework 24.09): Classifier and Heuristic Quality Filtering

7. PII means personal identifiable information, such as names, emails and phone numbers. Curator's tool finds and removes it at scale with Dask.

   What NVIDIA says (1)

   “The purpose of the personal identifiable information (PII) de-identification tool is to help scrub sensitive data out of datasets.”

   — NeMo Curator (NeMo Framework 24.09): PII Identification and Removal

8. An image embedding is a vector that summarizes an image. Quality scoring and duplicate checks then run on these vectors, which is far cheaper than on pixels.

   What NVIDIA says (1)

   “Image embeddings are the backbone to many data curation operations in NeMo Curator.”

   — NeMo Curator (NeMo Framework 24.09): Image Curation

Key terms

• NeMo Curator: NVIDIA's data curation library with classifiers, deduplication, quality filters and PII removal.
• Aesthetic classifier: A NeMo Curator model that scores image quality from 0 to 10 from CLIP image embeddings.
• NSFW classifier: A NeMo Curator model that scores how likely an image is unsafe, from 0 to 1.
• Semantic deduplication: Removing data pairs that mean nearly the same thing, found by clustering embeddings and comparing cosine similarity.
• Personal identifiable information: Data that can identify a person, such as a name, email or phone number.
• Embedding: A list of numbers that represents the meaning of an input such as text or an image.

Try it

• Deduplication lab: exact vs near-duplicate

Sample question

You want to drop low-quality images from a text-to-image training set. Which NeMo Curator classifier scores image quality from 0 to 10?