2.5 Monitoring data, experiments and processes

Key points

1. MLOps (machine learning operations) is a set of best practices for running AI successfully. It is modeled on DevOps and adds data scientists to the team.

   What NVIDIA says (3)

   “A shorthand for machine learning operations, MLOps is a set of best practices for businesses to run AI successfully.”

   — What is MLOps?

   “MLOps is modeled on the existing discipline of DevOps”

   — What is MLOps?

   “MLOps adds to the team the data scientists, who curate datasets and build AI models that analyze them.”

   — What is MLOps?

2. NVIDIA notes that datasets are massive and can change in real time. Models need careful tracking through cycles of experiments, tuning and retraining.

   What NVIDIA says (2)

   “AI models require careful tracking through cycles of experiments, tuning and retraining.”

   — What is MLOps?

   “Datasets are massive and growing, and can change in real time.”

   — What is MLOps?

3. NVIDIA defines an AI data flywheel as a self-improving loop. Data from AI interactions refines the models, which produces better outcomes and more valuable data.

   What NVIDIA says (2)

   “An AI data flywheel is a self-improving loop where data collected from AI interactions or processes is used to continuously refine AI models”

   — Data flywheel: What it is and how it works

   “generating better outcomes and more valuable data for continued improvement.”

   — Data flywheel: What it is and how it works

4. A foundation model only knows its pretraining and fine-tuning data, which becomes outdated over time. NVIDIA says retrieval-augmented generation (RAG) keeps the model grounded with external knowledge at query time.

   What NVIDIA says (2)

   “The knowledge of a foundation model is limited to the pretraining and fine-tuning data, becoming outdated over time”

   — Mastering LLM Techniques: LLMOps

   “workflow is used to maintain freshness and keep the model grounded with external knowledge during query time.”

   — Mastering LLM Techniques: LLMOps

Key terms

• MLOps: Best practices for running AI in production, modeled on DevOps.
• AI data flywheel: A loop where data from AI use improves the models, which produce better data.

Sample question

What is MLOps?