3.7 Writing software components and scripts

Key points

1. OpenAI-compatible means existing OpenAI client code can call it by changing the base URL. Chat completions take a message history and return the model's reply. VLM means vision language model. NIM means NVIDIA Inference Microservice.

   What NVIDIA says (2)

   “NIM VLM exposes an OpenAI-compatible inference API backed by vLLM”

   — NVIDIA NIM for Vision Language Models: API Reference

   “POST /v1/chat/completions Multi-turn chat completions with message history.”

   — NVIDIA NIM for Vision Language Models: API Reference

2. A model repository is a folder with each model and its configuration. Triton loads models from it at startup.

   What NVIDIA says (1)

   “tritonserver --model-repository=/models”

   — Quickstart — NVIDIA Triton Inference Server

Key terms

• Triton Inference Server: NVIDIA's open-source server for serving trained models.
• NVIDIA NIM: Containerized NVIDIA inference microservices; NIM for VLMs exposes an OpenAI-compatible API.

Sample question

You write a script to send an image question to a NIM VLM. Which API style does it expose?