NVIDIA NeMo Curator for Developers

Text Data Curation

This architecture diagram shows the various features available for processing text. At a high level, a typical text processing pipeline begins with downloading data from public sources or private repositories and performing cleaning steps, such as fixing Unicode characters. Next, heuristic filters—such as word count—are applied, followed by deduplication, advanced quality filtering using classifier models for quality and domain, and finally, data blending.

Synthetic Data Generation

NeMo Curator has a simple, easy-to-use set of tools that let you use pre-built synthetic data generation pipelines or build your own. Any model inference service that uses the OpenAI API is compatible with the synthetic data generation module, allowing you to generate your data from any model.

NeMo Curator provides pre-built pipelines for several use cases to help you get started easily, including prompt generation (open Q&A, closed Q&A, writing, math/coding), synthetic two-turn prompt generation, dialogue generation, and entity classification.

Video Data Processing

This architecture diagram illustrates the various features available through the early access program for processing high-quality videos. Through the Cosmos platform, filtered videos can be used to train or fine-tune WFMs.

A typical pipeline has the following steps

• Video decoding and splitting: Decode long videos and split them into semantically shorter clips.

• Transcoding: Convert all the short videos to a consistent format.

• Captioning: Caption videos using domain-specific state-of-the-art vision language models (VLMs) to describe the clips in detail.

• Text embedding: Create embeddings of text captions for downstream semantic search and deduplication.

Image Data Processing

This architecture diagram shows the various features available for processing images.

A typical pipeline begins with downloading the dataset in a WebDataset format, followed by creating CLIP embeddings. Next, the images are filtered for high quality using the NSFW and Aesthetic filters. Duplicate images are then removed using semantic deduplication, and finally, a high-quality dataset is created.