NVIDIA NeMo Curator for Developers

Text Data Processing

This architecture diagram shows the various features available for processing text. At a high level, a typical text processing pipeline begins with downloading data from public sources or private repositories and performing cleaning steps, such as fixing Unicode characters. Next, heuristic filters—such as word count—are applied, followed by deduplication, advanced quality filtering using classifier models for quality and domain, and finally, data blending.

Synthetic Data Generation

NeMo Curator has a simple, easy-to-use set of tools that let you use pre-built synthetic data generation pipelines or build your own. Any model inference service that uses the OpenAI API is compatible with the synthetic data generation module, allowing you to generate your data from any model.

NeMo Curator provides pre-built pipelines for several use cases to help you get started easily, including evaluating and customizing embedding models, prompt generation (open Q&A, closed Q&A, writing, math/coding), synthetic two-turn prompt generation, dialogue generation, and entity classification.

Video Data Processing

This architecture diagram illustrates the various features available through the early access program for processing high-quality videos.

A typical pipeline has the following steps

• Video decoding and splitting: Decode long videos and split them into semantically shorter clips.

• Transcoding: Convert all the short videos to a consistent format.

• Captioning: Caption videos using domain-specific state-of-the-art vision language models (VLMs) to describe the clips in detail.

• Text embedding: Create embeddings of text captions for downstream semantic search and deduplication.

Audio Data Processing

This architecture diagram shows the various features available for processing audio.

A typical pipeline has the following steps

• Data download and extraction: Fetch audio files from cloud, internet, or local disk sources.
  

• Speech-to-text inference: Transcribe audio with a NeMo ASR model, using GPU acceleration for speed.

• Metric calculation (WER): Compute Word Error Rate to assess transcription accuracy.

• Get audio duration: Extract duration metadata for each file.

• Feature-based filtering: Filter samples by WER and duration thresholds.

• Metadata conversion: Transform curated outputs to document format and export as JSONL.

Image Data Processing

This architecture diagram shows the various features available for processing images.

A typical pipeline begins with downloading the dataset in a WebDataset format, followed by creating CLIP embeddings. Next, the images are filtered for high quality using the NSFW and Aesthetic filters. Duplicate images are then removed using semantic deduplication, and finally, a high-quality dataset is created.