NVIDIA Cosmos for Developers
NVIDIA Cosmos™ is a platform purpose-built for physical AI, featuring state-of-the-art generative world foundation models (WFMs), guardrails, and an accelerated data processing and curation pipeline for autonomous vehicle (AV), robotics, and AI agent developers.
Build, evaluate, deploy, and simulate physical AI models faster while minimizing testing and validation risks in the real world.
How It Works
Cosmos WFMs accelerate physical AI development, helping developers augment datasets and post-train downstream world models for robots and autonomous vehicles.
Cosmos Predict generates next frames based on input to build datasets predicting various edge cases and serves as the foundation for all world models.
Cosmos Reason acts as a critic, using chain-of-thought reasoning to evaluate synthetic visuals and reward outcomes. It can also generate captions to speed up data curation.
Cosmos Transfer amplifies structured video across various environments and lighting conditions.
Developers can use the available PyTorch inference and post-training scripts along with model checkpoints. Cosmos NIM microservices are in development—Cosmos Predict NIM microservices are available here.
NVIDIA Cosmos World Foundation Models
A family of pretrained models for world generation as videos and world understanding for accelerating physical AI development. Available openly to developers on NGC, Hugging Face, and GitHub.
Cosmos Predict
For future world state generation or as a base for custom world models.
Input: Text or Image
Output: Video
Cosmos Transfer
Multicontrol net for fast, photorealistic video data augmentation.
Input: Segmentation maps, depth signals, HD maps, or CG simulation videos.
Pair with: NVIDIA Omniverse
Output: Photorealistic world scenes
Cosmos Reason
World reasoning for synthetic data curation, robot decision-making, and runtime video analytics for AI agents.
Input: Image or Video
Output: Chain-of-thoughts reasoning and text
Cosmos Curator
Filter, annotate, and deduplicate large datasets for physical AI development using advanced AI models and distributed computing.
Cosmos Guardrails
A set of guardrails, including a pre-guard to block harmful inputs and a post-guard to ensure safety and consistency in generations.
Cosmos Prompt Upsampler
Transform original input prompts into more detailed and enriched versions for higher-quality outputs from Cosmos WFMs.
Cosmos Dataset Search
Instantly search and retrieve scenarios from massive training datasets for targeted post-training using the Cosmos Dataset Search vector-based search workflow.
Introductory Resources
Cosmos Cookbook: A Practical Guide to Physical AI Models
The Cosmos Cookbook is an open-source guide with step-by-step workflows and examples for deploying and customizing NVIDIA Cosmos world foundation models in real-world applications.
Updated Versions of Cosmos Predict 2.5 & Transfer 2.5
Cosmos Predict 2.5 and Transfer 2.5 advance NVIDIA’s world foundation models by unifying multimodal world generation and improving spatially controlled world transformation, driving scalable, efficient physical AI.
World Simulation With Video Foundation Models
This NVIDIA Cosmos white paper presents an open platform of world foundation models, video curation tools, and tokenizers to help developers efficiently customize, generate, and simulate physics-based data for advancing physical AI applications like robotics and autonomous vehicles.
Starter Kits
Start solving physical AI challenges by developing custom world models with Cosmos or using Cosmos WFMs for downstream use cases. Explore implementation scripts, explainer blogs, and more how-to documentation for various stages of physical AI development.
Post-Training Cosmos WFMs
Cosmos WFMs are purpose-built for post-training. Use domain-specific datasets to build world models, or post-train for different types of output, such as action generation for policy models.
Synthetic Data Generation
Build and deploy world models for infinite domain-specific synthetic data. Use NVIDIA Omniverse for physics-based conditioning.
Vision-Language Models
Vision language models (VLMs) are multimodal, generative AI models that can understand and process video, images, and text.
Build a Video Analytics AI Agent With Cosmos Reason
Build a video analytics AI agent using NVIDIA Cosmos Reason with NVIDIA Blueprint for video search and summarization (VSS).
Cosmos Learning Library
More Resources
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility, and we have established policies and practices to enable development for a wide array of AI applications. When downloading or using this model in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
NVIDIA has collaborated with Google Deepmind to watermark generated videos from the NVIDIA API catalog.
For more detailed information on ethical considerations for this model, please see the System Card, Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI concerns here.
Get Started With NVIDIA Cosmos Today