Autonomous vehicle (AV) stacks are evolving from a hierarchy of discrete building blocks to end-to-end architectures built on foundation models. This transition demands an AV data flywheel to generate synthetic data and augment sensor datasets, address coverage gaps and, and ultimately, build a validation toolchain to safely develop and deploy autonomous vehicles.
In this blog post, we highlight the latest NVIDIA APIs, NVIDIA Cosmos world foundation models (WFMs), and NIM microservices for developers to kickstart data pipelines.
Neural reconstruction for AV simulation
Real world data collected from AV fleets serves as the foundation for AV workflows, however, it isn’t feasible to collect and annotate sensor data for rare events, objects, and scenarios. Through advanced 3D reconstruction techniques, neural reconstruction and rendering, developers can turn real world datasets into diverse, interactive simulations.
NVIDIA NuRec
NVIDIA NuRec is a set of APIs and tools for neural reconstruction and rendering. It enables developers to use their existing fleet data to reconstruct high-fidelity digital twins, simulate new events, and render sensor datasets from novel points of view. NuRec’s APIs and tools enable developers to:
- Prepare and process sensor data for reconstruction
- Reconstruct sensor data into 3D representations
- Perform Gaussian-based rendering to connect with simulation
Sensor configurations vary between vehicle platforms. Before reconstructing digital twins from arbitrary sensor data with different calibration, extrinsics, and capture quality, the sensor data must be formatted in a standardized way for data processing.
Voxel51 is a visual AI data platform company that has built powerful, widely used open source tools for data processing, visualization and formatting for AI workloads. NuRec data toolkits, data ingestion libraries as well as the NuRec container will be available on Voxel51’s toolchain, so developers can ingest their own datasets, evaluate the quality of their reconstructions, and create 3D digital twins for downstream simulation tasks. This pipeline will be featured at CVPR in a demo at the Voxel 51 booth (#1417).
Integrating real world reconstructions into simulation pipelines
Once a real world drive has been reconstructed, the next step is to either replay the original drive or simulate new scenarios from the digital twin. This requires a simulator that can drive an ego-vehicle, dictate the motion of other actors in the scene, and orchestrate all the events in the scene.
CARLA open source AV simulator
CARLA is one of the world’s most popular open source simulation platforms with more than 150,000 active developers, serving as a testbed for AV research and development. NVIDIA is partnering with CARLA to integrate the latest NuRec rendering APIs and Cosmos Transfer-1 world foundation model. This enables developers to generate sensor data from Gaussian representations with ray tracing and amplify diversity with Cosmos WFMs.
Below is an example of a scene where CARLA is orchestrating the motion of all agents, including the ego-vehicle, and rendering sensor data from the ego point of view using NuRec. By adding reconstructed scenes and simulating new events with CARLA’s APIs and traffic model integrations, we can create useful corner-case datasets.
Novel view generation with NuRec Fixer
When rendering a reconstructed scene from a novel view, there can be gaps in the reconstruction, which could lead to artifacts. NuRec Fixer is a transformer-based model post-trained on AV datasets to inpaint and resolve reconstruction artifacts. Developers can run Fixer during reconstruction or as a post-process during neural rendering to fix such artifacts. Fixer is based on the Difix3D+ paper released at CVPR 2025. With Fixer, novel view synthesis from reconstructed scenes becomes practical for open and closed-loop simulation workflows.
NVIDIA Physical AI Dataset
Developers can try out this pipeline using open source data available on the NVIDIA Physical AI Dataset. The latest dataset release includes 40,000 clips generated using Cosmos, as well as sample reconstructed scenes for neural rendering. With this latest version of CARLA, developers can now author completely new trajectories, reposition the camera, and simulate drives with this starter pack of reconstructed data.
Diversify with Cosmos Transfer
We can further scale and accelerate the data flywheel with the ability to improve data realism and diversity with Cosmos Transfer. Modeling particulate effects for weather, achieving variations in lighting and procedurally generating 3D content is a complex technical challenge. Cosmos WFMs—Reason, Predict and Transfer—have been trained on massive internet-scale data and have general understanding and prediction capabilities. Cosmos Transfer is a diffusion-based generative model that gives developers the ability to use a prompt as well as sensor data as input to condition the model and generate different variants of an existing scene, and is available in the latest version of CARLA.
Behavior: Traffic models in simulation
CARLA developers using behavioural directable agent models like Imagining The Road Ahead (ITRA) from Inverted AI, and AV developers using the Foretellix Foretify data-automation toolchain, pre-integrated with CARLA and NVIDIA Cosmos, can generate realistic variations in scenarios and behaviors and scale up behavioral diversity. CVPR attendees can try this workflow in the hands-on tutorial “Advancing Data Strategies for AI Success.”
NVIDIA Omniverse Blueprint for AV Simulation
NVIDIA is incorporating these core technologies into a reference workflow to help developers build an end-to-end simulation pipeline.
Coming soon, the Omniverse Blueprint for AV Simulation will enable developers to enhance their own simulators with neurally reconstructed scenes, insert synthetic actors into these scenes, model physics and animation, and render physically-based and NeRF content with composite rendering.
Organizations such as Foretellix, MathWorks, CARLA, and Mcity are all building workflows with the blueprint to accelerate AV development for their end users.
These models and workflows are designed to help the AV developer community take on the toughest challenges in standing up data pipelines for training, testing and validating autonomous vehicles.
Get started developing today
- Learn more about AV Simulation use cases
- Get started with NuRec and Cosmos Transfer on CARLA
- Download the Physical AI Dataset
- Download Cosmos Transfer
- Download NuRec Fixer
Explore the NVIDIA research papers to be presented at CVPR 2025, and watch the NVIDIA GTC Paris keynote from NVIDIA founder and CEO Jensen Huang.
Stay up-to-date by subscribing to NVIDIA news and following NVIDIA Omniverse on Discord and YouTube.
- Visit our Omniverse developer page to get all the essentials you need to get started
- Access a collection of OpenUSD resources, including the new self-paced Learn OpenUSD training curriculum
- Tune into upcoming OpenUSD Insiders livestreams and connect with the NVIDIA Developer Community
Get started with developer starter kits to quickly develop and enhance your own applications and services.
