Neural Reconstruction and 3D Gaussian Splatting With NVIDIA Omniverse NuRec
NVIDIA Omniverse™ NuRec is a set of agent-friendly, 3D Gaussian splatting libraries that ingest real sensor data to reconstruct and render interactive simulation in OpenUSD. NuRec libraries are integrated with NVIDIA Isaac Sim™ for robotics testing, and NVIDIA AlpaSim and CARLA open source simulator for autonomous driving development.
How Omniverse NuRec Works: Real-Time 3D Reconstruction and Rendering
Omniverse NuRec reconstructs camera or lidar data of a given environment and packages the result as a USD scene, including metadata such as trajectories. The open NCore data standard provides a consistent way to represent data inputs to reconstruction workflows, making NuRec libraries easy to integrate into automated pipelines and AI agent workflows.
Using open-source gsplat Gaussian splatting rendering libraries, NuRec then renders the scene interactively inside simulation frameworks like Isaac Sim or AlpaSim. Rendering is usually done by connecting the simulator/runtime to the NuRec service through a gRPC API. This can also be called by AI agents to generate, inspect, and refine reconstructed environments programmatically.
Developers can then further enhance the reconstruction using generative models such as Harmonizer, which corrects artifacts, lighting, and shadow, or Asset Harvester, which removes assets for use in other environments.
Enhance Neural Reconstructions With Harmonizer
Address artifacts and adjust blurs in reconstructed scenes using Harmonizer, built on NVIDIA Cosmos™.
Build Asset Libraries With Harvester for USD Scenes
Easily remove objects in a reconstruction for use in other scenes with Asset Harvester.
Reconstruct a 3D Scene Using Your Smartphone
Learn how to quickly turn smartphone captures into an interactive simulation environment in Isaac Sim using Omniverse NuRec.
Simulate Real-World Sensor Data in CARLA
Follow this step-by-step tutorial to turn real-world scenes into autonomous vehicle simulation in CARLA.
Get Started With Omniverse NuRec for Accelerated 3D Reconstruction
Omniverse NuRec on NGC for Reconstruction Pipelines
Access containers, libraries, and models to start your reconstruction pipelines.
Download Omniverse NuRec
NCore Data Standard for Sensor Data Reconstruction
Reconstruct sensor data using consistent data representations.
Access NCore Data Standard
Physical AI Open Datasets for Reconstructed Data
Kick-start your rendering pipelines with reconstructed data, openly available on the NVIDIA Physical AI Open Datasets collection.
Explore Open DatasetsStarter Kits
Neural Reconstruction and Rendering Starter Kit
Turn real-world sensor data into interactive simulation using 3D Gaussian splatting-based rendering for enhanced efficiency and accuracy.
Realistic Physics Simulation for Physical AI
Model the physical behavior of objects and systems foundational to physical AI.
Isaac Sim can accurately simulate rigid body and vehicle dynamics, multi-joint articulation, SDF colliders, and more.
Scalable Synthetic Data Generation for AI Model Training
Bootstrap AI model training with synthetic data.
Generate training data by randomizing attributes like lighting, reflection, color, and position of scene and assets.
FAQs about NuRec
NVIDIA Omniverse™ NuRec is a set of accelerated 3D Gaussian splatting libraries designed to ingest real sensor data to reconstruct and render interactive simulation in OpenUSD.
3D Gaussian Splatting is a modern rendering technique used by NuRec libraries. It takes a reconstructed USD scene, built from camera or lidar data, and renders it interactively inside simulation frameworks like Isaac Sim or AlpaSim using open-source gsplat libraries.
NuRec libraries are integrated to support two primary development areas: robotics testing through NVIDIA Isaac Sim™, and autonomous driving development using NVIDIA AlpaSim and the CARLA open source simulator.
You can access containers, libraries, and models for your reconstruction pipelines by downloading Omniverse NuRec from NGC (NVIDIA GPU Cloud).
The comprehensive documentation is available at the official NuRec documentation portal: View Documentation.
The open NCore data standard is a consistent way to represent data inputs for reconstruction workflows. It ensures uniform data representation when reconstructing sensor data.
Rendering is typically achieved by connecting the simulator or runtime environment (e.g., Isaac Sim or AlpaSim) to the NuRec service using a gRPC API.
Fixer is a generative model built on NVIDIA Cosmos™ that developers can use to enhance neural reconstructions by filling in artifacts or adjusting blurs in reconstructed scenes.
Asset Harvester is a generative model that allows you to easily remove objects from a reconstructed scene, making the extracted assets available for use in other environments or scenes.
Yes, the Neural Reconstruction and Rendering Starter Kit With NVIDIA Omniverse NuRec is available to help developers turn real-world sensor data into interactive simulations.
You can engage with the developer community and ask technical questions on the dedicated NuRec Forum and Omniverse Discord server.
Yes, a technical blog post demonstrates how to Reconstruct a 3D Scene Using Your Smartphone to quickly create an interactive simulation environment in Isaac Sim.
Yes, you can follow a step-by-step tutorial video that illustrates how to turn real-world scenes into autonomous vehicle simulation within CARLA.
The starter kit provides a learning path including documentation such as Getting Started With Neural Rendering, technical blogs, and information on advanced techniques like 3DGUT.
Realistic physics simulation involves accurately modeling the physical behavior of objects and systems, such as rigid body and vehicle dynamics, multi-joint articulation, and SDF colliders. This capability, present in simulators like Isaac Sim, is foundational to physical AI development.
NVIDIA uses its own proprietary engine, NVIDIA® PhysX®, and references the Newton open source physics engine for robotics simulation.
You can access reconstructed data openly available on the NVIDIA Physical AI Open Datasets collection to kick-start your rendering pipelines.
You can bootstrap AI model training by generating synthetic data and randomizing attributes like lighting, reflection, color, and scene position. The Scalable Synthetic Data Generation for AI Model Training section links to the Omniverse Replicator Getting Started Guide for detailed information.
The focused keywords for this page include Neural reconstruction, 3D reconstruction, and Gaussian Splatting.1
NVIDIA maintains that Trustworthy AI is a shared responsibility. Developers are required to work with the supporting model team to ensure the model meets industry requirements and to proactively address potential product misuse. Further detail is provided in the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
More Resources
Ethical AI
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns here.
Access NVIDIA Omniverse for NuRec to use in 3D Reconstruction.