1. [Home ](/) [Omniverse](/omniverse) NuRec # Neural Reconstruction and 3D Gaussian Splatting With NVIDIA Omniverse NuRec NVIDIA Omniverse™ NuRec is a set of agent-friendly, 3D Gaussian splatting libraries that ingest real sensor data to reconstruct and render interactive simulation in OpenUSD. NuRec libraries are integrated with [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim)™ for robotics testing, and [NVIDIA AlpaSim](https://github.com/NVlabs/alpasim) and [CARLA](https://carla.org/) open source simulator for autonomous driving development. [Download Omniverse NuRec](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/containers/nre-ga?version=latest "Download")[View Documentation](https://docs.nvidia.com/nurec/index.html "Documentation")[Agent Skills](https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-neural-reconstruction "Forum") * * * ## How Omniverse NuRec Works: Real-Time 3D Reconstruction and Rendering Omniverse NuRec reconstructs camera or lidar data of a given environment and packages the result as a USD scene, including metadata such as trajectories. The [open NCore data standard](https://github.com/nvidia/ncore) provides a consistent way to represent data inputs to reconstruction workflows, making NuRec libraries easy to integrate into automated pipelines and AI agent workflows. Using open-source [gsplat](https://github.com/nerfstudio-project/gsplat) Gaussian splatting rendering libraries, NuRec then renders the scene interactively inside simulation frameworks like Isaac Sim or AlpaSim. Rendering is usually done by connecting the simulator/runtime to the NuRec service through a gRPC API. This can also be called by AI agents to generate, inspect, and refine reconstructed environments programmatically. Developers can then further enhance the reconstruction using generative models such as Harmonizer, which corrects artifacts, lighting, and shadow, or [Asset Harvester,](https://huggingface.co/nvidia/asset-harvester) which removes assets for use in other environments. ![Flow diagram illustrating how Omniverse NuRec uses real sensor data for 3D reconstruction and interactive simulation rendering via Gaussian Splatting libraries.](https://developer.download.nvidia.com/images/nurec/public-sector-news-lilly-ari-1920x1280.png) ### Enhance Neural Reconstructions With Harmonizer Address artifacts and adjust blurs in reconstructed scenes using Harmonizer, built on NVIDIA Cosmos™. [Access Harmonizer](https://huggingface.co/nvidia/Fixer) ### Build Asset Libraries With Harvester for USD Scenes Easily remove objects in a reconstruction for use in other scenes with Asset Harvester. [Access Harvester](https://huggingface.co/nvidia/asset-harvester) ### Reconstruct a 3D Scene Using Your Smartphone Learn how to quickly turn smartphone captures into an interactive simulation environment in Isaac Sim using Omniverse NuRec. [Read the Blog](https://developer.nvidia.com/blog/reconstruct-a-scene-in-nvidia-isaac-sim-using-only-a-smartphone/) ### Simulate Real-World Sensor Data in CARLA Follow this step-by-step tutorial to turn real-world scenes into autonomous vehicle simulation in CARLA. [Watch the Tutorial (00:45)](https://www.youtube.com/watch?v=TRcFBSA-Lnk) * * * ## Get Started With Omniverse NuRec for Accelerated 3D Reconstruction ![decorative icon](https://developer.download.nvidia.com/icons/m48-workflow-complex.svg) ### Omniverse NuRec on NGC for Reconstruction Pipelines Access containers, libraries, and models to start your reconstruction pipelines. [Download Omniverse NuRec](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/containers/nre-ga?version=latest) ![decorative icon](https://developer.download.nvidia.com/icons/m48-demo-topics.svg) ### NCore Data Standard for Sensor Data Reconstruction Reconstruct sensor data using consistent data representations. [Access NCore Data Standard ](https://docs.nvidia.com/nurec/basics/get-data.html) ![decorative icon](https://developer.download.nvidia.com/icons/m48-neural-network-3.svg) ### Physical AI Open Datasets for Reconstructed Data Kick-start your rendering pipelines with reconstructed data, openly available on the NVIDIA Physical AI Open Datasets collection. [Explore Open Datasets](https://huggingface.co/collections/nvidia/physical-ai) * * * ## Starter Kits ### Neural Reconstruction and Rendering Starter Kit Turn real-world sensor data into interactive simulation using 3D Gaussian splatting-based rendering for enhanced efficiency and accuracy. - [Getting Started With Neural Rendering](https://docs.nvidia.com/nurec/index.html) - [How to Instantly Render Real-World Scenes in Interactive Simulation](https://developer.nvidia.com/blog/how-to-instantly-render-real-world-scenes-in-interactive-simulation/) - [3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting](https://research.nvidia.com/labs/toronto-ai/3DGUT/) - [Reconstruct a Scene in NVIDIA Isaac Sim Using Only a Smartphone](https://developer.nvidia.com/blog/reconstruct-a-scene-in-nvidia-isaac-sim-using-only-a-smartphone/) - [Simulate Robotic Environments Faster With NVIDIA Isaac Sim and World Labs Marble](https://developer.nvidia.com/blog/simulate-robotic-environments-faster-with-nvidia-isaac-sim-and-world-labs-marble) ### Realistic Physics Simulation for Physical AI Model the physical behavior of objects and systems foundational to physical AI. Isaac Sim can accurately simulate rigid body and vehicle dynamics, multi-joint articulation, SDF colliders, and more. - [Physics Simulation Fundamentals](https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/index.html) - [Getting Started Guides for Sensor Simulation](https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/index.html) - [NVIDIA® PhysX®](/physx-sdk) - [Newton](/newton-physics) ### Scalable Synthetic Data Generation for AI Model Training Bootstrap AI model training with synthetic data. Generate training data by randomizing attributes like lighting, reflection, color, and position of scene and assets. - [Synthetic Data Generation Use Cases](https://docs.nvidia.com/tao/tao-toolkit/text/qat_and_amp_for_training.html#knowledge-distillation) - [Omniverse Replicator Getting Started Guide](https://docs.nvidia.com/tao/tao-toolkit/text/qat_and_amp_for_training.html#knowledge-distillation) * * * ## FAQs about NuRec NVIDIA Omniverse™ NuRec is a set of accelerated 3D Gaussian splatting libraries designed to ingest real sensor data to reconstruct and render interactive simulation in [OpenUSD](https://developer.nvidia.com/usd). 3D Gaussian Splatting is a modern rendering technique used by NuRec libraries. It takes a reconstructed USD scene, built from camera or lidar data, and renders it interactively inside simulation frameworks like Isaac Sim or AlpaSim using open-source [gsplat](https://github.com/nerfstudio-project/gsplat) libraries. NuRec libraries are integrated to support two primary development areas: robotics testing through [NVIDIA Isaac Sim](/isaac/sim)™, and autonomous driving development using [NVIDIA AlpaSim](https://github.com/NVlabs/alpasim) and the [CARLA](https://carla.org/) open source simulator. You can access containers, libraries, and models for your reconstruction pipelines by downloading Omniverse NuRec from [NGC (NVIDIA GPU Cloud)](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/containers/nre-ga?version=latest). The comprehensive documentation is available at the official NuRec documentation portal: View [Documentation](https://docs.nvidia.com/nurec/index.html). The [open NCore data standard](https://github.com/nvidia/ncore) is a consistent way to represent data inputs for reconstruction workflows. It ensures uniform data representation when reconstructing sensor data. Rendering is typically achieved by connecting the simulator or runtime environment (e.g., Isaac Sim or AlpaSim) to the NuRec service using a gRPC API. [Fixer](https://huggingface.co/nvidia/Fixer) is a generative model built on NVIDIA Cosmos™ that developers can use to enhance neural reconstructions by filling in artifacts or adjusting blurs in reconstructed scenes. [Asset Harvester](https://huggingface.co/nvidia/asset-harvester) is a generative model that allows you to easily remove objects from a reconstructed scene, making the extracted assets available for use in other environments or scenes. Yes, the **Neural Reconstruction and Rendering Starter Kit With NVIDIA Omniverse NuRec** is available to help developers turn real-world sensor data into interactive simulations. You can engage with the developer community and ask technical questions on the dedicated [NuRec Forum](https://forums.developer.nvidia.com/c/omniverse/platform/nurec/752) and [Omniverse Discord](https://discord.com/channels/827959428476174346/1486466223012315366) server. Yes, a technical blog post demonstrates how to [Reconstruct a 3D Scene Using Your Smartphone](https://developer.nvidia.com/blog/reconstruct-a-scene-in-nvidia-isaac-sim-using-only-a-smartphone/)to quickly create an interactive simulation environment in Isaac Sim. Yes, you can follow a step-by-step [tutorial video](https://www.youtube.com/watch?v=TRcFBSA-Lnk) that illustrates how to turn real-world scenes into autonomous vehicle simulation within CARLA. The starter kit provides a learning path including documentation such as [Getting Started With Neural Rendering](https://docs.omniverse.nvidia.com/materials-and-rendering/latest/neural-rendering.html), technical blogs, and information on advanced techniques like [3DGUT](https://research.nvidia.com/labs/toronto-ai/3DGUT/). Realistic physics simulation involves accurately modeling the physical behavior of objects and systems, such as rigid body and vehicle dynamics, multi-joint articulation, and SDF colliders. This capability, present in simulators like Isaac Sim, is foundational to physical AI development. NVIDIA uses its own proprietary engine, NVIDIA® [PhysX](/physx-sdk)®, and references the [Newton](https://developer.nvidia.com/blog/announcing-newton-an-open-source-physics-engine-for-robotics-simulation) open source physics engine for robotics simulation. You can access reconstructed data openly available on the [NVIDIA Physical AI Open Datasets collection](https://huggingface.co/collections/nvidia/physical-ai) to kick-start your rendering pipelines. You can bootstrap AI model training by generating synthetic data and randomizing attributes like lighting, reflection, color, and scene position. The **Scalable Synthetic Data Generation for AI Model Training** section links to the [Omniverse Replicator Getting Started Guide](https://docs.omniverse.nvidia.com/extensions/latest/ext_replicator/getting_started.html)for detailed information. The focused keywords for this page include Neural reconstruction, 3D reconstruction, and Gaussian Splatting.1 NVIDIA maintains that Trustworthy AI is a shared responsibility. Developers are required to work with the supporting model team to ensure the model meets industry requirements and to proactively address potential product misuse. Further detail is provided in the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. * * * ## More Resources ![Decorative image representing Community](https://developer.download.nvidia.com/icons/m48-people-group.svg) ### Join the Community ![Decorative image representing Developer Newsletter](https://developer.download.nvidia.com/icons/m48-email-settings.svg) ### Sign Up for the Developer Newsletter ![Decorative image representing Developer Program](https://developer.download.nvidia.com/icons/m48-developer-1.svg) ### Join the NVIDIA Developer Program * * * ## Ethical AI NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). Access NVIDIA Omniverse for NuRec to use in 3D Reconstruction. [Download Omniverse NuRec](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/containers/nre-ga?version=latest)