Recreate High-Fidelity Digital Twins with Neural Kernel Surface Reconstruction

Reconstructing a smooth surface from a point cloud is a fundamental step in creating digital twins of real-world objects and scenes. Algorithms for surface reconstruction appear in various applications, such as industrial simulation, video game development, architectural design, medical imaging, and robotics.

Neural Kernel Surface Reconstruction (NKSR) is the new NVIDIA algorithm for reconstructing high-fidelity surfaces from large point clouds. NKSR can process millions of points in seconds and achieves state-of-the-art quality on a wide range of benchmarks. NKSR is an excellent substitute for traditional Poisson Surface Reconstruction, providing greater detail and faster runtimes.

NKSR leverages a novel 3D deep learning approach called Neural Kernel Fields to achieve high-quality reconstruction. First introduced in 2022 by the NVIDIA Toronto AI Lab, Neural Kernel Fields predict a data-dependent set of basis functions used to solve the closed-form surface reconstruction problem. This new approach enables unprecedented generalization (for training on objects and reconstructing scenes) and multimodal training on scenes and objects at different scales. For more technical details about the method, visit the NKSR project page.

A 3D image of a street captured using NKSR. — *Figure 1. Reconstructed geometry on Waymo Open Dataset*

The kitchen sink model

Alongside the code release, we are excited to introduce the kitchen sink model, a comprehensive model trained on datasets of varying scales. By incorporating object-level and scene-level data, we have ensured the model’s versatility across different scenarios. To demonstrate its effectiveness, we have successfully applied the kitchen sink model to diverse datasets.

Figure 2 shows a room-level reconstruction result using a sparse input point cloud. Our method outperforms other baselines by generating smooth and accurate geometry.

A GIF showing a 3D model of a room filled with chairs and accessories captured using NKSR. — *Figure 2. Reconstruction performance of indoor rooms compared to the baselines*

Figure 3 showcases the application of our method on a race track, and Figure 4 shows a neighborhood scene. These scenes were captured using an autonomous vehicle equipped with a lidar sensor. Both scenes span several kilometers in length, and we were able to efficiently process them on a GPU.

A GIF showing a 3D reconstruction of a race track using NKSR. — *Figure 3. Reconstructed geometry of a long race track*

A GIF of a neighborhood that switches between the RGB capture and the 3D model rendered by NKSR — *Figure 4. Reconstructed geometry of a neighborhood*

How to use NKSR

NKSR is easily accessible through pip, with PyTorch being a key dependency. This integration enables the direct installation of the package, ensuring a streamlined setup process.

Use the following installation command:

pip install nksr -f https://nksr.s3.ap-northeast-1.amazonaws.com/whl/torch-2.0.0%2Bcu118.html

The core computing operations of NKSR are accelerated using GPU, which results in high-speed processing and efficient performance. When deploying NKSR, it is necessary to define the positions and normals of your input point cloud. Alternatively, you can input the positions of the sensors capturing these points.

The code snippet below demonstrates how easy it is to use NKSR:

import nksr
import torch

device = torch.device("cuda:0")
reconstructor = nksr.Reconstructor(device)

# Note that input_xyz and input_normal are torch tensors of shape [N, 3] and [N, 3] respectively.
field = reconstructor.reconstruct(input_xyz, input_normal)
# input_color is also a tensor of shape [N, 3]
field.set_texture_field(nksr.fields.PCNNField(input_xyz, input_color))
# Increase the dual mesh's resolution.
mesh = field.extract_dual_mesh(mise_iter=2)

# Visualizing
from pycg import vis
vis.show_3d([vis.mesh(mesh.v, mesh.f, color=mesh.c)])

The culmination of this process is a triangulated mesh, which you can save directly or visualize based on your specific needs.

If the default configuration (kitchen sink model) does not adequately meet your requirements, the training code is provided. This additional resource offers the flexibility to train a custom model or to integrate NKSR into your existing pipeline. Our commitment to customization and usability ensures that NKSR can be adapted to various applications and scenarios.

Conclusion

NVIDIA Neural Kernel Surface Reconstruction presents a novel and cutting-edge approach to extract high-quality 3D surfaces from sparse point clouds. By employing a sparse Neural Kernel Field approach, the NKSR kitchen sink model can generalize to arbitrary inputs at any scale from a fixed training set of shapes. NKSR is fully open source and comes with training code and the pretrained kitchen sink model, which you can install directly with pip. You can further fine-tune NKSR to your specific datasets and problem domains, to enable even higher reconstruction quality in specialized applications.

Use NKSR for a chance to win a free GPU

We want to see what you can do with NKSR. Demonstrate your most impressive reconstruction results and captivating visualizations using NKSR. Then enter the NVIDIA NKSR Sweepstakes by September 8, 2023 for your chance to win a GeForce RTX 3090 Ti.