NVIDIA AI Infrastructure for
Autonomous Vehicles
The NVIDIA AI Enterprise Software Suite delivers the essential tools you need for streamlining the development and deployment of AV software. This includes everything from data preparation and training to optimizing for inference and deploying at scale.
.png)
Click to enlarge
Note: Before getting started, make sure you have the latest NVIDIA products (e.g. DGX) and the corresponding drivers for your NVIDIA GPU(s).
Step 1: Data Preprocessing and Extract, Transform, and Load (ETL)
Transforming Raw Sensor Data
Use Apache Spark, plus the NVIDIA RAPIDS™ suite of open source libraries and APIs, to help accelerate cleaning, filtering, and transforming your raw sensor data into a more structured format.
Analyze your workloads using the Accelerated Spark Analysis Tool.
Video Processing
Decode and convert compressed data to frames using NVIDIA Video Processing Framework (VPF).
Adjust image size and aspect ratio, standardize pixel values, and enhance image diversity and clarity by using NVIDIA® CV-CUDA™ — an open-source, GPU-accelerated library for large-scale image-processing tasks.
Auto-Labeling
Use NVIDIA Triton™ Inferencing Server for your auto-labeling DNN models, preferably in NVIDIA TensorRT™ format to get the highest throughput. You may also execute pre-processing (e.g., decoding, re-shaping, filtering, etc.) and post-processing (e.g., encoding bounding boxes, reshaping, etc.) operations on the same GPU for the end-to-end maximum auto-labeling speed and high GPU utilization. You can use the tools introduced in the Video Processing section on this page.
Scene Mining
It’s critical to minimize the time spent on extracting relevant scenes to build training or test datasets. In addition to keyword search within the meta data, DNN models can help with synthetic and semantic mining of scenes. Use Triton Inferencing Server for your auto-labeling DNN models, preferably in NVIDIA TensorRT format to get the highest throughput. Similar to auto-labeling, you can use the tools introduced in the Video Processing section on this page.
Step 2: Training and Optimizations
NVIDIA offers Docker containers for your preferred deep learning framework through the NGC catalog.
GPU Acceleration
Enable GPU acceleration using NVIDIA CUDA® Toolkit and NVIDIA CUDA Deep Neural Network library (cuDNN). The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler, and a runtime library. While cuDNN offers GPU-accelerated library of primitives for deep neural networks.
Multi-GPU Acceleration
Achieve multi-GPU acceleration using NVIDIA Magnum IO™ and NCCL. A variety of software and techniques support distributed training, allowing multiple GPUs to train larger deep learning models. These libraries are compatible with all NVIDIA networking solutions, including NVLink™ and NVSwitch™.
Mixed-Precision Training
Mixed-precision training is a technique that uses lower-precision floating-point numbers (e.g., half-precision) to accelerate training by using Tensor Cores. Learn more about using automatic mixed precision for major deep learning frameworks.
NVIDIA DALI®
NVIDIA Data Loading Library (DALI®) is a portable, open-source software library that can be used for decoding and augmenting images, videos, and speech. This helps accelerate DLI applications by reducing data access latency and training time.
Step 3: Simulation and Testing
Simulation
Simulation of real-world scenarios isn't only critical for training for unforeseen scenarios, but also vital for testing and validation of autonomous vehicles. Deploying autonomy in a safe and trustworthy manner requires achieving real-world fidelity, which is incredibly challenging without simulation of rare real-world scenarios. It requires accurate modeling of the physics and behavior of an autonomous systems sensors and surroundings.
NVIDIA Omniverse Cloud Sensor RTX for AV simulation deliver large-scale, high-fidelity sensor simulation to accelerate AV development, giving you a seamless way to effectively test countless “what if'' scenarios and diverse environmental conditions.
Replay
Playing back recorded logs and recorded sensor data can be used to analyze and diagnose issues that may have occurred during testing or in the field. Sensor data, system logs, and other data can be replayed to recreate the conditions under which a problem occurred and help identify the root cause. Replay can also be used to evaluate the performance of different algorithms and systems under different conditions.