NVIDIA DRIVE Perception
NVIDIA DRIVE™ Perception enables robust perception of obstacles, paths, and wait conditions (such as stop signs and traffic lights) right out of the box with an extensive set of pre-processing, post-processing, and fusion processing modules. Together with DRIVE Networks, these form an end-to-end perception pipeline for autonomous driving that uses data from multiple sensor types (e.g. camera, radar, LIDAR). DRIVE Perception makes it possible for developers to innovate mapping and/or planning/control/actuation autonomous vehicle (AV) functionalities without having to first develop and validate the underlying live perception building blocks.
DRIVE Perception is ideal for a variety of objectives:
- Track detected objects (such as other vehicles, pedestrians, road markings) from frame to frame.
- Estimate distances of detected objects.
- Fuse inputs from sensors of different modalities.

Details
Obstacle Perception
The NVIDIA DRIVE AV Obstacle Perception pipeline consists of interacting algorithmic modules built around core deep neural networks (DNNs), plus DNN post-processing. Capabilities include:
Camera-based:
- Obstacle detection including cars, pedestrians, and traffic lights (based on DriveNet DNN)
- Lane detection and assignment (based on MapNet DNN)
- Distance-to-obstacle detection (based on DepthNet DNN)
- Drivable free-space detection (based on OpenRoadNet DNN)
- Camera image clarity detection and classification (based on ClearSightNet DNN)
- Semantic motion segmentation (SMS) for detection of both static and dynamic objects
Radar-based:
- Obstacle detection and tracking over time
Path Perception
The NVIDIA DRIVE AV Path Perception pipeline consists of interacting algorithmic modules built around core DNN(s), using HD Map and DNN post-processing. Capabilities include:
- Camera-based path perception (using LaneNet DNN)
- Camera-based path perception (using PathNet DNN)
- Path perception signal generation using HD Map data
- Algorithm modules that enable diversity and redundancy in path perception by combining multiple individual path perception signals (e.g. multiple DNN-based outputs, HD Map-based outputs, egomotion-based outputs) and generating a confidence metric on the combined (ensemble) path-perception output
Wait Perception
The NVIDIA DRIVE AV Wait Perception pipeline consists of interacting algorithmic modules built around the core DNN(s) and DNN post-processing. Capabilities include:
- Camera-based traffic light perception (using LightNet DNN)
- Camera-based traffic sign perception, such as stop, yield, speed limits (using SignNet DNN)
- Camera-based wait conditions perceptions, such as intersections, distance to intersection (using WaitNet DNN)
NVIDIA DRIVE AGX Pegasus Highway Test
The autonomous test car powered by NVIDIA DRIVE AGX Pegasus(™) with DRIVE Software completes a 50-mile route around Silicon Valley. It autonomously drives on four different freeways: I-280, CA-92, CA-101, and CA-85, making lane changes and merging on and off highway interchanges without disengagements or human intervention.
Developing with DRIVE Perception
How to set up
You will need:
- Host Development PC
- DRIVE AGX Developer Kit
- DRIVE Hyperion Kit
Steps:
- Install DRIVE Perception using the SDK Manager
- Consult the Roadrunner* Users Guide (DRIVE AV → Roadrunner → Roadrunner Reference → Architecture) included in the DRIVE Software Documentation * Roadrunner is NVIDIA’s autonomous driving application developed using the DriveWorks SDK.
How to develop
Development Tasks | Getting Started |
---|---|
Evaluate NVIDIA DRIVE Perception |
|
Additional Development Resources:
Documentation
- Consult the Roadrunner Users Guide (DRIVE AV Roadrunner Roadrunner Reference Architecture)
- All NVIDIA DRIVE documentation can be found here: https://developer.nvidia.com/drive/documentation