NVIDIA DRIVE Perception

NVIDIA DRIVE™ Perception enables robust perception of obstacles, paths, and wait conditions (such as stop signs and traffic lights) right out of the box with an extensive set of pre-processing, post-processing, and fusion processing modules. Together with DRIVE Networks, these form an end-to-end perception pipeline for autonomous driving that uses data from multiple sensor types (e.g. camera, radar, LIDAR). DRIVE Perception makes it possible for developers to innovate mapping and/or planning/control/actuation autonomous vehicle (AV) functionalities without having to first develop and validate the underlying live perception building blocks.

DRIVE Perception is ideal for a variety of objectives:

  • Track detected objects (such as other vehicles, pedestrians, road markings) from frame to frame.
  • Estimate distances of detected objects.
  • Fuse inputs from sensors of different modalities.

Details


Obstacle Perception

The NVIDIA DRIVE AV Obstacle Perception pipeline consists of interacting algorithmic modules built around core deep neural networks (DNNs), plus DNN post-processing. Capabilities include:

Camera-based:

  • Obstacle detection including cars, pedestrians, and traffic lights (based on DriveNet DNN)
  • Lane detection and assignment (based on MapNet DNN)
  • Distance-to-obstacle detection (based on DepthNet DNN)
  • Drivable free-space detection (based on OpenRoadNet DNN)
  • Camera image clarity detection and classification (based on ClearSightNet DNN)
  • Semantic motion segmentation (SMS) for detection of both static and dynamic objects

Radar-based:

  • Obstacle detection and tracking over time

LIDAR-based:

  • Obstacle and free-space detection


Path Perception

The NVIDIA DRIVE AV Path Perception pipeline consists of interacting algorithmic modules built around core DNN(s), using HD Map and DNN post-processing. Capabilities include:

  • Camera-based path perception (using LaneNet DNN)
  • Camera-based path perception (using PathNet DNN)
  • Path perception signal generation using HD Map data
  • Algorithm modules that enable diversity and redundancy in path perception by combining multiple individual path perception signals (e.g. multiple DNN-based outputs, HD Map-based outputs, egomotion-based outputs) and generating a confidence metric on the combined (ensemble) path-perception output

Wait Perception

The NVIDIA DRIVE AV Wait Perception pipeline consists of interacting algorithmic modules built around the core DNN(s) and DNN post-processing. Capabilities include:

  • Camera-based traffic light perception (using LightNet DNN)
  • Camera-based traffic sign perception, such as stop, yield, speed limits (using SignNet DNN)
  • Camera-based wait conditions perceptions, such as intersections, distance to intersection (using WaitNet DNN)

DRIVE Perception In Action

NVIDIA DRIVE AGX Pegasus Highway Test

The autonomous test car powered by NVIDIA DRIVE AGX Pegasus(™) with DRIVE Software completes a 50-mile route around Silicon Valley. It autonomously drives on four different freeways: I-280, CA-92, CA-101, and CA-85, making lane changes and merging on and off highway interchanges without disengagements or human intervention.


Developing with DRIVE Perception


How to set up

You will need:

Steps:

  • Install DRIVE Perception using the SDK Manager
  • Consult the Roadrunner* Users Guide (DRIVE AV → Roadrunner → Roadrunner Reference → Architecture) included in the DRIVE Software Documentation
    • * Roadrunner is NVIDIA’s autonomous driving application developed using the DriveWorks SDK.

How to develop

Development Tasks Getting Started
Evaluate NVIDIA DRIVE Perception
  • Save and analyze the DRIVE AV Application (Roadrunner) data log; refer to the RoadCast section in the NVIDIA Roadrunner user guide (found in the DRIVE Software Documentation).
* = Membership to the NVIDIA DRIVE Developer Program for DRIVE AGX is required to access this file.

Additional Development Resources:

Documentation