Detecting Obstacles and Drivable Free Space with RadarNet

Detecting drivable free space is a critical component of advanced driver assistance systems (ADAS) and autonomous vehicle (AV) perception. Obstacle detection is usually performed to detect a set of specific dynamic obstacles, such as vehicles and pedestrians.

In contrast, free space detection is a more generalized approach for obstacle detection. It enables autonomous vehicles to navigate safely around many types of obstacles, such as trees or curb stones, even without being explicitly trained to identify the specific obstacle class.

Traditionally, camera systems have been used to solve this task. However, camera perception performance can suffer in adverse weather and low-light conditions, or when identifying objects at greater distances from the vehicle. To overcome these challenges, we have developed a free space detection system using radar. It is robust against weather and illumination challenges, and is able to directly measure distance.

Our system works as part of the ADAS and AV perception in order to detect drivable free space and to further improve 3D perception during multisensor fusion. More specifically, RadarNet is a deep neural network (DNN) that detects dynamic obstacles and drivable free space using automotive radar sensors. For more details, see NVRadarNet: Real-Time Radar Obstacle and Free Space Detection for Autonomous Driving.

Watch an example of the RadarNet deep neural network in action in the NVIDIA DRIVE Dispatch video below.

Video 1. Watch an example of the RadarNet deep neural network in action

Dense occupancy probability representation

Traditional radar perception algorithms struggle to robustly classify drivable free space using the radar data alone due to the sparse nature of the radar detections. To overcome this challenge, RadarNet is able to regress a dense occupancy probability map in a 2D birds-eye-view (BEV). This approach was inspired by the work presented in Probably Unknown: Deep Inverse Sensor Modelling Radar.

The labels are generated by first combining existing human-labeled 3D-obstacle and free-space labels, followed by an automatic occupancy calculation using the observations derived from the raw lidar point clouds. Ray tracing from the vehicle origin to each obstacle enables reasoning about which regions are:

observed and free
observed and occupied
unobserved
partially observed

Visual representation of a semi-automatically generated occupancy training label, including observed and free areas (black), observed and occupied areas (white), unobserved areas (light gray), and partially observed areas (dark gray). — Figure 1. Visual representation of a semi-automatically generated occupancy training label, including observed and free areas (black), observed and occupied areas (white), unobserved areas (light gray), and partially observed areas (dark gray)

The DNN is trained to output the following occupancy predictions:

high probabilities for regions that are observed directly and occupied
low probabilities for regions that are observed directly and are free of any obstacles
medium probabilities, staying explicitly uncertain, for regions that are unobserved, outputting a probability of (p = 0.5)

Figure 2. Inferred dense occupancy probability map, with dark areas showing low probability, bright areas showing high probability, and red areas showing uncertain probability. — *Figure 2. Inferred dense occupancy probability map, with dark areas showing low probability, bright areas showing high probability, and red areas showing uncertain probability*

The observed and occupied regions can be further broken down into other categories, such as dynamic obstacle classes. Regions can even be marked as elevated by postprocessing information from the lidar sensor. This leads to a segmentation map with the following classifications:

dynamic obstacles (vehicles, pedestrians)
general non-driveable obstacles (guard rails, bushes, light poles)
general under-drivable obstacles (bridges, trees overhanging the road)
unobserved

A GIF of an occupancy probability map overlaid with fine-grained semantic segmentation showing the ego vehicle (purple), other vehicles (green), general obstacles (blue), and elevated obstacles (red). — *Figure 3. Occupancy probability map overlaid with fine-grained semantic segmentation showing the ego vehicle (purple), other vehicles (green), general obstacles (blue), and elevated obstacles (red)*

The occupancy probability map was trained alongside object detection as an additional task-head of the RadarNet DNN. Doing so guarantees consistency between these tasks and reduces the overall computational requirements. The total latency increase is no more than 10% when compared to the original RadarNet.

Extraction of a drivable free space radial distance map

This dense occupancy probability representation can be further postprocessed to detect the drivable free space boundary. This is done by tracing rays from the vehicle origin in all azimuth directions until the occupancy probability reaches a certain threshold.

In particular, the Cartesian BEV occupancy probability map was first resampled into a polar coordinate system centered around the vehicle origin. Second, we iterated over all of the angular directions marking the first bin above the desired probability threshold as the final boundary at that angle. This process can be further optimized by precomputing the Cartesian-to-polar resampling grid followed by a nearest-neighbor interpolation. The final Radial Distance Map (RDM) output is demonstrated as a set of polylines around the ego vehicle.

DNN BEV input image with radar input (left) and predictions projected to the camera (right). — *Figure 4. DNN BEV input image with radar input (left) and predictions projected to the camera (right)*

Conclusion

The radar-only drivable free space detector presented in this post predicts a dense occupancy probability to robustly detect the free space boundary with remarkable accuracy, while running fast enough to easily fit alongside the rest of the NVIDIA AV stack on NVIDIA DRIVE AGX Orin.

The current RadarNet supports both dynamic object detection and drivable free space detection. Object detections are classified into multiple classes, such as cars, bikes, and pedestrians. Occupied areas are further segmented into elevated, non-elevated, or unobserved. RadarNet output is then used as an input to a multisensor fusion system with a camera-based detector, improving AV robustness.

For a more detailed explanation, see NVRadarNet: Real-Time Radar Obstacle and Free Space Detection for Autonomous Driving. To learn more about the software functionality, see Perceiving with Confidence: How AI Improves Radar Perception for Autonomous Vehicles from the NVIDIA DRIVE Lab video series, and the NVIDIA DRIVE Dispatch video describing RadarNet free space.

Register for NVIDIA GTC 2023 for free and join us March 20–23 for more inspiring content and related expert-led sessions.