NVIDIA Isaac ROS GEMs

The NVIDIA® Isaac ROS GEMs are hardware accelerated packages that make it easier for ROS developers to build high-performance solutions on NVIDIA hardware.

Get started

Composite Image from 3 Isaac ROS GEMs
Composite Image from 3 Isaac ROS GEMs -- DNN (PeopleSemSegnet)/AprilTags/Disparity(Depth)



High Throughput Perception

Isaac ROS GEMs provide packages which include image, computer vision, and DNN processing that are highly optimized for NVIDIA GPUs and Jetson.

Modular, Flexible Packages

Modular packages allow the ROS developer to take exactly what they need to integrate in their application. This means that they can replace an entire pipeline or simply swap out an algorithm.

Reduced Development Times

Isaac ROS GEMs are designed and tested to be similar to existing and familiar ROS nodes to make them easier to integrate in existing applications.




Rich Collection of Perception AI Packages for ROS Developers

ROS 2 nodes that address common image, computer, and DNN processing functionality that are key ingredients to delivering high performance perception to AI-based ROS robotics applications.

Isaak ROS



DNN Inference Processing

DNN Inference GEM is a set of ROS2 packages that allow developers to use any of NVIDIA’s numerous inference models available on NGC or even provide their own DNN. Further tuning of pre-trained models or optimizations of developers' own models can be done with the NVIDIA TAO Toolkit.

After optimization, these packages are deployed by TensorRT or Triton, NVIDIA’s inference server. Optimal inference performance will be achieved with the nodes leveraging TensorRT, NVIDIA’s high performance inference SDK. If the desired DNN model isn't supported by TensorRT then Triton can be used to deploy the model.


Additional GEMs incorporating model support are available and include support for U-Net and DOPE. The U-Net package, based on TensorRT, can be used for generating semantic segmentation masks from images. The DOPE package can be used for 3D pose estimation for all detected objects.


This tool is the fastest way to incorporate performant AI inference in a ROS application. The pre-trained model, PeopleSemSegNet, pictured in the image (right) runs at 25fps @544p.


Isaac ROS DNN Inference Isaac ROS Pose Estimation Isaac ROS Image Segmentation
DNN Inference GEM

Camera/Image Processing

Isaac ROS Image Processing

In a typical robotics image processing pipeline, raw data from the camera sensor must be processed before being passed off to a DNN or classic computer vision module for perception processing. This image processing consists of things like Lens Distortion Correction (LDC), image resizing, and image format conversion. If stereo cameras are involved then estimating disparity is also required. The image processing GEMs have been designed to take advantage of the specialized computer vision hardware available in Jetson like the GPU, the VIC (Video and Image Compositor) and the PVA (Programmable Vision Accelerator).

For robots using cameras connected via a CSI interface, NVIDIA provides the hardware accelerated Argus package.

Image shows lens distorted camera image (left) and rectified image using LDC GEM (right)

Isaac ROS Image Processing

Visual SLAM based Localization

As autonomous machines move around in their environments they must keep track of where they are. Visual odometry solves this problem by estimating where a camera is relative to its starting position. The Isaac ROS GEM for Stereo Visual Odometry provides this powerful functionality to ROS developers.

This GEM offers the best accuracy for a real-time stereo camera visual odometry solution. Publicly available results based on the widely used KITTI database can be referenced here. In addition to being very accurate, this GPU accelerated package runs extremely fast . In fact, it is now possible to run SLAM on HD resolution (1280x720) in real-time (>60fps) on a Jetson Xavier AGX.

Isaac ROS Stereo Visual Odometry
Isaak ROS Stereo Visual Odometry



Isaac ROS Camera Partners

Isaac ROS partners offer drivers that seamlessly integrate with the Isaac ROS GEMs. A complete list of drivers and compatible hardware can be found here.

d3_logo.png
d3_logo.png



More AI/Robotics Information




Accelerate your robotic application development today with NVIDIA Isaac ROS GEMs.

Subscribe