Building Robotics Applications Using ROS and NVIDIA Isaac SDK

The Robot Operating System (ROS) offers many software libraries and tools to help build robot applications, including framework, algorithms, sensors, and robot platforms. It is extremely popular with roboticists and researchers everywhere. A continuous influx of packages and libraries from the ecosystem makes it easy to get started on robotics projects.

Figure 1. Using the ROS bridge to connect the Isaac and ROS software stacks.

NVIDIA Isaac SDK is an open robotics platform to accelerate the development and deployment of AI-powered robots. Isaac SDK brings a SW framework, Isaac Engine, for efficient processing on NVIDIA Jetson platforms as well as GPUs. It also includes AI and GPU-accelerated algorithms, Isaac GEMS, as well as an integration to Isaac Sim for NVIDIA RTX-powered photo-real simulation. These capabilities can be used in a ROS-based application stack using the Isaac SDK ROS bridge (Figure 1). This allows developers and researchers to leverage both platforms for building robotics applications.

In this post, we show you how to use the ROS bridge to take an image from a camera in ROS format and use it with an Isaac GEM (Superpixel). Using examples, we also look at connecting Isaac GEMs and Isaac Sim with a ROS application stack as well as modules using the Isaac ROS bridge. Finally, we explain how to write a message converter (ROS bridge) for ROS to Isaac or Isaac to ROS scenarios.

Getting started

First, install ROS and Isaac SDK on your PC. Here’s what you can do with a ROS bridge:

bazel run packages/ros_bridge/apps/ros_to_superpixels
roslaunch realsense2_camera rs_rgbd.launch

Or, you can download and save ROS bag:

roslaunch realsense2_camera rs_from_file.launch rosbag_filename:="PathToRosBag"

Figure 2 shows the ROS bridge in action. A ROS camera image (or a ROS bag visualized in ROS rviz) is fed into an Isaac SDK GEM called Superpixel (visualized in Isaac Sight). The following section looks at how this plays out.

Image shows converting ROS camera image to Isaac proto for Isaac GEM.
Figure 2. Using a ROS bridge to connect to the Isaac GEM.

Isaac Superpixel GEM over ROS images

There are many computer vision algorithms in Isaac SDK that are GPU-accelerated. One of them is Superpixels. A superpixel is a group of connected pixels that are similar in appearance. Superpixel segmentation divides an image into hundreds of non-overlapping superpixels (rather than thousands or millions of individual pixels.) By using superpixels, you can compute features on more meaningful regions and reduce the number of input entities for consuming algorithms. There are many different superpixel algorithms of varying complexity and performance. Isaac SDK comes with a GPU-accelerated superpixel implementation for RGB-D images.

In this example, you produce RGB-D input sensor messages from ROS. This ROS image is then be consumed by the Isaac Superpixels GEM to produce superpixel segmentation.             

First, install ROS on your NVIDIA Jetson device. For more information about the ROS installation requirements to work with Isaac SDK, see Install ROS. This can be done on any of your preferred Jetson devices.

Next, connect an RGB-D camera, such as Realsense 435d, to the Jetson Developer Kit. You can also use a ROS Realsense rosbag such as structured.bag. To start the ROS Realsense camera driver and start producing color and depth sensor_msg types, follow the instructions.

You need a ROS bridge converter to convert ROS messages into Isaac image depth proto messages (as shown in the earlier example), to be consumed by the Isaac Superpixels GEM. Modify the existing RosToImage converter to include the ROS depth message along with the existing color image message. Do this on the development workstation on which Isaac SDK is downloaded (Figure 3).

bool RosToImage::rosToProto(const sensor_msgs::Image::ConstPtr& ros_message,
                            std::optional<ros::Time>& ros_time,
                            alice::ProtoTx<ImageProto>& tx_proto) {
  ros_time = ros_message->header.stamp;
  auto builder = tx_proto.initProto();
  std::vector<SharedBuffer>& buffers = tx_proto.buffers();
  // This converter is currently only supporting rgb8 color encoding and 16UC1 depth encoding
  if (ros_message->encoding == "rgb8") {
      ImageConstView3ub rgb_image =
          CreateImageView<uint8_t, 3>(static_cast<const uint8_t*>(&ros_message->data[0]),
                                      ros_message->height, ros_message->width);
      Image3ub color_image(rgb_image.dimensions());
      Copy(rgb_image, color_image);
      show("image", [&](sight::Sop& sop) { sop.add(rgb_image); });

      ToProto(std::move(color_image), builder, buffers);
  } else if (ros_message->encoding == "16UC1") {
      // ROS depth image with 16UC1 encoding is of type uint8_t
      ImageConstView1ui16 image = CreateImageView<uint16_t, 1>(
          reinterpret_cast<const uint16_t*>(&ros_message->data[0]),
          ros_message->height, ros_message->width);
      Image1f depth_image(image.dimensions());
      ConvertUi16ToF32(image, depth_image, 0.001);

      ToProto(std::move(depth_image), builder, buffers);
  } else {
      reportFailure("Unsupported image format: %s", ros_message->encoding.c_str());
      return false;
  }

  builder.setRows(ros_message->height);
  builder.setCols(ros_message->width);
  builder.setDataBufferIndex(0);

  return true;
}

Figure 3 shows the flow of ROS topics to Isaac proto.

ROS messages converted to Isaac proto using a ROS bridge.
Figure 3. ROS messages to Isaac proto converter: ROS topics
  • ROS converters help convert the ROS depth/Color and Camerainfo topics to Isaac proto using RosToImage and RosToCameraIntrinsics, respectively.
  • The SuperPixel codelet consumes the ROS camera driver images to produce superpixel segmentation.

The following code example shows how the Isaac app.json file looks for the ROS bridge converter edges and config:

{
  "name": "ros_to_superpixels",
  "modules": [
    "sight"
  ],
  "graph": {
    "nodes": [
      {
        "name": "ros_to_perception",
        "subgraph": "packages/ros_bridge/apps/ros_to_perception.subgraph.json"
      },
      {
        "name": "superpixels",
        "subgraph": "packages/superpixels/apps/superpixels.subgraph.json"
      }
    ],
    "edges": [
      {
        "source": "ros_to_perception.subgraph/interface/color",
        "target": "superpixels.subgraph/interface/color"
      },
      {
        "source": "ros_to_perception.subgraph/interface/depth_intrinsics",
        "target": "superpixels.subgraph/interface/depth_intrinsics"
      },
      {
        "source": "ros_to_perception.subgraph/interface/depth",
        "target": "superpixels.subgraph/interface/depth"
      }
    ]
  },
  "config": {
    "ros_to_perception.ros_converters": {
      "RosToDepth": {
        "channel_name": "/camera/depth/image_rect_raw"
      },
      "RosToCameraIntrinsics": {
        "channel_name": "/camera/color/camera_info"
      },
      "RosToImage": {
        "channel_name": "/camera/color/image_raw"
      }
    }
}

Now, deploy your application on your Jetson device from the development workstation:

bob@desktop:~/isaac$  ./engine/build/deploy.sh --remote_user <username_on_robot> -p //packages/ros_bridge/apps:ros_to_superpixel-pkg -d jetpack43 -h <robot_ip>

Run your application on your Jetson device and use your own app name:

bob@jetson:~/$ cd deploy/<bob>/ros_to_superpixel/
bob@jetson:~/deploy/<bob>/ros_to_superpixel$ packages/ros_bridge/apps/ros_to_superpixel

Finally, check the output from the Superpixels GEM using Isaac Sight (Figure 6).

Video 1. ROS bridge, ROS Image to Isaac proto.

Using the Isaac Global Localization GEM for the ROS navigation stack

The ROS navigation stack can benefit from using the Isaac SDK GPU-accelerated Global Localization GEM to solve the kidnapped robot problem.

The Isaac SDK 2020.1 release has an example to navigate ROS TurtleBot 3 Waffle Pi in Isaac Sim  Unity3D, which uses the Global Localization GEM. The following example is a full application with Isaac Sim and so on, but ROS users are free to use only the Global Localization GEM.

Revisit an example application from the Isaac release documentation in the context of this post. First, understand the different Isaac ROS bridge converters that are used for ROS topics. Figure 2 shows various Isaac ROS bridge converters are used for the ROS TurtleBot3 application with the Isaac Global Localization GEM. The ROS bridge converter used for Isaac Global Localization is PoseMessageToRos, which publishes the output /initialpose to the ROS TurtleBot3 application.

Figure 4. Isaac ROS bridges used in ROS Turtlebot3 reference app in Isaac SDK.

Here’s what the Isaac PoseMessageToRos converter does: receives the output of the Global Localization algorithm from within Isaac as PoseTreeEdgeProto::Reader and publishes the initialpose topic to ROS. The following code example shows this conversion.

bool PoseMessageToRos::protoToRos(PoseTreeEdgeProto::Reader reader, const ros::Time& ros_time

geometry_msgs::PoseWithCovarianceStamped& ros_message) {
  // Read data from Isaac type
  const Pose3d pose = FromProto(reader.getPose());
  const auto& translation = pose.translation;
  const auto& rotation = pose.rotation.quaternion();
  // Populate data for ROS type
  ros_message.header.stamp = ros_time;
  ros_message.header.frame_id = get_frame_id();
  ros_message.pose.pose.position.x = translation.x();
  ros_message.pose.pose.position.y = translation.y();
  ros_message.pose.pose.position.z = translation.z();
  ros_message.pose.pose.orientation.x = rotation.x();
  ros_message.pose.pose.orientation.y = rotation.y();
  ros_message.pose.pose.orientation.z = rotation.z();
  ros_message.pose.pose.orientation.w = rotation.w();
 if (get_report_success()) {
    reportSuccess();
  }
  return true;
}

Now, see the ROS bridge converter in action. Launch the small-warehouse scene by running the following command:

bob@desktop:~isaac_sim_unity3d/builds$ ./sample.x86_64 --scene small_warehouse --scenarioFile ~/isaac/packages/navsim/scenarios/turtlebot3_waffle_pi.json --scenario 0

Run the Isaac SDK application that communicates with both Isaac Sim Unity3D and ROS:

bob@desktop:~/isaac$ bazel run packages/ros_bridge/apps:ros_to_navigation_unity3d -- --more apps/assets/maps/virtual_small_warehouse.json --config ros_navigation:packages/ros_bridge/maps/small_warehouse_map_transformation.config.json,ros_navigation:packages/ros_bridge/apps/ros_to_navigation_turtlebot3_waffle_pi.config.json

Launch TurtleBot3 in ROS:

TURTLEBOT3_MODEL=waffle_pi roslaunch turtlebot3_navigation turtlebot3_navigation.launch map_file:=$(realpath packages/ros_bridge/maps/small_warehouse.yaml)

In the output, Isaac Sim shows a robot navigating to a goal based on ROS navigation, which also uses the Isaac Global Localization GEM. From the Isaac side, you can monitor the robot in Isaac Sight and in rviz from the ROS side. Change the goal for navigation by dragging the pose_as_goal marker of the Map window in Sight.

Video 2. ROS TurtleBot 3 app using Isaac SDK GEM.

Developing a custom ROS bridge converter

In Figure 3 and Figure 4, standard ROS imaging topics are getting converted into Isaac proto to run the Perception GEM on them. Isaac SDK has ROS bridge converters that convert between ROS messages and Isaac messages. NVIDIA also has a library of open source converters included as part of the Isaac SDK. However, you may need to create a converter someday.

The following code example shows converting the ROS custom_msgs::Custom1 message type to the Isaac Proto1 message type:

class RosToProto1 : public RosToProtoConverter<Proto1, custom_msgs::Custom1> {
 public:
  bool rosToProto(const custom_msgs::Custom1::ConstPtr& ros_message,
                    std::optional<ros::Time>& ros_time,  
                    alice::ProtoTx<Proto1>& tx_proto) override;
};

This is how a converter, which would convert Proto2 type from Isaac to ROS custom_msgs::Custom2 type would look.

class Proto2ToRos : public ProtoToRosConverter<Proto2, custom_msgs::Custom2> {
 public:
  bool protoToRos(const alice::ProtoRx<Proto2>& rx_proto, const ros::Time& ros_time,
                  custom_msgs::Custom2& ros_message) override;
}; 

Add the required ROS topic name as channel_name in an Isaac application JSON config file as follows:

      "RosToProto1": {
        "channel_name": "/custom_input",
        ...
      },
      "Proto2ToRos": {
        "channel_name": "/result",
        ...
      }

The isaac.ros_bridge.ImageToRos codelet receives ImageProto data within the Isaac application and publishes it to ROS. That is, it converts the data from ImageProto to sensor_msgs::Image.

Conclusion

This post showed how a ROS developer can collaborate with NVIDIA Isaac SDK. You learned how to selectively use Isaac GEMs, for example, a global localization, low latency, high fidelity perception algorithm in your existing ROS application, with your favorite solutions still in ROS. You also learned how to create your own new ROS-Isaac bridge converter.

We would love to hear your experience on ROS collaboration with Isaac SDK. Connect with us on the Isaac forum with your rich set of ROS bridge converters and feedback.

For more information, see the following resources: