NVIDIA released a new NVIDIA Omniverse Experience for Robotics for NVIDIA Isaac Sim to help support the accelerating need for accurate, reliable, easy-to-use simulation tools in robotics. With many research labs and universities closed indefinitely, roboticists around the world are separated from the physical hardware and environments that they need for developing, testing, and deploying autonomous machines. Now more than ever, better simulation tools can help keep vital efforts in robotics and automation moving forward.
This Robotics Experience presents the NVIDIA Isaac Sim simulation toolkit within Omniverse Kit. Building Isaac Sim into the Omniverse platform brings several useful improvements over existing robotics workflows:
- It takes advantage of Omniverse Kit’s highly accurate physics simulation, bringing direct integration with industry-leading physics frameworks, such as the NVIDIA PhysX SDK for rigid body dynamics and NVIDIA Flow for fluid dynamics.
- It has a renewed focus on interoperability, with deep integration with the NVIDIA Isaac SDK and extensions for ROS.
- Omniverse Experience for Isaac Sim is readily extensible. You can now use Omniverse Kit’s Python-based scripting interface and extension system to tailor Isaac Sim to your own use cases.
- It is built to be deployable, with an architecture that supports workflows on your local workstation, at the edge with the NVIDIA Jetson family of hardware, and through the cloud with NVIDIA NGC.
Here’s a closer look at the previews and example code included within this release.
Bridging workflows
The Omniverse Isaac Sim simulation environment supports two prominent robotics frameworks: NVIDIA Isaac SDK and ROS.
Isaac SDK is an enterprise-grade framework for developing robotics AI, navigation, and manipulation applications. Isaac Sim communicates to Isaac SDK using the Robot Engine Bridge (REB) extension to send simulated sensor data, drive commands, ground truths, and scenario management over TCP/IP. The Isaac SDK docs highlight a range of simulation scenarios, from complex multi-robot autonomous warehouse navigation to simple manipulation with Jupyter notebooks. Moreover, Isaac SDK with Isaac Sim bridges virtual and physical robots for several Jetson-based reference platforms, like the Carter and Kaya robots.
Omniverse Isaac Sim also supports ROS, a popular framework for roboticists. The Isaac Sim ROS Bridge extension enables robotics researchers to augment their existing workflows with the Omniverse Isaac Sim powerful, multi-GPU, real-time simulation and collaboration platform. This release provides a rostopic sample, which walks through how to publish and subscribe to topics, including Joint State, Joint Command, TF, Camera, and Lidar topics.
Agile automation
Another advantage of Isaac Sim in the Omniverse is that complex rigid body dynamics are now easy to simulate for robotics and automation scenarios. The UR10 sample applications illustrate bin filling and bin stacking procedures using Universal Robot’s UR10 robot arm and a suction-based gripper. In the bin-filling example, a user drops different parts into a bin that a robot is holding until the bin is so heavy that the suction gripper fails. Parts fall everywhere. The robot arm is then able to detect the fallen bin, pick it up, and position it to receive new parts. While bin filling and stacking are standard procedures, these examples show how they can be intelligently augmented to make robotic automation more agile and adaptive to dynamic environments.
The UR10 sample applications come with highly detailed source code and documentation. They showcase several core simulation components:
- Loading a robot model using the URDF importer
- Creating a state machine for lean automation
- Combining traditional path planning with advanced motion generation using Riemannian motion policies (RMPs)
- Simulating sensors on and off the robot.
Review this extension to learn how to build end-to-end simulations for your own robotics applications.
The robot arm about to pick up the dropped bin. 
The robot arm stacking empty bin in a warehouse.
Custom robots
In addition to the UR10 robot, Omniverse Isaac Sim has an asset library of robots, end effectors, and environments ready for simulation. However, you can also work with your own custom robots using the Isaac Sim URDF importer extension. This extension imports and converts a URDF file to the Omniverse USD representation, making it a more usable and persistent format for advanced simulation. The URDF importer extension also comes with several sample imports and a Python API for loading and configuring your own robots.
In this release, the documentation has a Kaya Sample that walks you through the process of importing and configuring a custom robot for Omniverse Isaac Sim. It features NVIDIA Kaya as its reference hardware. Kaya is a 3D printed robot powered by NVIDIA Jetson Nano. It is designed to make advanced robotics accessible for hobbyists and education. However, it is a particularly challenging reference platform to simulate accurately. It has several CAD files, a complex holonomic drive, and many sensors that must be configured for simulating its 3D mapping, localization, object recognition, and odometry capabilities. You can review the documentation about creating a physically accurate virtual Kaya to learn how to go from a CAD or URDF file to a ready-to-simulate robot.
Sensing the world
In this Omniverse Robotics Experience, you can preview the benefits of real-time raytracing for robotics simulation. Beyond realistic visuals, the Omniverse real-time raytracing simulates the physical behavior of light. This enables Isaac Sim to accurately model the physical properties of several light-based sensors, including RGB cameras, depth cameras, and a LIDAR. The LIDAR sample shows how to leverage the Isaac Sim LIDAR API to create and activate these sensors for training autonomous robots. Set the resolution, field of view, and rotation rate to match the hardware properties of LIDAR used for a physical robot.
Get simulating
This latest release of the new Omniverse Isaac Sim experience showcases how a modern robotics simulation toolkit can benefit researchers and developers. Building the toolkit on top of the NVIDIA Omniverse platform enables new workflows for robotics simulation, and is a major upgrade in usability, extensibility, and interoperability. For more information about development details, see the Isaac Sim: Omniverse Experience for Robotics documentation. To try the new Omniverse Robotics Experience, sign up through the NVIDIA Early Access portal.
