Automating Smart Pick-and-Place with Intrinsic Flowstate and NVIDIA Isaac Manipulator

Picture of cardboard in bin.

We are announcing our collaboration with on learning foundation skill models for industrial robotics tasks.

Many pick-and-place problems in industrial manufacturing are still completed by human operators as it is still challenging to program robots for these tasks. For instance, in a machine-tending setting, a collaborative robot could be used to pick raw material parts from a bin and feed them into a CNC or bending machine, as part of a complex multi-step part-fabrication process. 

Such robots can be programmed with foundation models, which have the potential to generalize to significant variability in terms of object parts, robot embodiments, and real-world industrial environments.

We present a workflow using NVIDIA Isaac Manipulator to generate grasp poses and robot motions, evaluating them first in an NVIDIA Isaac Sim simulation and then executing them in the real world with Intrinsic Flowstate

Perception is also done through Flowstate, where we use the object pose estimation package to get the positions and orientations of graspable objects in the cluttered scene.

For this post, we demonstrate the system on a challenging smart pick-and-place application: a robot grasping metallic parts in a cluttered bin and singulating them at precise placement poses. These thin metal sheet objects are difficult to perceive because commodity depth cameras struggle with specular surfaces. They are also difficult to manipulate because suction grippers rather than fingers, are needed for grasping. 

Isaac Manipulator for grasp and motion generation 

To generate synthetic data for vacuum grasping, we used CAD models of sheet metals and the suction gripper. In simulation, we can try out thousands of grasps for each object and find the best ones. 

Good suction grasps need to be sealed and avoid the holes that are present in the metal sheets. We also want to avoid any wobble of the metal sheet during transport which we also ensure in simulation.

GIF of grasp poses on Isaac Sim, on multiple grippers
Figure 1. Examples of grasps in Isaac Sim

After we generated a dense set of grasp poses, we transformed them to the robot frame using the object pose information. We used NVIDIA cuMotion (powered by cuRobo), a CUDA-accelerated motion generation library, to generate a collision-free trajectory to move the robot to achieve one of these target grasp poses. 

Some of these poses may not be kinematically reachable or may be in collision, both of which are filtered by cuRobo while planning the trajectory. For reference, cuRobo can generate motion plans within 30 ms on an NVIDIA RTX 4090.

Evaluation in Isaac Sim

We first evaluated our pipeline in Isaac Sim. The simulated robot is derived from the CAD file of the custom Kuka K10 manipulator arm equipped with a suction gripper. Objects are randomly positioned to simulate a real-world bin-picking scenario. 

Instead of an object detector, the object pose information is acquired directly from the simulator. We leveraged ground truth information about the object type and determined the optimal reachable grasp given the object’s initial pose. 

Video 1. Smart Pick-and-Place in NVIDIA Isaac Sim

Real-world execution with Intrinsic Flowstate

Our workflow directly transferred from simulation to the real-world workcell. The real hardware setup consists of an extrinsically calibrated overhead multi-camera system and a Kuka K10 manipulator arm equipped with a suction gripper. 

We used the native object pose estimation package in Intrinsic Flowstate to detect the positions of the sheet metal objects. We then used Isaac Manipulator to compute the grasp positions for the objects and plan a collision-free trajectory for execution. 

Finally, we executed this robot trajectory on the robot arm through the position controller in Flowstate. Overall, we achieved a cycle time of around 8s/pick in the demo.

Video 2. Grasping skill, based on an NVIDIA foundation model, used in Intrinsic Flowstate. 

Looking ahead

We plan to expand our framework to more advanced forms of smart pick and place such as for machine tending. In this example, the robot would have to place the grasped object precisely at a location or fixture for downstream processing by a machine. 

For more information, see Isaac Manipulator and the announcement from Intrinsic, Unlocking new value in industrial automation with AI.

Video 3. Pick-and-Place for Machine Tending in NVIDIA Isaac Sim
Discuss (0)