Trimble Explores Acceleration of Autonomous Robot Training with Synthetic Data Generation and NVIDIA Isaac Sim

Deploying an autonomous robot to a new environment can be a tough proposition. How can you gain confidence that the robot’s perception capabilities are robust enough, so it performs safely and as planned?

Trimble faced this challenge when it started building plans to deploy Boston Dynamics’ Spot in a variety of indoor settings and construction environments. Trimble needed to tune the machine learning (ML) models to the exact indoor environments so Spot could autonomously operate in these different indoor settings.

 “As we deploy Spot equipped with our data collection sensors and field control software to indoor environments, we need to develop a cost effective and reliable workflow to train our ML-based perception models,” said Aviad Almagor, Vice President of Emerging Technologies at Trimble.

“At the heart of this strategy is an ability to analyze synthetic environments. Using NVIDIA Isaac Sim on Omniverse we could seamlessly import different environments from CAD tools like Trimble SketchUp. Generating perfectly labeled ground truth synthetic data then becomes a straightforward exercise.”

Image consists of a modern office building viewed in SketchUp’s user interface.
Figure 1. Office building viewed in Trimble SketchUp, a 3D modeling application

To ensure that models work robustly, developers working on robotics and automation applications need diverse datasets that include all assets of the target environment. In case of indoors, the list might include assets such as partitions, staircases, doors, windows, and furniture.

While these datasets can be constructed manually with real photographers and human labelers, that approach requires much preplanning and high costs and often gates when your project can start. With synthetic data, you can bootstrap your ML training and get started immediately.

When building this dataset, you could choose to include segmentation data, depth data, or bounding boxes. This perfectly labeled ground truth data can open many doors of exploration. Some things like 3D bounding boxes can be easily obtained synthetically while they are notoriously difficult to label by hand.

In this post, we outline the steps taken to build a training workflow using synthetic data generated from simulation. Although this workflow includes sophisticated simulation and ML technology, the steps required to complete this project are simple:

    1. Import the environment from CAD to NVIDIA Omniverse platform.
    2. Build the synthetic dataset using NVIDIA Isaac Sim on Omniverse.
    3. Train the ML models using NVIDIA TAO Toolkit.

Importing the environment from Trimble SketchUp to NVIDIA Omniverse

In this project, the environment was available in Trimble SketchUp, a 3D modeling application for designing buildings. To import assets, NVIDIA Omniverse supports the USD format for scene description. The SketchUp model is converted to USD and imported using one of the Omniverse Connectors. For more information, see What is NVIDIA Omniverse Connect?

To ensure all the assets are properly imported, you must inspect the environment using NVIDIA Isaac Sim or the Create or View apps in Omniverse. In some cases, this process may require a few iterations until the environment is satisfactorily represented in Omniverse.

Image shows the process chart of importing a scene into Omniverse, evaluating the scene, and making changes until the scene is satisfactory and can be successfully used in NVIDIA Isaac Sim.
Figure 2. Process of importing a SketchUp Scene into Omniverse
Importing the scene from SketchUp into Omniverse.
Figure 3. Side-by-side view of the same office building (left) after importing to Omniverse and (right) original scene in Trimble’s SketchUp
Video 1. Importing the scene from SketchUp into Omniverse

Build the synthetic dataset using NVIDIA Isaac Sim

Synthetic data is an important tool in training ML models for computer vision applications, but collecting and labeling real data can be time consuming and cost-prohibitive. Moreover, collecting real training data for corner cases can be sometimes tricky or even impossible. For example, imagine training an autonomous vehicle to recognize and react properly to ensure safety of pedestrians crossing a busy street. It would be reckless and dangerous to set up a photoshoot in a crosswalk with live traffic.

As Trimble plans to deploy autonomous robots in different environments for different use cases, they faced a training data dilemma: How to safely get the right training datasets for these models in a reasonable timeframe and at a reasonable cost?

The built-in synthetic data generation capabilities of NVIDIA Isaac Sim directly address this challenge. For more information, see What is Omniverse Isaac Sim?

A key requirement for generating synthetic datasets is support of the right set of sensors for the ML models that are being deployed. As noted in the later example, NVIDIA Isaac Sim supports rendering images with bounding boxes, depth, and segmentation, which are all important for helping a robot perceive its surroundings. Additional sensors like lidar and ultrasonic sensors (USS) are also supported in NVIDIA Isaac Sim and can be useful in some robotic applications.

Image shows different boxes of synthetic data generated in NVIDIA Isaac Sim: the RGB, instance segmentation, and semantic segmentation images, as well as the 3D bounding
Figure 4. Synthetic data generated in NVIDIA Isaac Sim showing RGB, segmentation, and bounding boxes

The other superpower of generating synthetic data is domain randomization. Domain randomization varies the parameters that define a simulated scene, such as the lighting, color, and texture of materials in the scene. One of the main objectives is to enhance ML model training by exposing the neural network to a wide variety of domain parameters in simulation. This helps the model to generalize well when it encounters real world scenarios. In effect, this technique helps teach models what to ignore.

Randomizable parameters in NVIDIA Isaac Sim:

    • Color
    • Movement
    • Scale
    • Light
    • Texture
    • Material
    • Mesh
    • Visibility
    • Rotation
Video 2. Generating synthetic data from the imported scene
Image of an office environment where the chair and doors have been rendered with randomized parameters. The chairs are shown with random colors and the doors feature random textures.
Figure 5. Domain-randomized scene in NVIDIA Isaac Sim

Train the ML models using NVIDIA TAO Toolkit

Figure 6 shows pretrained models and proprietary data (real or synthetic) as inputs and a customized model as the output.

Image shows two inputs to TAO Toolkit: pre-trained model and proprietary data. The output is a customized model.
Figure 6. Functional block diagram of TAO Toolkit

Figure 7 shows that the simulator can be controlled from both ROS and Python. The outputs of the simulator include the digital twin and synthetic data, which can be used to train perception models.

Image shows three inputs to NVIDIA Isaac Sim: robot model, environment model, and 3D assets.
Figure 7. Functional block diagram of NVIDIA Isaac Sim on Omniverse

After the datasets are generated, formatting them properly to work with the NVIDIA TAO Toolkit enables you to greatly reduce the amount of time and expense of training the models while ensuring that the models are accurate and performant. The toolkit supports segmentation, classification, and object detection models.

The datasets that are synthetically generated in NVIDIA Isaac Sim are output in the KITTI format to be used seamlessly with the TAO toolkit. For more information about outputting data in NVIDIA Isaac Sim for training, see Offline Training with TLT.

When working with synthetic datasets compared to real data, you may want to iterate the dataset to get better results. Figure 8 shows this iterative process of training with synthetic datasets.

Process chart shows that, after the model performance is evaluated, new synthetic data is generated to improve model performance.
Figure 8. Iterative training to improve model performance


Trimble faced an all-too-common challenge of getting training data for its ML models for an autonomous robot in a cost-effective workflow. The solution to this challenge was to leverage the power of the connectors in NVIDIA Omniverse to import CAD data into USD efficiently. The data could then be brought into NVIDIA Isaac Sim.

In the simulator, the powerful synthetic data capabilities of Isaac Sim make generating the required datasets straightforward. You can provide synthetic data to enable a more efficient training workflow and safer autonomous robot operation.

Discuss (1)