Estimating Visual Odometry with Prerecorded Stereo Images and the NVIDIA Isaac SDK

The NVIDIA Isaac SDK includes a production-ready, real-time visual odometry solution. According to the KITTI visual odometry algorithm comparison benchmark, it is 11th for accuracy and the fastest for performance.

Chart of various solutions includes IsaacElbrus highlighted with .0023 deg/m and .0095 s on an AGX Jetson Xavier (0.03s Jetson Nano).
Figure 1. Performance of Isaac SDK visual odometry solution, #32 IsaacElbrus.

Figure 2 shows that Isaac provides better speed and accuracy than the widely used ORB Slam2.

Four charts each compare translation errors and rotation errors for Isaac and ORB-SLAM2 on path length and speed.
Figure 2. Performance metrics for both Isaac and ORB-SLAM2 Left: Isaac VO. Right: ORB-SLAM2.

This post shows you how to run Isaac SDK visual odometry with a prerecorded sequence of stereo images from KITTI. The following video shows using KITTI Vision Benchmark Suite to compare computer vision algorithms.

Video. Example of Isaac Sight in use on car traversing city streets.

KITTI Vision Benchmark Suite

The KITTI Vision Benchmark Suite is a high-quality dataset to benchmark and compare various computer vision algorithms. Among other options, the KITTI dataset has sequences for evaluating stereo visual odometry.

To get the KITTI test sequences, download the odometry data set (grayscale, 22 GB). After registering, you receive a download link to the file. When you unpack it, you get the following structure:


Each of the 20 folders has a stereo sequence and calibration of cameras.

Running Isaac visual odometry with a KITTI sequence

To get started with this solution, download the Isaac SDK. Install the SDK using the provided instructions and create the subfolder visual_odometry under the isaac/apps/tutorials folder.

Write a KittiLoader codelet

First, you must implement a codelet to import the stereo images from disk. Create the KittiLoader.hpp file:

#pragma once
#include "engine/alice/alice_codelet.hpp"
#include "messages/camera.capnp.h"
namespace isaac {
class KittiLoader : public alice::Codelet {
 void start() override;
 void tick() override;
 ISAAC_PROTO_TX(ImageProto, left_image);
 ISAAC_PROTO_TX(ImageProto, right_image);
 ISAAC_PROTO_TX(CameraIntrinsicsProto, left_intrinsics);
 ISAAC_PROTO_TX(CameraIntrinsicsProto, right_intrinsics);
 int frame_id_ = 1;
}  // namespace isaac

Also, create its implementation, KittiLoader.cpp:

#include "KittiLoader.hpp"
#include <iomanip>
#include <string>
#include <utility>
#include "engine/core/image/image.hpp"
#include "messages/camera.hpp"
namespace isaac {
std::string KittiFileName(int camera_id, int frame_id) {
 std::stringstream s;
 s << camera_id << "/" << std::setw(6) << std::setfill('0') << frame_id << ".png";
 return "/path/to/dataset/dataset_odometry_gray/dataset/00/image_" + s.str();
void FillIntrinsics(auto& tx_intrinsics) {
 auto camera = tx_intrinsics.initProto();
 auto pinhole = camera.initPinhole();
 auto distortion = camera.initDistortion();
 ToProto(Vector2d{718.855999999, 718.855999999}, pinhole.getFocal());
 ToProto(Vector2d{185.2157, 607.192799999}, pinhole.getCenter());
 ToProto(VectorXd::Constant(4, 0), distortion.getCoefficients());
void KittiLoader::start() {
 Pose3d extrinsics = Pose3d::Identity();
 extrinsics.translation.x() = 0.1 * 0.5371657188644179;
 node()->pose().set("left_camera", "right_camera", extrinsics, 0);
void KittiLoader::tick() {
 std::array<Image1ub, 2> images;
 for (int camera_id = 0; camera_id < 2; ++camera_id) {
   const std::string filename = KittiFileName(camera_id, frame_id_);
   if (!LoadImage(filename, images[camera_id])) {
     reportFailure("Couldn't load image from file '%s'", filename.c_str());
 show("left_image", [&](sight::Sop& sop) {
 ToProto(std::move(images[0]), tx_left_image().initProto(), tx_left_image().buffers());
 ToProto(std::move(images[1]), tx_right_image().initProto(), tx_right_image().buffers());
 const int64_t acqtime = getTickTimestamp();
}  // namespace isaac

Create a test app

Figure 3 shows the KITTI workflow and objects created.

Figure 3. KITTI workflow.

To make a test application, create a file called

 "name": "kitti_run",
 "modules": [
 "graph": {
   "nodes": [
       "name": "kitti",
       "components": [
         {"name": "MessageLedger", "type": "isaac::alice::MessageLedger"},
         {"name": "Loader"       , "type": "isaac::KittiLoader"}
       "name": "tracker",
       "components": [
         {"name": "MessageLedger", "type": "isaac::alice::MessageLedger"},
         {"name": "SVO"          , "type": "isaac::StereoVisualOdometry"}
       "name": "keypoint_viewer",
       "components": [
         {"name": "isaac.alice.MessageLedger", "type": "isaac::alice::MessageLedger"},
         {"name": "Viewer"                    ,"type": "isaac::viewers::ImageKeypointViewer"}
   "edges": [
     {"source": "kitti/Loader/left_image"      , "target": "tracker/SVO/left_image"},
     {"source": "kitti/Loader/right_image"     , "target": "tracker/SVO/right_image"},
     {"source": "kitti/Loader/left_intrinsics" , "target": "tracker/SVO/left_intrinsics"},
     {"source": "kitti/Loader/right_intrinsics", "target": "tracker/SVO/right_intrinsics"},
     {"source": "tracker/SVO/coordinates"      , "target": "keypoint_viewer/Viewer/coordinates"},
     {"source": "tracker/SVO/features"         , "target": "keypoint_viewer/Viewer/features"}
 "config": {
   "kitti": {
     "Loader": {"tick_period": "30hz"}
   "tracker": {
     "SVO": {
       "horizontal_stereo_camera": true,
       "process_imu_readings": false,
       "lhs_camera_frame": "left_camera",
       "rhs_camera_frame": "right_camera",
       "num_points": 10000
   "websight": {
     "WebsightServer": {
       "port": 3000,
       "ui_config": {
         "windows": {
           "Left image": {
             "renderer": "2d",
             "channels": [
               {"name": "kitti_run/kitti/Loader/left_image"},
               {"name": "kitti_run/keypoint_viewer/Viewer/keypoints"}
           "Camera Pose": {
             "renderer": "3d",
             "channels": [
               {"name": "kitti_run/tracker/SVO/current_pose"},
               {"name": "kitti_run/tracker/SVO/pose_trail"}

Create a build script

To build it all together, create a build script named BUILD that contains the following code:

load("//bzl:module.bzl", "isaac_cc_module", "isaac_app")
   name = "kitti_loader_components",
   srcs = ["KittiLoader.cpp"],
   hdrs = ["KittiLoader.hpp"],
   name = "kitti_run",
   modules = [

Your folder should look like the following:

~/isaac$ ls apps/tutorials/visual_odometry/
BUILD  KittiLoader.cpp  KittiLoader.hpp

View the results

In the Isaac root folder, run the following command:

bazel run //apps/tutorials/visual_odometry:kitti_run

To see the result in the Isaac websight tool, open in your browser (Figure 4).

Figure 4. Results of KITTI run in the browser.

Figure 5 shows the result:

Figure 5. Map of ground truth and visual odometry overlap.


In this post, I showed you how to run Isaac SDK visual odometry with a prerecorded sequence of stereo images from KITTI. For more information, see the following resources:

Discuss (3)