Build a Multi-Camera 3D Tracking Application with NVIDIA DeepStream 9.1 Skills

Developers building video analytics applications across large spaces must track the same object as it moves between camera views. Single-camera 2D tracking lacks reliable depth information and typically loses track of the object when it leaves the frame, limiting applications such as warehouse safety, retail analytics, and smart-building monitoring. Current 3D tracking methods require manual camera calibration and complicated calculations.  

NVIDIA DeepStream 9.1 addresses this challenge with AutoMagicCalib (AMC) and Multi-View 3D Tracking (MV3DT). AMC and MV3DT fuses detections from multiple auto-calibrated cameras into a shared 3D coordinate system and maintains a consistent object ID across views.

This post will cover the details of MV3DT and AMC, and how to get started with DeepStream 9.1, a major leap forward in simplifying and accelerating vision AI pipeline development. 

With a strong focus on modularity, automation, and edge performance, DeepStream 9.1 introduces a set of powerful agentic skills that help developers move from concept to deployment faster and with greater accuracy. 

What’s new with DeepStream 9.1:

• Access to 13 agentic skills designed to fast-track vision AI development.
• Multi-camera 3D tracking (MV3DT) skill for accurate, end-to-end object tracking across multiple camera streams.
• AutoMagicCalib (AMC) skill that automates camera calibration, reducing manual effort and minimizing errors.
• NVIDIA JetPack 7.2 support, enabling accelerated vision AI performance on Jetson edge platforms such as Orin and Thor.
• Open-source availability through a unified GitHub repository, making adoption, customization, and maintenance simpler.

Available now via the NVIDIA DeepStream GitHub repository, DeepStream 9.1 includes comprehensive source code and reference applications. 

How MV3DT tracks objects across cameras

MV3DT projects detections from multiple calibrated cameras into a shared 3D coordinate system. It then associates observations of the same object across camera views and assigns one globally consistent object ID. Each camera independently projects objects into 3D, and the system fuses these inputs to ensure globally consistent tracking. 

This ensures that every object maintains a unique, stable ID across the entire environment, with positions calculated in a unified world-coordinate system.

The underlying multi-view association algorithms and 3D fusion techniques are detailed in the research paper, Fully Distributed Multi-View 3D Tracking in Real-Time.

Understanding the MV3DT architecture

MV3DT extends DeepStream tracker with support for distributed multi-view 3D tracking across a network of calibrated cameras. 

Here is how data flows through the system:

• Detection Capabilities: DeepStream pipeline processes all camera streams, with the MV3DT tracker independently detecting and tracking objects in each view. It supports three detector models out of the box:
  • PeopleNetTransformer: Transformer-based people detector, default for pedestrian scenes.
  • PeopleNet v2.6.3: A high-efficiency detector based on the DetectNet_v2 architecture.
  • RT-DETR 2D: Multi-class detector, ideal for industrial environments, detecting pedestrians, transporters, and forklifts.

• Monocular 3D Perception: Each camera uses a 3×4 projection matrix (stored in a YAML calibration file) to back-project 2D bounding-box detections into 3D world-space coordinates using a ground-plane assumption.

• Multi-View Association: The tracker uses Message Queuing Telemetry Transport (MQTT),  a lightweight pub/sub messaging protocol that shares tracklets across cameras. When two cameras observe the same person, the multi-view association algorithm matches their tracklets using proximity in 3D world space and assigns a single globally consistent object ID.

• Output: Tracking results stream out in three forms:
  • On-Screen Display (OSD): Provides a live grid of camera feeds with overlaid 2D/3D bounding boxes and shared IDs.
  • Bird’s-Eye View (BEV): Offers a real-time 2D top-down map showing object trajectories in world coordinates, rendered over a layout image.
  • Kafka Messaging: Delivers structured protobuf metadata per frame such as sensor ID, object ID, and 3D bounding box for downstream applications.

How to calibrate a camera network with AutoMagicCalib

MV3DT requires calibrated cameras that accurately map image pixels to world coordinates on a 3×4 projection matrix. Traditional approaches are manual and time-consuming, often requiring the placement of calibration patterns (like checkerboards) in front of the cameras, which means interrupting operations and placing special equipment in the space.

AMC simplifies and automates the camera network calibration process by using DeepStream to analyze tracked objects moving across existing video files or streams. 

AMC automatically estimates each camera’s intrinsic (focal length, principal point, lens distortion) and extrinsic (rotation, translation, world position) parameters for each camera to produce calibration files for applications such as MV3DT. AMC can optionally use Visual Geometry Grounded Transformer (VGGT), a model-based approach that leverages learned models for higher accuracy and robustness when object movement is limited. 

The internal calibration pipeline runs through these stages:

1. Per-camera trajectory extraction: DeepStream detects and tracks objects in each video. AMC collects the resulting trajectory data.
2. Single-view calibration and rectification: For each camera independently, AMC estimates intrinsic parameters (focal length, projection matrix, lens distortion) from the trajectories and produces a rectified view.
3. Multi-view tracklet matching: AMC matches object tracklets across cameras, using the manually provided alignment points as an initial anchor between camera views and the layout map.
4. Bundle adjustment: All camera parameters are jointly refined across every view to minimize reprojection error globally.
5. Optional VGGT calibration: Users can optionally run VGGT, a separate model-based calibration workflow that can be useful when object movement is limited or when learned geometry provides more robust calibration results.

Users need only to provide a layout image and define a few alignment points by selecting corresponding landmarks in the camera views and on the map. AMC is available as a microservice featuring both REST APIs and a web interface.

Build and deploy applications with NVIDIA DeepStream agent skills

DeepStream 9.1 introduces modular skills that are designed for use by coding agents such as Claude Code, Codex or any other agent of your choice, including MV3DT and AMC. Instead of manually running scripts and editing configuration files, you describe what you want in plain natural language prompts and the agent handles setup, configuration, and execution. 

You can learn more about DeepStream skills and coding agents in the “How to Build Vision AI Pipelines Using NVIDIA DeepStream Skills and Coding Agents” blog post.

We will deploy multi-camera applications using the following skills:

• MV3DT Skill: This comprehensive skill manages the MV3DT deployment lifecycle, including:
  • Validating prerequisites (OS, GPU driver, Docker runtime)
  • Pulling or building the required DeepStream container
  • Installing Kafka and Mosquitto broker services
  • Downloading detection model weights (PeopleNetTransformer, RT-DETR)
  • Generating DeepStream pipeline configuration from your dataset
  • Automatically triggering the AMC skills if calibration files are not found
  • Launching the full multi-camera tracking pipeline

• AMC Skills: These skills handle the AMC lifecycle:
  • amc-setup-calibration-stack: Pulls the AMC microservice container and starts the service stack (web UI + REST API).
  • amc-run-sample-calibration: Runs end-to-end calibration on the bundled sample dataset; useful for verifying a fresh AMC install before using your own data.
  • amc-run-video-calibration: To help calibrate a new dataset of video files, along with a layout image, and alignment points to produce calibration YAML files ready for downstream applications like MV3DT.
  • amc-run-rtsp-calibration: To help calibrate a new dataset directly from RTSP streams.

The complete set of latest skills can be found here in the DeepStream GitHub repository.

The following workflow shows how a developer can move from repository setup to a running MV3DT application. The exact prompt can be adjusted for the coding agent and dataset being used.

Prerequisites

• Ubuntu 24.04 (x86_64)
• NVIDIA driver version 580 or higher
• Docker with NVIDIA Container Toolkit
• Claude Code or Codex installed and authenticated
• An NGC API key (to pull DeepStream and AMC containers from `nvcr.io`)
• Hugging Face key to pull VGGT model for AMC refinement step
• For display: X11 or VNC remote desktop (optional – headless mode saves output videos instead)

Let’s get started! 

Step 1: Clone the repository and install skills

git clone https://github.com/NVIDIA/DeepStream.git
cd DeepStream

Copy the skills into your coding agent’s skill directory. The path depends on which agent you use:

# Claude Code: ~/.claude/skills/
# Codex: ~/.codex/skills/
# Cursor: ~/.cursor/skills/

# Example for Codex (adjust path for your agent):
mkdir -p ~/.codex/skills

cp -r skills/* ~/.codex/skills/

Skills can also be installed at the workspace level (scoped to a single project). See the DeepStream skills README for full installation details and workspace-level instructions.

Step 2: Launch the coding agent

From the DeepStream repo root, start your coding agent:

claude
# or
codex

That’s all the setup needed. From here, you interact using natural language prompts.

Scenario A: Run MV3DT on the 12-camera sample dataset

The MV3DT skill includes 4-camera and 12-camera sample datasets with calibration already included, no AMC step required. For this example, we will be using the 12-camera dataset. Paste this sample prompt into the agent:

Sample prompt: deploy mv3dt on the 12-camera sample dataset

The agent will guide you through the following steps:

1. Check your system for display access (X11/VNC) and detect headless mode if no display is found.
2. Ask for your approval before running privileged Docker commands (`sudo xhost +` and `--privileged`).
3. Run the setup script to download models, start Kafka and Mosquitto services, and prepare the pipeline.
4. Launch the DeepStream container and start tracking. The first run may take a few minutes as it builds and loads the model engine.

If a display is available, two windows open:

• DeepStreamTest5App: A tiled grid of all 12 camera feeds with 2D and 3D bounding boxes
• Bird’s-Eye View of Multi-View 3D Tracking: A real-time trajectory map in world coordinates

Press `q` in either window to exit. The agent will remind you of this. In headless mode, the agent generates output videos (tiled_display_raw.mp4 and a BEV trajectory video) in the experiment directory. For sample outputs, see the 12-camera BEV trajectory GIFs in the MV3DT repository. 

experiments/deepstream/12cam/
├── config_deepstream.txt           # Generated pipeline config
├── config_tracker.yml              # MV3DT tracker config
├── outVideos/
│   └── tiled_display_raw.mp4      # Multi-camera grid with 2D and 3D overlays
└── bev_outputs/
    └── trajectory_video_<timestamp>.mp4   # BEV trajectory video

Scenario B: Run MV3DT on your own cameras (along with AMC calibration)

To use custom synchronized video streams without prior calibration, provide the directory path:

Sample prompt: deploy mv3dt on these videos ~/my-camera-dataset/videos

Ensure the folder contains time synchronized video files named sequentially by camera (e.g. cam_00.mp4, cam_01.mp4), and you should also have a layout BEV image (layout.png). The agent then performs these steps:

1. Validates the video source folder.
2. Detect that `camInfo/*.yml` calibration files are missing, and calibration is required.
3. Automatically invoke the AMC skills in order, first amc-setup-calibration-stack to bring up the AMC microservice, then amc-run-video-calibration to start calibration.

Before starting calibration, the agent will ask a few configuration questions. Here you may have to specify the detector type and calibration settings when prompted. 

> Agent: What detector type would you like to use for calibration? (resnet or transformer)
> User: Resnet

> Agent: Do you have a calibration settings file to upload?
> User: No, I will update on the UI

The AMC microservice starts and the agent provides the web UI address. Open it in your browser to complete the manual alignment step—alignment points between camera views and your layout map. Once saved, notify the agent:

> User: alignment is done

AMC runs the full calibration pipeline: trajectory extraction, single-view rectification, multi-view tracklet matching, and bundle adjustment. The agent polls for completion and, when done, downloads the MV3DT-compatible calibration export and places the YAML files into `~/my-camera-dataset/camInfo/`.

MV3DT then runs automatically on your dataset, using the generated calibration data.

A successful MV3DT deployment produces the following outputs:

• Live OSD window: All camera feeds in a tiled grid with 2D and 3D bounding boxes and consistent object IDs. Click any camera to zoom in; right-click to return to the grid.
• Live BEV window: Top-down trajectory mapping with screenshot and recording options.
• Kafka metadata stream: Per-frame protobuf messages on the `mv3dt` topic, containing object IDs, 3D bounding box coordinates, confidence scores, and sensor IDs. These are ready for downstream analytics, dashboards, or alert systems.
• Saved videos: With file output enabled, the system saves a tiled OSD video and a BEV trajectory video.

Getting started

All source code, skills, reference applications, and sample datasets for MV3DT and AMC are available in the NVIDIA DeepStream GitHub repository as part of the DeepStream 9.1 release:

• MV3DT reference app: `src/apps/reference_apps/deepstream-tracker-3d-multi-view`
• MV3DT skill: `skills/deepstream-run-mv3dt`
• AMC skills: `skills/amc-setup-calibration-stack`, `skills/amc-run-video-calibration`, `skills/amc-run-sample-calibration`, `skills/amc-run-rtsp-calibration`.

The prebuilt MV3DT microservice can also be used with NVIDIA VSS blueprint and agent skills to integrate the pipeline with additional microservices, databases and agents.

For questions and community discussions, visit the NVIDIA DeepStream Developer Forum.