Vision Programming Interface (VPI)

VPI is a computer vision and image processing software library from NVIDIA that implements algorithms on computing engines, including central processing units (CPUs), graphics processing units (GPUs), programmable vision accelerator (PVA), Video and Image Compositor (VIC), and Optical Flow Accelerator (OFA). VPI optimizes the algorithms on both NVIDIA Jetson™ modules and x86 devices with discrete GPUs and features an interface that lets developers access multiple computing engines to achieve high image throughput and easily interoperate with OpenCV.

VPI Computer Vision and Image Processing Use Cases for Software Developers

Use VPI to solve the wide range of challenges developers face when working with Jetson embedded devices.

Pyramidal Lucas-Kanade (LK) Optical Flow estimates the two dimensional (2D) translation of sparse feature points from a previous frame to the next on the Golden Gate Bridge.
Kanade-Lucas-Tomasi (KLT) Feature Tracker

Maximum Computing Performance

Software developers need to maximize computing performance when building and delivering computer vision systems, particularly as computer vision and image processing pipelines are becoming increasingly complex and time-consuming. If computer vision and image processing software developers aren’t hitting your target or desired frames per second (FPS) rate, they should consider VPI for highly optimized algorithms to increase performance, especially to replace non-performant OpenCV algorithms in the processing pipeline. Optimized VPI algorithms include background subtraction, perspective warp, temporal noise reduction, histogram equalization, and lens distortion.

Better Workload Distribution of Computer Vision and Image Processing Pipelines

Coding computer vision pipelines with multiple hardware backends is critical to take full advantage of the device’s compute capacity. VPI lets you experiment with different hardware accelerators for one or multiple processing stages and efficiently program multiple compute engines—including CPU and GPU—through one uniform interface. VPI also provides a zero-copy mechanism to share memory buffers between supported backends, enabling software developers to optimally and efficiently distribute workload across multiple compute engines.

Python Application Programming Interface (API)

VPI gives software developers the flexibility to develop computer vision and image processing pipelines. In addition to C support, it also offers Python bindings for the algorithms, which lets you use VPI directly in Python scripts.

VPI Performance Benchmarks

VPI’s computer vision and image processing algorithms are highly optimized. Its performance exceeds that of other well-known computer vision and image processing libraries, including OpenCV, to deliver typical speeds 11X and 7X faster than that of OpenCV on GPUs and CPUs respectively. Read more about VPI’s performance benchmarks.

VPI processing performance speeds up OpenCV performance on CPU and GPU.

VPI’s Computer Vision & Image Processing Algorithms

VPI’s library supports algorithms in image processing, disparity estimation, and feature detector and tracking:

  • Gaussian Pyramid Generator
  • Laplacian Pyramid Generator
  • Separable and Direct Image Convolution
  • Box Image Filter
  • Gaussian Image Filter
  • Bilateral Image Filter
  • Image Rescaling
  • Image Flip
  • Image Views / Crop
  • Image Remapping
  • Image Histogram
  • Image Histogram Equalization
  • Erode and Dilate
  • Minimum/Maximum Location
  • Direct and Inverse Fast Fourier Transform
  • Image Format Converter
  • Perspective Warp
  • Background Subtraction
  • Lens Distortion Correction
  • Temporal Noise Reduction
  • Pyramidal LK Optical Flow
  • Dense Optical Flow






Background Subtractor separates foreground objects from background image, like people in a scene Background Subtractor separates foreground objects from background image, like people in a scene
Background Subtractor
Stereo Disparity Estimator helps infer scene depth
Stereo Disparity
  • Stereo Disparity
  • KLT Bounding Box Tracker
  • Harris Corners Detector
  • ColorNames Features Detector
  • Histogram of Oriented Gradients
Harris Corner Detector helps developers identify keypoints and infer features of an image
KLT Bounding Box Tracker

VPI Computing Engine Support

VPI lets computer vision software developers use multiple compute engines simultaneously—including VIC, PVA, NVENC, and OFA—through one interface, so you don’t have to change or use multiple computer vision and image processing libraries. In fact, VPI requires little to no change to existing computer vision and image processing pipelines. The table below summarizes and compares VPI’s compute engine support to OpenCV.

VPI 2.1 Hardware Backend Support
Computer Vision & Image Processing Library
CPU
GPU
VIC
PVA
NVENC
OFA
VPI
X
X
X
X
X
X
OpenCV
X
X

VPI 2.1 Updates

VPI 2.1 is a developer release that includes PVA backend support for Gaussian Pyramid and KLT Feature Tracker as well as new algorithms for GPU and CPU backends.


The VPI 2.1 update is part of NVIDIA JetPack™ 5.0.2 for Jetson devices and is also available for x86 devices with discrete GPUs.

VPI 2.1 highlights include:

  • Unsigned/signed Int8 and signed Int16 support for KLT Feature Tracker on PVA backend
  • Unsigned Int8 support for Gaussian Pyramid on PVA backend
  • New Algorithms:
        • Fast Keypoint Detection on GPU and CPU backends
        • Image Channel Extraction on GPU and CPU backends
        • Median Filter on GPU and CPU backends

The VPI support matrix details algorithm-specific backend support.


Visit the JetPack download page

VPI Resources

Webinars

Introduction to VPI

The Implementing Computer Vision and Image Processing Solutions with VPI webinar overviews the computer vision and image processing software library. This tutorial reviews VPI programming concepts and best practices, how to build a complete and efficient stereo disparity-estimation pipeline, and VPI interoperability with OpenCV input and OpenGL output using computing platform and CUDA.

Watch the overview
(Registration required)

New Algorithms and Python Bindings Overview

The Accelerate Computer Vision and Image Processing using VPI 1.1 webinar (Registration Required) discusses the new algorithms and Python support included in VPI-1.1 as part of JetPack 4.6.

See the Python support
(Registration required)

VPI and PyTorch Interoperability Demo

The VPI and PyTorch Interoperability Demo (Registration Required) shows how to build a Python-based application to improve object detection using PyTorch without copying data.

Learn how to program with Python
(Registration required)

Blogs

Improved Interoperability between VPI and PyTorch

This blog demonstrates how VPI is interoperable with PyTorch and other Pytorch-based libraries. This post shares how to use a PyTorch-based object detection and tracking example on a noisy video.

Read how Interoperability works

Reducing Temporal Noise on Images with VPI

The Reducing Temporal Noise blog demonstrates how to build the VPI pipeline and run the Temporal Noise Reduction (TNR) sample application on Jetson devices, including how to submit and synchronize processing tasks in a video stream.

Learn to build the VPI pipeline

VPI References

Release Notes

Read details about the latest release highlights, new features, application programming interface (API) updates, known issues, and bug fixes.

Review the release notes  

Documentation

Learn more about the technical background and installation information, including architecture.

Review documentation  

Computer Vision Solutions

Learn more about computer vision technology, image processing solutions, computer vision machine learning, and deep learning models and the solutions NVIDIA provides.

Explore CV solutions  

Computer Vision Glossary

Learn more about the field of computer vision and applications of computer vision.

Learn Computer Vision