NVIDIA Optical Flow SDK


NVIDIA Optical Flow SDK


Turing hardware generated optical flow map sample --- source footage

Optical Flow SDK exposes the latest hardware capability of Turing GPUs dedicated to computing the relative motion of pixels between images. The hardware uses sophisticated algorithms to yield highly accurate flow vectors, which are robust to frame-to-frame intensity variations, and track the true object motion. Computation of these flow vectors is faster than most other available methods at comparable accuracy, with very little load on CPU or GPU, as the vectors are computed on dedicated hardware which is independent of the GPU’s CUDA cores.


Key Features

NVIDIA Optical Flow SDK exposes a new set of APIs for this hardware functionality:

  • C-API – Windows and Linux
    • Windows – CUDA and DirectX
    • Linux – CUDA
  • Granularity: 4x4 vectors at ¼ pixel resolution
    • Raw vectors – directly from hardware
    • Pre-/post-processed vectors via algorithms to improve accuracy
  • Accuracy: low average EPE (End-Point-Error) and outliers
  • Performance: Up to 150 fps at 4K resolution*
  • Robust to intensity changes
  • OpenCV integration (GitHub)
  • NEW to 2.0: - Support for Ampere generation GPUs, with improved optical flow hardware engine, independent of NVENC
  • NEW to 2.0: - Increased accuracy in flow vector’s cost, which indicates the confidence of the vector
  • NEW to 2.0: - Ability to get flow vectors for a specific region of interest
  • NEW to 2.0: - Hardware optical-flow-assisted object tracker library
  • NEW to 2.0: - Support for 1x1 and 2x2 grid sizes
*Clock and preset dependent
Operating SystemWindows, Linux
Dependencies

NVIDIA GeForce, Quadro and Tesla products with Turing(except TU117) and Ampere generation GPUs

NVIDIA Windows display driver 445.87 or newer
NVIDIA Linux display driver 450.51 or newer

Development EnvironmentGCC 5.1 or newer
C/C++ Compiler
CUDA Toolkit

Download Optical Flow SDK 2.0.23

By clicking the "Agree & Download" button, you are confirming that you have read and agree to be bound by the SOFTWARE DEVELOPER KITS, SAMPLES AND TOOLS LICENSE AGREEMENT for use of the SDK package. The download will begin immediately after clicking on the "Agree & Download" button.


Windows / Linux
Agree & Download     Documentation       Get Source Code


Archive (Click to open)
Get Started

An Introduction to the NVIDIA Optical Flow SDK


Optical Flow - Typical Use-Cases



Optical Flow benefits use-cases such as

  • Object tracking for intelligent video analytics
  • Video frame interpolation, and extrapolation
  • Stereo depth estimation
  • Video action recognition

Object tracking for Intelligent Video Analytics


Until a few years ago, intelligent videos analytics (IVA) required physical viewing and manual analysis of the captured videos. With rapid developments in computer vision and deep learning techniques, most of such analysis is now automated. Identifying the objects/persons/vehicles of interest from security camera footage, and tracking them across the video frames can be achieved using computer vision algorithms. GPUs are a natural choice of platform for running these applications because of high computational complexity and parallelism requirements. The most common method is to detect and identify the objects/persons/vehicles of interest in every video frame and running an algorithm to correlate the objects in successive frames to track their movement. This method, although accurate, suffers from extremely high computational complexity requirements, since the object detection/identification is a computationally intensive operation (typically using neural network inference).

The hardware optical flow engine alleviates the computational complexity requirements significantly by helping the tracking of objects. For example, once an object is detected/identified by the detector, it can be tracked in successive video frames by an algorithm based on GPU’s hardware optical flow. This eliminates the need for running the detection algorithm every frame, resulting in a massive reduction in computational complexity. The detector can now run every Kth frame without compromising the accuracy (K > 1, typically K = 4). This leaves the GPU’s CUDA cores free to run inference on additional channels or other computationally intensive tasks.

Optical Flow SDK 2.0 includes an object tracker library based on optical flow, along with source code and ready-to-use API. In our experiments, the optical flow-based object tracker has shown to reduce the GPU utilization by up to 80%, compared to some of the most popular algorithms without compromising the accuracy of tracking.


Video Frame Interpolation and Extrapolation


Optical flow can also be used very effectively for interpolating or extrapolating the video frames in real-time. This can be useful in improving the smoothness of video playback, generating slow-motion videos or reducing the apparent latency in VR experience, as used by Oculus (details)


Optical Flow functionality in Turing and Ampere GPUs accelerates these use-cases by offloading the intensive flow vector computation to a dedicated hardware engine on the GPU silicon, thereby freeing up GPU and CPU cycles for other tasks. This functionality in hardware is independent of CUDA cores.


FAQ


Q: How do I choose the right driver version to install?

 A: After you download the SDK, please refer to the "ReadMe.txt" which lists the minimum required display driver version. You need to install the right drivers or else the SDK will fail to start and this is the first thing you should check in case there is an initialization failure.

Q: Is there a way to know if the feature is supported on the GPU I have?

 A: The client application should query the capabilities using the the capability API before enabling any feature.

Q: How do I write an efficient application and what are the right settings I should use?

 A: We strongly recommend all application developers to go through the programming guides in detail before writing any application. In particular, for some hints on this question, please go through the following section in the documentation:

  1. "Guidelines for Efficient Usage of NVOF API" in NVOF API Programming Guide, included in the Optical Flow SDK
Q: How do I measure performance?

 A: The Optical Flow SDK provides samples specifically designed to give optimal performance. You can measure performance using any of the samples.

Q: Since every release of Optical Flow SDK requires a minimum display driver version installed on the system, how would I write an application that will work across devices running on different display driver versions?

 A: NVOF API exposes APIs which allow users to query the maximum API versions supported by the underlying driver. Depending on the maximum API version supported by driver, the application can launch code at runtime compiled with the appropriate API.

Q: How do I find out how many streams of a specific resolution and frame rate (e.g. 1080p30) can a given GPU encode or decode?

 A: Please refer to the NVOFA application note included in the SDK documentation to get an idea about performance. The performance depends on many factors, some of which include: GPU in use and its clock speed, settings used(for e.g. preset, grid size and features like ROI), memory bandwidth available, application design.



Support


Our forum community is where Developers can ask questions, share experiences and participate in discussions with NVIDIA and other experts in the field.
Check out the forums here.



Additional Resources