DeepStream SDK

Build and deploy AI-powered Intelligent Video Analytics apps and services. DeepStream offers a multi-platform scalable framework with TLS security to deploy on the edge and connect to any cloud.

Get Started

There are billions of cameras and sensors worldwide, capturing an abundance of data that can be used to generate business insights, unlock process efficiencies and improve revenue streams. Whether it’s at a traffic intersection to reduce vehicle congestion, health and safety monitoring at hospitals, surveying retail aisles for better customer satisfaction, sports analytics or at a manufacturing facility to detect component defects- every application demands reliable, real-time Intelligent Video Analytics (IVA).

Powerful & Flexible SDK

A unified SDK suitable for a multitude of use-cases across a broad set of industries.

Real-time Insights

Understand rich and multimodal sensor data at the edge.

Managed AI services

Deploy AI services in cloud native containers and orchestrate using Kubernetes.

Reduced TCO

Train with Transfer Learning Toolkit and use DeepStream to increase stream density.

NVIDIA’s DeepStream SDK delivers a complete streaming analytics toolkit for AI-based multi-sensor processing, video, audio and image understanding.

DeepStream is for vision AI developers, software partners, startups and OEMs building IVA apps and services.

DeepStream is also an integral part of NVIDIA Metropolis, the platform for building end-to-end services and solutions that transform pixel and sensor data to actionable insights.

Achieving Higher Accuracy & Real-Time Performance Using DeepStream

DeepStream offers exceptional throughput for a wide variety of object detection, image classification and instance segmentation based AI models. To reduce development efforts and increase throughput, developers can use highly accurate pre-trained models from Transfer Learning Toolkit (TLT) and deploy with DeepStream. The following table shows the end-to-end application performance from data ingestion, decoding, image processing to inference. It takes multiple 1080p/30fps streams as input. Note that running on the DLAs for Jetson Xavier NX and Jetson AGX Xavier frees up GPU for other tasks.

					Jetson Nano*	Jetson TX2*	Jetson Xavier NX			Jetson AGX Xavier			T4	A100
Application	Models	Inference Resolution	Precision	Model Accuracy	GPU (FPS*)	GPU (FPS)	GPU (FPS)	DLA1 (FPS)	DLA2 (FPS)	GPU (FPS)	DLA1 (FPS)	DLA2 (FPS)	GPU (FPS)	GPU (FPS)
People Detect	PeopleNet-ResNet34	960x544	INT8	84%	11	28	168	54	54	292	70	70	890	3392
People Detect	PeopleNet-ResNet18	960x544	INT8	80%	14	35	218	72	72	395	97	97	1086	3841
Vehicle Detect	TrafficCamNet-ResNet18	960x544	INT8	84%	19	52	264	105	105	478	140	140	1358	4013
Vehicle Detect	DashCamNet-ResNet18	960x544	INT8	80%	18	46	254	100	100	453	133	133	1320	3993
Face Detect	FaceDetect-IR-ResNet18	384x240	INT8	96%	101	275	1192	553	553	2010	754	754	2568	5549
License Plate Recognition	TrafficCamNet LPDNet LPRNet NvDCF Tracker	960x544 640x480 96x48	MIXED	84% 98% 97%	8	23	72	-	-	147	-	-	459	1070

Tabulated data is in FPS with 1080p/30fps input. Batch size is equivalent to the number of input streams.which is total FPS from the table divide by 30.
* FP16 inference on Jetson Nano and Tx2

With DeepStream SDK you can apply AI to streaming video and can simultaneously optimize video decode/encode, image scaling and conversion and edge-to-cloud connectivity for complete end-to-end performance optimization. This plot summarizes stream density achieved at 1080p/30 FPS across various NVIDIA products. You can learn more about the performance using DeepStream in the documentation.

Learn more about performance best practices in this video tutorial.

Numbers generated using the DeepStream reference app

Why Use DeepStream SDK?

Seamless Development

Developers can build seamless streaming pipelines for AI-based video, audio and image analytics using DeepStream. DeepStream brings development flexibility by giving developers the option to develop in C/C++ or Python. DeepStream ships with various hardware accelerated plugins, see the full list below.

DeepStream is built for both- developers and enterprises and offers extensive AI model support for popular object detection and segmentation models such as state of the art SSD, YOLO, FasterRCNN, and MaskRCNN. You can also integrate OpenCV functions and libraries in DeepStream.

Deepstream offers the flexibility for rapid prototyping to full production level solutions and greater flexibility by allowing you to choose your inference path. With native integration to Triton Inference Server, you can deploy models in native frameworks such as PyTorch and TensorFlow for inference or achieve the best possible performance using NVIDIA TensorRT for high throughput inference with options for multi-GPU, multi-stream and batching support.

Managed IVA Apps & Services

For a real world IVA app/ service deployment, remote management and control of applications is critical. DeepStream SDK can run in any cloud and at the edge which makes it a powerful SDK to handle IoT requirements such as effective bi-directional messaging between edge and the cloud, security, smart recording and Over-the-Air AI model update.

With bi-directional messaging between edge and cloud, you can add greater control for use-cases such as remote triggers for event recording, change operating parameters and app configurations or request system logs.
The smart record feature in DeepStream app allows you to save valuable disk space on the edge with selective recording that enables faster searchability. You can use cloud-to-edge messaging to quickly trigger recording from the cloud.
Seamless Over-the-Air (OTA) update for the entire app or individual AI models from any cloud registry to continuously improve accuracy with zero downtime.
For secure IoT device communication, DeepStream provides two-way TLS authentication based on SSL certificates and encrypted communication based on public key authentication

DeepStream offers an IoT integration interface with Kafka, MQTT and AMQP and turnkey integration with AWS IoT and Microsoft Azure IoT.

You can build high performance DeepStream cloud native applications with NVIDIA NGC containers. By using DeepStream, you can deploy at scale and manage containerized apps with Kubernetes and Helm Charts.

Use bi-directional IoT messaging capability to trigger specific event recording using DeepStream

Powerful End-to-End AI Solutions

Speed up overall development efforts and unlock greater real-time performance by building an end-to-end vision AI system with NVIDIA Transfer Learning Toolkit (TLT), production quality vision AI models and deploying at the edge using DeepStream. DeepStream offers turnkey integration of several detection and segmentation models trained with TLT including SSD, MaskRCNN, YOLOv3, RetinaNet and more.

DeepStream SDK Plug-ins

H.264 and H.265 video decoding
Stream aggregation and batching
TensorRT-based inferencing for detection, classification and segmentation
Object tracking reference implementation

On-screen display API for highlighting objects and text overlay
Frame rendering from multi-source into a 2D grid array
Accelerated X11/EGL-based rendering
Filtering based on Region of Interest (ROI)

JPEG decoding
Scaling, format conversion, and rotation
Dewarping for 360-degree camera input
Metadata generation and encoding
Messaging to cloud
Audio/Video Template Plug-In

Testimonials

Improving operational efficiency and reducing loss are key issues facing many retailers. Today’s large supermarkets have numerous in-store cameras, which can be used to mitigate these problems, but real-time video processing of so many streams can be a challenge. By leveraging NVIDIA T4 GPUs, DeepStream and TensorRT, Malong’s state-of-the-art Intelligent Video Analytics (IVA) solution achieves 3X higher throughput with industry-leading accuracy to help their retail customers significantly improve their business performance.

Malong Technologies

Extracting actionable insights from a sea of data created by the world’s billions of cameras and sensors is a huge task, and maintaining a connection from these devices to the cloud for processing may be overly expensive or infeasible due to security, regulatory, or bandwidth restrictions. Microsoft Azure IoT Edge deploys applications and services built using DeepStream to edge devices, allowing organizations to process data locally to trigger alerts and take actions automatically and to upload to the cloud when needed. Combining Azure IoT Edge, NVIDIA DeepStream and Azure IoT Central brings device management, monitoring and custom business logic to millions of edge devices for real-time insights and easy deployment.

Microsoft

As a leader in fulfillment and logistics management, SF Technology needed to track goods and vehicles across tens of thousands of locations. Every site requires detailed analytics around fleet management, loading times, and other operational activities. Using DeepStream and NVIDIA GPUs, they were able to increase the efficiency of AI Argus; an intelligent video analytics product that brings smarter video insights and can process 32 video streams simultaneously. The company is also looking at using next-generation GPUs, which is expected to increase the number of video streams processed.

SF Technology

We are bringing AI and machine learning to the trade sector with a fleet of real-time analytics based products that help businesses secure the cash point area and carefully supervise store entry/exit to prevent loss of goods. By switching to a DeepStream-based solution running on Jetson Nano, we achieved 5X stream density increasing the platform efficiency, reducing hardware and installation costs.

Signatrix

General FAQ

Is DeepStream open source?

DeepStream is a closed source SDK. Note that source for all reference applications and several plugins are available.

What applications are deployable using the DeepStream SDK?

The DeepStream SDK can be used to build end-to-end AI-powered applications to analyze video and sensor data. Some popular use cases are: retail analytics, parking management, managing logistics, robotics, optical inspection and managing operations.

What platforms and OS are compatible with DeepStream?

See the Platforms and OS compatibility table.

Can I run my models natively in TensorFlow or PyTorch with DeepStream?

Yes, it is possible with the integration of Triton Inference server. Triton integration is an alpha feature and has few limitations for DeepStream SDK 5.0 developer preview. Triton supports TensorFlow, TensorFlow-TensorRT, PyTorch and ONNX on x86 and Tensorflow and TensorFlow-TensorRT on Jetson. More information can be found in the release notes.