NVIDIA DeepStream SDK
Analyze Data From Cameras, Sensors and IoT Gateways in Real-Time
NVIDIA’s DeepStream SDK delivers a complete streaming analytics toolkit for AI-based video and image understanding, as well as multi-sensor processing. DeepStream is an integral part of NVIDIA Metropolis, the platform for building end-to-end services and solutions for transforming pixels and sensor data to actionable insights.
Seamlessly Develop Complex Stream Processing Pipelines
The DeepStream application framework features hardware-accelerated building blocks, called plugins, that bring deep neural networks and other complex processing tasks into a stream processing pipeline. DeepStream enables real-time understanding of video and sensor data that is rich and multi-modal.
The SDK uses the open source GStreamer to deliver high throughput with a low-latency streaming framework. The runtime system is pipelined to enable deep learning capabilities, as well as image and sensor processing and fusion algorithms in a streaming application.
The SDK lets you integrate the edge to the cloud with standard message brokers like Kafka and MQTT for large-scale, wide-area deployments. This is ideal for applications like retail analytics, intelligent traffic control, automated optical inspection, freight and goods tracking, and more. The SDK also offers a complete set of reference applications and pre-trained neural networks to jumpstart development.
Why DeepStream?
Build your own application for smart cities, retail analytics, industrial inspection, logistics, and more.
Low latency and exceptional performance optimized for NVIDIA GPUs for real-time edge analytics.
Provides ready to use building blocks and IP simplify building your innovative product.
Pushbutton IoT solution integration to build applications and services with Cloud Service Providers.
Fast, flexible, and reliable containerized deployment and support for NVIDIA Tesla and Jetson platforms using NGC.
Iterate and integrate by quick plug and play of popular plug-ins that are pre-packaged or build your own.
Real-Time Performance with DeepStream
Achieve high throughput for applications involving object detection, tracking, and classification. With Deepstream you can build a common application and deploy it across NVIDIA platforms. Table shows the number of 1080p/30 FPS streams processed on various platforms.
| NVIDIA Products |
|
|
| Jetson Nano |
|
|
| Jetson TX1 |
|
|
| Jetson TX2 |
|
|
| Jetson AGX Xavier |
|
|
| T4 |
|
|
detector concurrently on eight independent 1080p 30fps video streams.
What's New in DeepStream 4.0.2
- A unified release across all our platforms will allow application portability. This includes support for Jetson Nano, our newest AI platform.
- 50% reduction in memory footprint resulting in exceptional stream processing density.
- Turnkey integration with Azure Edge IoT to build applications and services, leveraging the power of Azure cloud.
- Containerized deployment for Jetson platforms. This enhances the ability to deploy applications to Dockers, dramatically enhancing the delivery and maintenance of applications at large scale.
- Plugin sources for inference, message schema converter, and message broker plugins.
- A new reference tracker for robust object tracking.
- Added support for heterogeneous cameras, segmentation networks, monochrome images and hardware accelerated JPEG decode and encode.
DeepStream Plug-ins
DeepStream also offers the ability to build custom plugins for user-created libraries and functions. One of many open-source plugins can also be easily adapted for use with the DeepStream framework.
- H.264 and H.265 video decoding
- Stream aggregation and batching
- TensorRT-based inferencing for detection, classification and segmentation
- Object tracking reference implementation
- JPEG decoding
- On-screen display API for highlighting objects and text overlay
- Frame rendering from multi-source into a 2D grid array
- Accelerated X11/EGL-based rendering
- Scaling, format conversion, and rotation
- Dewarping for 360-degree camera input
- Metadata generation and encoding
- Messaging to cloud
- On-screen display API for highlighting objects and text overlay
- Frame rendering from multi-source into a 2D grid array
- Accelerated X11/EGL-based rendering
- Scaling, format conversion, and rotation
- Dewarping for 360-degree camera input
- Metadata generation and encoding
- Messaging to cloud
Testimonials
Improving operational efficiency and reducing loss are key issues facing many retailers. Today’s large supermarkets have numerous in-store cameras, which can be used to mitigate these problems, but real-time video processing of so many streams can be a challenge. By leveraging NVIDIA T4 GPUs, DeepStream and TensorRT, Malong’s state-of-the-art Intelligent Video Analytics (IVA) solution achieves 3X higher throughput with industry-leading accuracy to help their retail customers significantly improve their business performance.
Malong Technologies
Extracting actionable insights from a sea of data created by the world’s billions of cameras and sensors is a huge task, and maintaining a connection from these devices to the cloud for processing may be overly expensive or infeasible due to security, regulatory, or bandwidth restrictions. Microsoft Azure IoT Edge deploys applications and services built using DeepStream to edge devices, allowing organizations to process data locally to trigger alerts and take actions automatically and to upload to the cloud when needed. Combining Azure IoT Edge, NVIDIA DeepStream and Azure IoT Central brings device management, monitoring and custom business logic to millions of edge devices for real-time insights and easy deployment.
Microsoft
As a leader in fulfillment and logistics management, SF Technology needed to track goods and vehicles across tens of thousands of locations. Every site requires detailed analytics around fleet management, loading times, and other operational activities. Using DeepStream and NVIDIA GPUs, they were able to increase the efficiency of AI Argus; an intelligent video analytics product that brings smarter video insights and can process 32 video streams simultaneously. The company is also looking at using next-generation GPUs, which is expected to increase the number of video streams processed.
SF Technology
Start in Seconds, Scale Instantly with NGC
NGC is a container registry of GPU-optimized AI software. Applications built with the DeepStream SDK can be deployed on NVIDIA Tesla and Jetson platforms, enabling flexible system architectures and straightforward upgrades that greatly improve system manageability. The DeepStream SDK Docker containers with full reference applications are available on NGC.
End-to-End AI Workflow for IVA with Transfer Learning Toolkit
You can accelerate the development of efficient deep learning networks with the Transfer Learning Toolkit. The toolkit abstracts and accelerates deep learning training by allowing developers to fine-tune NVIDIA provided domain specific pre-trained models instead of going through the time-consuming process of building Deep Neural Networks (DNNs) from scratch. The pre-trained models accelerate the developer’s deep learning training process and eliminate higher costs associated with training models from scratch.
Transfer Learning Toolkit can also prune networks to tightly pack complex applications, delivering high throughput and stream density. When integrated with DeepStream, this offers an end-to-end deep learning solution for IVA.
Get Started With Hands-On Training
The NVIDIA Deep Learning Institute (DLI) offers hands-on training for developers, engineers and researchers in AI and accelerated computing. Get experience with the DeepStream SDK in a self-paced course or request a full day workshop focused on deep learning for IVA by contacting DLI directly.
DeepStream FAQ
The DeepStream SDK is a general purpose streaming analytics SDK that enables system software engineers and developers to build high performance intelligent video analytics applications using NVIDIA Jetson or NVIDIA Tesla platforms.
The DeepStream SDK uses the open-source GStreamer framework to deliver high throughput with low latency. GStreamer is a library for constructing graphs of media-handling components. You can build applications ranging from simple video streaming and playback to complex processing using AI.
To get started with GStreamer, please refer to the GStreamer Application Development Manual and a Step-By-Step Tutorial.
The DeepStream SDK can be used to build end-to-end AI-powered applications to analyze video and sensor data. Some popular use cases are: retail analytics, parking management, managing logistics, optical inspection and managing operations.
Yes, DeepStream SDK can be downloaded here.
To learn about the DeepStream SDK versions supported for various NVIDIA products, click here.
Yes, the DeepStream SDK ships with 10+ reference samples. For more information about reference designs, see here.
DeepStream developer guide can be found here.
For a technical FAQ, check out the DeepStream Plugin FAQ.
Developer News
Documentation
How-to Guides
- Using custom YOLO models in DeepStream 4.0.1
- Migration to DeepStream 4.0.1
- Calibration of a 360 degree camera to yield geo-coordinates
Blogs
- Building a Real-time Redaction App using NVIDIA DeepStream Part 1 | Part 2
- Build and Deploy Accurate Deep Learning Models for Intelligent Image and Video Analytics
- Breaking the Boundaries of Intelligent Video Analytics with DeepStream SDK
- Jetson Nano Brings AI Computing to Everyone
- Accelerating Intelligent Video Analytics using Transfer Learning Toolkit
- Multi-Camera Large-Scale Intelligent Video
- Accelerate Video Analytics Development with DeepStream SDK
Jetson community projects
- Check out Jetson community projects
Forum
- Post your questions or feedback in the DeepStream SDK forum.
Webinars
- DeepStream edge-to-cloud with Azure IoT
- DeepStream SDK- Accelerating Real-Time AI Based Video And Image Analytics
- DeepStream: An SDK to Improve Video Analytics
- Real-time Object Detection for Disaster Response