Technical Walkthrough

GPU-Accelerated Video Processing with NVIDIA In-Depth Support for Vulkan Video

Discuss (0)
Graphic of Vulkan Video architecture

Vulkan Video gives developers the choice of a powerful new API for accessing video processing acceleration. NVIDIA is expanding its commitment to Vulkan Video with tools and samples to help applications efficiently harness this significant new functionality. This post will help you discover whether Vulkan Video is right for your application—and if so, how to get started.

Why Vulkan Video?

The open standard Vulkan API from the Khronos Group enables applications with flexible, low-level GPU acceleration, including fine-grained control over scheduling, synchronization, and memory allocation. 

Vulkan Video seamlessly integrates hardware-accelerated stream compression and decompression with the full power of Vulkan. This enables developers to combine GPU rendering and compute acceleration with video processing within a single efficient run time. 

Vulkan Video opens new opportunities for applications such as inferencing and training for machine learning, video editing, intelligent video analytics, video transcoding, game and movie streaming, and even virtualized desktop experiences.

As an added bonus, Vulkan Video is the industry’s first cross-platform video acceleration API, enabling software portability across multiple operating systems and hardware vendors—a significant win for developers seeking to expand the market reach of their rich media applications and engines.

Flowchart of stages in a typical Vulkan Video decode and encode application: allocate memory using standard Vulkan objects; create a video session; create session parameter objects; record and submit video decode/encode commands.
Figure 1. Stages in a typical Vulkan Video decode and encode application

Vulkan Video evolution

NVIDIA has been a driving force behind the development of Vulkan Video at Khronos. Vulkan Video extension specifications for fully accelerated H.264 and H.265 decode were released in December 2022, and NVIDIA shipped beta drivers on the same day for both Windows and Linux.

LunarG has just shipped a new version of the open source Vulkan SDK, Vulkan SDK 1.3.239. This version integrates all the components necessary to use the Vulkan Video extensions, including support within the SDK validation layers, a vital step in streamlining Vulkan Video development for developers.

Vulkan Video was designed with flexible extensibility to accommodate a wide range of existing and future video formats by leveraging universal “core” extensions that, in turn, enable additional codec-specific decode and encode extensions. NVIDIA is committed to continuing collaboration with Khronos to expand Vulkan Video’s support for popular codecs, including encode for H.264 and HEVC, and full decode and encode support for AV1, with accelerated drivers to follow.

As the next step in this development process, all developers are invited to provide feedback on the Khronos draft Vulkan Video extensions for H.264 and HEVC encode listed below:

See the resources section at the end of the post for how to provide feedback.

Video Codec APIs at NVIDIA

NVIDIA has provided hardware-accelerated video processing on GPUs for over a decade through the NVIDIA Video Codec SDK. This is a comprehensive set of APIs, high-performance tools, samples, and documentation for hardware-accelerated video encode and decode on Windows and Linux. 

Video Codec SDK APIs provide high-level access to video acceleration, easing application development for many common use cases. The Video Codec SDK also exposes the very latest in NVIDIA video processing technology. For example, Video Codec SDK 12.0 will support split encoding across multiple encoders in NVIDIA Ada Lovelace architecture, enabling real-time 8k60 encoding for AV1 and HEVC formats.

NVIDIA also supports GPU-accelerated encode and decode through Microsoft’s DirectX Video, a cross-vendor API for Windows developers. In contrast to the NVIDIA Video Codec SDK, both DirectX Video and Vulkan Video are low-level APIs. While the Video Codec SDK provides automation for C++ developers, DirectX Video and Vulkan Video enable granular tuning of how data streams are routed through, and processed by, video acceleration hardware. 

Whether you prefer DirectX or Vulkan, you can combine flexible GPU-accelerated video encoding and decoding with other GPU acceleration, like 3D and AI, using the language of your choice.

The low-level Vulkan Video extensions are also attractive to developers of popular open-source streaming media frameworks such as GStreamer and FFmpeg, both of which are being actively ported to Vulkan Video. The cross-platform availability of Vulkan will enable accelerated GPU processing for these frameworks across multiple platforms without needing to port to multiple proprietary video APIs.

Video Codec SDK, DirectX Video, and Vulkan Video are providing complementary support to GPU-accelerated video workflows. NVIDIA will continue to support all three, providing developers with the option to use the APIs that best suit their needs.

Vulkan Video Support at NVIDIA

NVIDIA Vulkan 1.3 drivers with Vulkan Video support are expected to leave beta and ship in general release drivers in February 2023. And NVIDIA support for Vulkan Video extends beyond optimized drivers to include tooling and sample code to help developers efficiently leverage Vulkan Video to its full potential.

NVIDIA Nsight Developer Tools are a collection of debuggers, profilers, and optimizers that support performance tuning for applications using many graphics APIs, including Vulkan. With version 2022.7 expected in February 2023, the NVIDIA Nsight Graphics frame debugger will explicitly support Vulkan Video, enabling developers to scrub through all events contributing to a decoded video frame and browse the corresponding resource allocations.

NVIDIA Nsight Graphics frame debugger profiling Vulkan Video decode screenshot.
Figure 2. NVIDIA Nsight Graphics frame debugger profiling Vulkan Video decode

Additionally, the NVIDIA Nsight Systems system profiler will enable parsing of the specific workload of a Vulkan Video decoding queue, providing insights into processing bottlenecks within the context of the application.

Screenshot showing how the Vulkan Video workload can be identified in the NVIDIA Nsight System timeline below the Vulkan tab. 
Figure 3. The Vulkan Video workload can be identified in the NVIDIA Nsight System timeline below the Vulkan tab 

The Vulkan Video extensions provide powerful low-level flexibility, and there can be a learning curve for developers. To help applications quickly leverage Vulkan Video, NVIDIA has added Vulkan Video video decode and encode example applications to a library of open source samples, showcasing how the extensions interact efficiently with graphics and compute queues for video decode. 

Visit nvpro-samples/vk_video_samples on GitHub to see an example that encapsulates how to parse a video stream from an input file, employ hardware accelerated decoding, and pass the decoded stream frame-by-frame into graphics processing and presentation—all handled by the Vulkan API.

These Vulkan Video example applications can be used as customizable libraries, enabling developers to use a higher-level abstraction for coding productivity, while still leveraging Vulkan Video low-level functionality directly when needed.

Vulkan Video resources

If you are interested in learning more about Vulkan Video, see An Introduction to Vulkan Video and Khronos Finalizes Vulkan Video Extensions for Accelerated H.264 and H.265 Decode. You can also download the newly-released LunarG Vulkan SDK with Vulkan Video support.

Looking to get started with profiling Vulkan Video-based applications? Check out Optimizing Vulkan 1.3 Applications with NVIDIA Nsight Graphics and NVIDIA Nsight Systems and NVIDIA Nsight Systems – Vulkan Trace. As an extension of the Vulkan API, profiling Vulkan Video aligns with these best practices. You can also learn more about Vulkan at NVIDIA. 

Finally, Tony Zlatinski of NVIDIA will deliver a live presentation and demo on setting up and using Vulkan Video at the Khronos Group’s Vulkanized 2023 event in Munich, February 7–9. 

We value your feedback as we continue to evolve Vulkan Video. We encourage you to take the Vulkan Video Developer Survey to provide input on features you would like to see in Vulkan Video. You can also reach out to us at the NVIDIA Vulkan Developer Forum or the NVIDIA Vulkan Developer Discord Channel.

Khronos and Vulkan are registered trademarks of the Khronos Group, Inc.