VRWorks 360 Video is NVIDIA’s SDK to enable VR developers and content creators to capture, stitch, and stream 360-degree videos.

The SDK supports 360-degree mono and stereo workflows, post production (offline mode) and real-time processing.


Hardware: Compatible with: Maxwell, Pascal and Volta based GPUs. (GeForce GTX 900 series and Quadro M5000 and higher)

Note: recommended spec for 360 stitching:
Mono: GeForce GTX 1060 6GB / Quadro P4000
Stereo offline: GeForce GTX 1080 / Quadro P4000
Stereo Live: Dual GeForce GTX 1080ti / Dual Quadro P6000
Software: 64 bit Windows
NVIDIA graphics driver 388.59 or later
Microsoft Visual Studio 2015 (MSVC14.0) or later
CUDA 9.0 Toolkit (NVCalib only)
CMake 3.2 or above

360-degree video processing is complex and computationally intensive. By leveraging the GPU, VRWorks 360 Video SDK provides a high-performance, high-quality and low-latency GPU-accelerated implementation that can be integrated into 360 video workflows.


Real-time stereo stitching is particularly challenging. The basic process generally involves ingesting, decoding and calibrating multiple high-resolution streams, stereo stitching, and encoding. The entire pipeline has to be executed in real time while maintaining the highest level of image quality. To accomplish that, NVIDIA has developed a new set of motion based algorithms for superior-quality stereo stitching, optimized for real time. Using consecutive frames, the algorithms estimate the motion of objects in a video stream, noting how they match and move across a seam while accounting for stereo disparity.


NVIDIA VRWorks 360 Video SDK

NVIDIA VRWorks 360 Video SDK consists of a library, set of APIs, sample applications and documentation.


NOTE: The earlier VRWorks 360 Video Beta SDK 0.9 is deprecated with the recent release of VRWorks 360 Video SDK 1.1. We strongly recommend moving to the latest version of the SDK.

Download VRWorks 360 Video SDK

Key Features

  • High performance mono and stereo stitching GPU accelerated and optimized
  • Automated calibration
  • Mono stitch supports general rig configuration
  • Stereo stitch supports rigs with up to 32 cameras in equatorial ring configuration
  • Stereo stitching performance scalability over multiple GPUs
  • Hardware-accelerated decode and encode
  • Output panorama(s) in equirectangular projection
  • Different input/output forms (buffers, files) are supported to cater to different use cases (offline or live)
  • Audio can be included in the input videos or ingested from an external source
  • Blended audio output can be muxed into the output panoramic video or output as a separate audio stream

360-degree video

360-degree videos have gained popularity due to their ability to encompass a wide field of view. The proliferation of virtual reality devices has given rise to an increasing need for videos with a 360 degree field of view to deliver an enhanced sense of immersion and presence.

360-degree videos can be generated by aligning streams and combining them appropriately in the areas of overlap to produce a seamless 360-degree view. These videos can be generated by using a rig that has sufficient number of cameras with a wide enough field of view to capture the 360-degree space.


Content provided by Kevin Alderweireldt of Yume Vr and features Umamido

360-degree video pipeline



Ingest

The VRWorks 360 Video SDK can ingest MP4 compressed videos. It can also read from RGB files as well as CUDA arrays. The SDK inherently supports HW accelerated decode of compressed input. VRWorks 360 Video SDK currently supports up to 32 inputs which should accommodate most 360 camera arrays. Audio can be included in the input videos or ingested from an external source. There is no restriction on the number of audio inputs being ingested.

Decode

NVIDIA GPUs contain one or more hardware-based decoder and encoder(s) (separate from the CUDA cores) which provides fully-accelerated hardware-based video decoding and encoding for most video formats. With decoding/encoding offloaded, the graphics engine and the CPU are free for other operations.

During import, video files are decompressed and decoded into a raw format for image processing. VRWorks 360 video SDK makes use of the integrated hardware and NVDEC (part of the NVIDIA Video Codec SDK)

Calibrate

During the calibration process, the individual video streams are calibrated in relation to each other. This SDK calibrates for lens distortion as well as the rotation and translation between each camera within the rig. Using intrinsic and extrinsic parameters particular to the rig, the individual streams are aligned and calibrated, then stitched into a single 360-degree video. The SDK also supports camera calibration without requiring input estimates for camera parameters. It automatically computes estimated focal length, rotation, principal point and fisheye radius if not provided. Non homogenous camera rigs are also supported given constraints related to overlap are satisfied. These rigs can be composed of cameras with different resolution, lens distortion type or focal length. This SDK also includes auto balancing for automatic equatorial alignment of the camera rig.

Stitch

The process of generating 360-degree videos typically includes capturing monocular footages that are then stitched into a single 360-degree video. In the case of monoscopic 360-degree video, the output is a single 360-degree panorama generated by aligning input videos and blending in the areas of overlap. Stereoscopic stitching produces a pair of 360-degree panoramas, one for each eye. This projection model is often referred to as Omnidirectional stereo. To avoid artifacts, this process requires careful handling of issues such as camera calibration, parallax in the source videos, and temporal consistency. Because of this, the stitching process is complex and computationally challenging.

VRWorks 360 Video SDK is a GPU-accelerated solution designed to achieve real-time stitching of images into stereo 360-degree videos. The performance of the SDK is GPU scalable allowing two or more Pascal based GPUs to produce a live stitched 4K 30fps stream from an 8 camera 4K 30fps input.


Click on image to enlarge

VRWorks 360 Video SDK offers real-time solutions for both monoscopic and stereoscopic 360 stitching. Both modes utilize hardware accelerated decoding and encoding to achieve real-time ingest and output of high resolution streams. Mono stitching uses GPU accelerated Multiband blending to avoid over-smoothing, distortion and visible seams. Stereo stitching is more computationally intensive and applies motion vector estimation to accomplish seamless stereoscopic output regardless of the captured scene.

The SDK supports both AAC and PCM audio and allows for user specified gain to be applied. Audio stereo spread can be optionally applied to the incoming streams during blending.

Encode

During export, video files are encoded and compressed into the desired delivery format. VRWorks 360 video SDK makes use of the integrated hardware and NVENC (part of the NVIDIA Video Codec SDK)

Output

The VRWorks 360 Video SDK supports various output formats: RGB textures, H.264 compressed streams, MP4 files. Compression is HW accelerated. Equirectangular projection is supported. The SDK allows the user the option of multiplexing the audio and video into an MP4 or outputting audio and video as separate files

Hardware Partners

Z CAM

Z CAM is a product line created by a team of innovative and creative people dedicated to develop high performance imaging products and solutions.

We designed Z CAM V1 Pro for VR professionals who want the best image quality as well as speedy workflow. The successful integration of VRWorks to Z CAM WonderStitch ushers in a new era of stereo 360º video. Nvidia's VRWorks makes high quality stereo 360º video post production workflow faster and easier than ever. And it will also enable Z CAM WonderLive to perform live stereo 360º video streaming soon." said Jason Zhang, Founder & Chairman of Z CAM

Learn more about Z CAM


Z-cam S1 Pro
Z-cam V1 Pro

Production Partners

YumeVR

The idea for Yume originated in early 2012 when one of our founders returned from an amazing experience at the Orangutan sanctuary in Indonesia. What if he could go back there for a brief moment, at any time, from any place? Recreating a dream.

“I love it. What is amazing about this live capability is that the sense of depth which is a lot more immersive than monoscopic video,” said Kevin Alderweireldt co-founder of YumeVR.”

Learn more about YumeVR

Grainger Films

Grainger Films is a multi-award winning, independent production company founded by brothers Jefferson and Chris Grainger, with a focus on film production, branded content, advertising and corporate works. Jefferson and Chris each have vast experience in the film industry, including production of award-winning commercial and independent projects, and working with some of the most prestigious companies in Australia and internationally.

"We have clients across a wide range of industries, from travel through to sports, who are demanding high quality 360 degree video," said Chris Grainger, CEO of Grainger VR. "Z CAM with NVIDIA VRWorks and 360 live stereo stitch capabilities makes it easy for us to deliver amazing 360 content. It also opens the doors to the future of live storytelling which allows filmmakers to push the boundaries of content creation.”

Learn more about Grainger Films