VRWorks 360 Video is NVIDIA’s SDK to enable VR developers and content creators to capture, stitch, and stream 360-degree videos.

The SDK supports 360-degree mono and stereo workflows, post production (offline mode) and real-time processing.


360-degree video processing is complex and computationally intensive. By leveraging the GPU, VRWorks 360 Video SDK provides a high-performance, high-quality and low-latency GPU-accelerated implementation that can be integrated into 360 video workflows.

Real-time stereo stitching is particularly challenging. The basic process generally involves ingesting, decoding and calibrating multiple high-resolution streams, stereo stitching, and encoding. The entire pipeline has to be executed in real time while maintaining the highest level of image quality. To accomplish that, NVIDIA has developed a new set of motion based algorithms for superior-quality stereo stitching, optimized for real time. Using consecutive frames, the algorithms estimate the motion of objects in a video stream, noting how they match and move across a seam while accounting for stereo disparity.

Hardware: Compatible with: Maxwell, Pascal and Volta based GPUs. (GeForce GTX 900 series and Quadro M5000 and higher)

Note: recommended spec for 360 stitching:
Mono: GeForce GTX 1060 6GB / Quadro P4000
Stereo offline: GeForce GTX 1080 / Quadro P4000
Stereo Live: Dual GeForce GTX 1080ti / Dual Quadro P6000
Software: Windows: 64 bit Windows
NVIDIA graphics driver 388.59 or later
Microsoft Visual Studio 2015 (MSVC14.0) or later
CUDA 9.0 Toolkit (NVCalib only)
CMake 3.2 or above

Linux: Ubuntu 16.04 and higher and Fedora 25 and higher


NVIDIA VRWorks 360 Video SDK

NVIDIA VRWorks 360 Video SDK consists of a library, set of APIs, sample applications and documentation.


Download VRWorks 360 Video SDK


  • Added Linux support for Linux version -
    Ubuntu 16.04 and up and Fedora 25 and up
  • Added support for GP100
  • Enhanced low level audio SDK to use static buffers for
    consistent processing performance
  • Enhanced quality for stereo panoramas when the rig is
    calibrated with nearby objects (e.g. indoor scenes)
  • Critical bug fixes

Key Features

  • High performance mono and stereo stitching GPU accelerated and optimized
  • Automated calibration
  • Mono stitch supports general rig configuration
  • Stereo stitch supports rigs with up to 32 cameras in equatorial ring configuration
  • Stereo stitching performance scalability over multiple GPUs
  • Hardware-accelerated decode and encode
  • Output panorama(s) in equirectangular projection
  • Different input/output forms (buffers, files) are supported to cater to different use cases (offline or live)
  • Audio can be included in the input videos or ingested from an external source
  • Blended audio output can be muxed into the output panoramic video or output as a separate audio stream

Partners (Click logos to learn more)






Z CAM

Z CAM is a product line created by a team of innovative and creative people dedicated to develop high performance imaging products and solutions.

We designed Z CAM V1 Pro for VR professionals who want the best image quality as well as speedy workflow. The successful integration of VRWorks to Z CAM WonderStitch ushers in a new era of stereo 360º video. Nvidia's VRWorks makes high quality stereo 360º video post production workflow faster and easier than ever. And it will also enable Z CAM WonderLive to perform live stereo 360º video streaming soon." said Jason Zhang, Founder & Chairman of Z CAM

Learn more about Z CAM

GTC2018 Session Video - How Z CAM VR Camera Benefited from VRWorks Integration
Z-cam S1 Pro
Z-cam V1 Pro





YumeVR

The idea for Yume originated in early 2012 when one of our founders returned from an amazing experience at the Orangutan sanctuary in Indonesia. What if he could go back there for a brief moment, at any time, from any place? Recreating a dream.

“I love it. What is amazing about this live capability is that the sense of depth which is a lot more immersive than monoscopic video,” said Kevin Alderweireldt co-founder of YumeVR.”

Learn more about YumeVR






Gainger Films

Grainger Films is a multi-award winning, independent production company founded by brothers Jefferson and Chris Grainger, with a focus on film production, branded content, advertising and corporate works. Jefferson and Chris each have vast experience in the film industry, including production of award-winning commercial and independent projects, and working with some of the most prestigious companies in Australia and internationally.

"We have clients across a wide range of industries, from travel through to sports, who are demanding high quality 360 degree video," said Chris Grainger, CEO of Grainger VR. "Z CAM with NVIDIA VRWorks and 360 live stereo stitch capabilities makes it easy for us to deliver amazing 360 content. It also opens the doors to the future of live storytelling which allows filmmakers to push the boundaries of content creation.”

Learn more about Grainger Films






STRIVR

STRIVR is revolutionizing the way companies train and develop their employees through the use of immersive technology. STRIVR’s end-to-end platform includes easy-to-use creation tools, flexible content management, and prescriptive analytics and insights. Using scientifically proven research into human behavior and engagement, STRIVR delivers superior training and better learning outcomes. With STRIVR, organizations around the world are leveraging the power of immersive learning to drive real results.

“Integrating VRWorks 360 video SDK accelerated the STRIVR stitching process from 15 fps to between 45 fps and 60 fps, a 3 to 4 times performance gain which translates into much faster turnaround time from filming to delivery.”

Brian Meek - CTO, STRIVR

Learn more about STRIVR






Pixvana

Pixvana is a Seattle-based software startup building a video creation and delivery platform for the emerging mediums of virtual, augmented, and mixed reality (XR).

"NVIDIA VRWorks 360 Video SDK has been very easy to integrate into our Linux cloud platform. The SDK shared the same API between Windows and Linux versions of the SDK and that made the port to Linux very straight forward. The whole process for integration has take about 4 weeks. The availability of a generalized automatic stitching process as part of the SDK has been instrumental in opening up our pipeline to accept just about any camera rig,"

Sean Safreed, Product Director and Cofounder at Pixvana

Learn more about Pixvana

GTC2018 Session Video - Stitching 8K Video in the Cloud with Pixvana SPIN Studio and VRWorks


360-degree video

360-degree videos have gained popularity due to their ability to encompass a wide field of view. The proliferation of virtual reality devices has given rise to an increasing need for videos with a 360 degree field of view to deliver an enhanced sense of immersion and presence.

360-degree videos can be generated by aligning streams and combining them appropriately in the areas of overlap to produce a seamless 360-degree view. These videos can be generated by using a rig that has sufficient number of cameras with a wide enough field of view to capture the 360-degree space.


Content provided by Kevin Alderweireldt of Yume Vr and features Umamido
See full result video (over/under) here




Content provided by Z Cam

360-degree video pipeline



Ingest

The VRWorks 360 Video SDK can ingest MP4 compressed videos. It can also read from RGB files as well as CUDA arrays. The SDK inherently supports HW accelerated decode of compressed input. VRWorks 360 Video SDK currently supports up to 32 inputs which should accommodate most 360 camera arrays. Audio can be included in the input videos or ingested from an external source. There is no restriction on the number of audio inputs being ingested.

Decode

NVIDIA GPUs contain one or more hardware-based decoder and encoder(s) (separate from the CUDA cores) which provides fully-accelerated hardware-based video decoding and encoding for most video formats. With decoding/encoding offloaded, the graphics engine and the CPU are free for other operations.

During import, video files are decompressed and decoded into a raw format for image processing. VRWorks 360 video SDK makes use of the integrated hardware and NVDEC (part of the NVIDIA Video Codec SDK)

Calibrate

During the calibration process, the individual video streams are calibrated in relation to each other. This SDK calibrates for lens distortion as well as the rotation and translation between each camera within the rig. Using intrinsic and extrinsic parameters particular to the rig, the individual streams are aligned and calibrated, then stitched into a single 360-degree video. The SDK also supports camera calibration without requiring input estimates for camera parameters. It automatically computes estimated focal length, rotation, principal point and fisheye radius if not provided. Non homogenous camera rigs are also supported given constraints related to overlap are satisfied. These rigs can be composed of cameras with different resolution, lens distortion type or focal length. This SDK also includes auto balancing for automatic equatorial alignment of the camera rig.

Learn more about the calibration process in this developer blog: Calibrating Stitched Videos with VRWorks 360 Video SDK

Stitch

The process of generating 360-degree videos typically includes capturing monocular footages that are then stitched into a single 360-degree video. In the case of monoscopic 360-degree video, the output is a single 360-degree panorama generated by aligning input videos and blending in the areas of overlap. Stereoscopic stitching produces a pair of 360-degree panoramas, one for each eye. This projection model is often referred to as Omnidirectional stereo. To avoid artifacts, this process requires careful handling of issues such as camera calibration, parallax in the source videos, and temporal consistency. Because of this, the stitching process is complex and computationally challenging.

VRWorks 360 Video SDK is a GPU-accelerated solution designed to achieve real-time stitching of images into stereo 360-degree videos. The performance of the SDK is GPU scalable allowing two or more Pascal based GPUs to produce a live stitched 4K 30fps stream from an 8 camera 4K 30fps input.


Click on image to enlarge

VRWorks 360 Video SDK offers real-time solutions for both monoscopic and stereoscopic 360 stitching. Both modes utilize hardware accelerated decoding and encoding to achieve real-time ingest and output of high resolution streams. Mono stitching uses GPU accelerated Multiband blending to avoid over-smoothing, distortion and visible seams. Stereo stitching is more computationally intensive and applies motion vector estimation to accomplish seamless stereoscopic output regardless of the captured scene.

The SDK supports both AAC and PCM audio and allows for user specified gain to be applied. Audio stereo spread can be optionally applied to the incoming streams during blending.

Encode

During export, video files are encoded and compressed into the desired delivery format. VRWorks 360 video SDK makes use of the integrated hardware and NVENC (part of the NVIDIA Video Codec SDK)

Output

The VRWorks 360 Video SDK supports various output formats: RGB textures, H.264 compressed streams, MP4 files. Compression is HW accelerated. Equirectangular projection is supported. The SDK allows the user the option of multiplexing the audio and video into an MP4 or outputting audio and video as separate files

Audio

Audio can be included in the input videos or ingested from an external source. Blended audio output can be muxed into the output panoramic video or output as a separate audio stream. Ambisonic audio is now supported as well...