VRWorks - 360 Video
VRWorks 360 Video is NVIDIA’s SDK to enable VR developers and content creators to capture, stitch, and stream 360-degree videos.
The SDK supports 360-degree mono and stereo stitching, post production (offline mode) and real-time workflows.
360-degree video processing is complex and computationally intensive. By leveraging NVIDIA GPUs, VRWorks 360 Video SDK provides a high-performance, high-quality and low-latency GPU-accelerated implementation that can be integrated into 360 video workflows.
Real-time stereo stitching is particularly challenging. The basic process generally involves ingesting, decoding and calibrating multiple high-resolution streams, stereo stitching, and encoding. The entire pipeline has to be executed in real time while maintaining the highest level of image quality. To accomplish this, NVIDIA has developed a new set of motion based algorithms for superior-quality stereo stitching, that are optimized for real time processing. Using consecutive frames, the algorithms estimate the motion of objects in a video stream, noting how they match and move across a seam while accounting for stereo disparity.
|Hardware:||Compatible with: Maxwell and later generations of GPUs. (GeForce GTX 900 series and Quadro M5000 and higher)
Note: recommended spec for 360 stitching:
Mono: GeForce GTX 1060 6GB / Quadro P4000
Stereo offline: GeForce GTX 1080 / Quadro P4000
Stereo Live: Dual GeForce GTX 1080Ti / Dual Quadro P6000
|Software:||Windows: 64 bit Windows
NVIDIA graphics driver 411.63 or later
Microsoft Visual Studio 2015 (MSVC14.0) or later
CUDA 10.0 Toolkit
CMake 3.2 or above
Linux: Ubuntu 16.04 and higher and Fedora 25 and higher
NVIDIA VRWorks 360 Video SDK
NVIDIA VRWorks 360 Video SDK consists of a library, set of APIs, sample applications and documentation.
Download and Release Notes
- High performance mono and stereo stitching
- Automated calibration
- Mono stitch supporting general rig configurations
- Stereo stitch supporting rigs of up to 32 cameras in an equatorial ring configuration
- Scalable stereo stitching performance over multiple GPUs
- GPU-accelerated decode and encode
- Output panorama(s) in equirectangular projection
- Different input/output forms (buffers, files) are supported to cater to different use cases (offline or live)
- Audio can be included in the input videos or ingested from an external source
- Blended audio output can be muxed into the output panoramic video or output as a separate audio stream
- Ambisonic audio
- Region of interest stitching (allows for custom Field of view outputs)
- Depth based stitching
- NVIDIA BLOG - NVIDIA RTX-Powered VRWorks 360 Video SDK Brings Big Speedups and New Features
- DEVBLOG - VRWorks 360 Video SDK 2.0 Adds Features, Turing Support
- GTC 2018 VIDEO - Stitching 8K Video in the Cloud with Pixvana SPIN Studio and VRWorks
- GTC 2018 VIDEO - How Z CAM VR Camera Benefited from VRWorks Integration
- NVIDIA BLOG - Industry Leaders Adopt NVIDIA VRWorks to Push Boundaries of 360-Degree Video Storytelling
- DEV BLOG - Calibrating Stitched Videos with VRWorks 360 Video SDK
- NEWS BLOG - NVIDIA VRWorks 360 Video SDK 1.1 released
Partners (Click logos to learn more)
Z CAM is a product line created by a team of innovative and creative people dedicated to develop high performance imaging products and solutions.
We designed Z CAM V1 Pro for VR professionals who want the best image quality as well as speedy workflow. The successful integration of VRWorks to Z CAM WonderStitch ushers in a new era of stereo 360º video. Nvidia's VRWorks makes high quality stereo 360º video post production workflow faster and easier than ever. And it will also enable Z CAM WonderLive to perform live stereo 360º video streaming soon." said Jason Zhang, Founder & Chairman of Z CAMGTC2018 Session Video - How Z CAM VR Camera Benefited from VRWorks Integration
The idea for Yume originated in early 2012 when one of our founders returned from an amazing experience at the Orangutan sanctuary in Indonesia. What if he could go back there for a brief moment, at any time, from any place? Recreating a dream.
“I love it. What is amazing about this live capability is that the sense of depth which is a lot more immersive than monoscopic video,” said Kevin Alderweireldt co-founder of YumeVR.”
Grainger Films is a multi-award winning, independent production company founded by brothers Jefferson and Chris Grainger, with a focus on film production, branded content, advertising and corporate works. Jefferson and Chris each have vast experience in the film industry, including production of award-winning commercial and independent projects, and working with some of the most prestigious companies in Australia and internationally.
"We have clients across a wide range of industries, from travel through to sports, who are demanding high quality 360 degree video," said Chris Grainger, CEO of Grainger VR. "Z CAM with NVIDIA VRWorks and 360 live stereo stitch capabilities makes it easy for us to deliver amazing 360 content. It also opens the doors to the future of live storytelling which allows filmmakers to push the boundaries of content creation.”
STRIVR is revolutionizing the way companies train and develop their employees through the use of immersive technology. STRIVR’s end-to-end platform includes easy-to-use creation tools, flexible content management, and prescriptive analytics and insights. Using scientifically proven research into human behavior and engagement, STRIVR delivers superior training and better learning outcomes. With STRIVR, organizations around the world are leveraging the power of immersive learning to drive real results.
“Integrating VRWorks 360 video SDK accelerated the STRIVR stitching process from 15 fps to between 45 fps and 60 fps, a 3 to 4 times performance gain which translates into much faster turnaround time from filming to delivery.”Brian Meek - CTO, STRIVR
Pixvana is a Seattle-based software startup building a video creation and delivery platform for the emerging mediums of virtual, augmented, and mixed reality (XR).
GTC2018 Session Video - Stitching 8K Video in the Cloud with Pixvana SPIN Studio and VRWorks
"NVIDIA VRWorks 360 Video SDK has been very easy to integrate into our Linux cloud platform. The SDK shared the same API between Windows and Linux versions of the SDK and that made the port to Linux very straight forward. The whole process for integration has take about 4 weeks. The availability of a generalized automatic stitching process as part of the SDK has been instrumental in opening up our pipeline to accept just about any camera rig,"Sean Safreed, Product Director and Cofounder at Pixvana
360-degree videos have gained popularity due to their ability to encompass a wide field of view. The proliferation of virtual reality devices has given rise to an increasing need for videos with a 360 degree field of view to deliver an enhanced sense of immersion and presence.
360-degree videos can be generated by aligning streams and combining them appropriately in the areas of overlap to produce a seamless 360-degree view. These videos can be generated by using a rig that has sufficient number of cameras with a wide enough field of view to capture the 360-degree space.
Content provided by Kevin Alderweireldt of Yume Vr and features Umamido
See full result video (over/under) here
Content provided by Z Cam
360-degree video pipeline
The VRWorks 360 Video SDK can ingest MP4 compressed videos. It can also read from RGB files as well as CUDA arrays. The SDK inherently supports HW accelerated decode of compressed input. VRWorks 360 Video SDK currently supports up to 32 inputs which should accommodate most 360 camera arrays. Audio can be included in the input videos or ingested from an external source. There is no restriction on the number of audio inputs being ingested.
NVIDIA GPUs contain one or more hardware-based decoder and encoder(s) (separate from the CUDA cores) which provides fully-accelerated hardware-based video decoding and encoding for most video formats. With decoding/encoding offloaded, the graphics engine and the CPU are free for other operations.
During import, video files are decompressed and decoded into a raw format for image processing. VRWorks 360 video SDK makes use of the integrated hardware and NVDEC (part of the NVIDIA Video Codec SDK)
During the calibration process, the individual video streams are calibrated in relation to each other. This SDK calibrates for lens distortion as well as the rotation and translation between each camera within the rig. Using intrinsic and extrinsic parameters particular to the rig, the individual streams are aligned and calibrated, then stitched into a single 360-degree video. The SDK also supports camera calibration without requiring input estimates for camera parameters. It automatically computes estimated focal length, rotation, principal point and fisheye radius if not provided. Non homogenous camera rigs are also supported given constraints related to overlap are satisfied. These rigs can be composed of cameras with different resolution, lens distortion type or focal length. This SDK also includes auto balancing for automatic equatorial alignment of the camera rig.
Learn more about the calibration process in this developer blog: Calibrating Stitched Videos with VRWorks 360 Video SDK
The process of generating 360-degree videos typically includes capturing monocular footage that are then stitched into a single 360-degree video. In the case of monoscopic 360-degree video, the output is a single 360-degree panorama generated by aligning input videos and blending in the areas of overlap. Stereoscopic stitching produces a pair of 360-degree panoramas, one for each eye. This projection model is often referred to as Omnidirectional stereo. To avoid artifacts, this process requires careful handling of issues such as camera calibration, parallax in the source videos, and temporal consistency. Because of this, the stitching process is complex and computationally challenging.
VRWorks 360 Video SDK is a GPU-accelerated solution designed to achieve real-time stitching of images into stereo 360-degree videos. The performance of the SDK is GPU scalable allowing multiple GPUs to produce a live stitched 4K 30fps stream from an 8 camera 4K 30fps input.
VRWorks 360 Video SDK offers real-time solutions for both monoscopic and stereoscopic 360 stitching. Both modes utilize GPU accelerated decoding and encoding to achieve real-time ingest and output of high resolution streams. Mono stitching uses GPU accelerated Multiband blending to avoid over-smoothing, distortion and visible seams. Stereo stitching is more computationally intensive and applies motion vector estimation to accomplish seamless stereoscopic output regardless of the captured scene.
Depth-based mono stitch is a new stitching pipeline that uses depth based alignment to improve the stitching quality in scenes with objects close to the camera rig and improves the quality across the region of overlap between two cameras.
Region of interest stitch enables adaptive stitching by defining the desired field of view rather than stitching a complete panorama. This opens up new use cases such as 180-degree VR and can reduce execution time and improve performance.
Moveable Seams is a feature that enables developers to adjust the seams location in the region of overlap between 2 cameras to preserve visual fidelity in the region of overlap particularly when objects are close to the camera.
During export, video files are encoded and compressed into the desired delivery format. VRWorks 360 video SDK makes use of the integrated hardware and NVENC (part of the NVIDIA Video Codec SDK)
The VRWorks 360 Video SDK supports various output formats: RGB textures, H.264 compressed streams, MP4 files. Compression is GPU-accelerated. Equirectangular projection is supported. The SDK gives the user the option of multiplexing the audio and video into an MP4 or outputting audio and video as separate files.
The SDK supports both AAC and PCM audio and allows for user specified gain to be applied. Audio stereo spread can be optionally applied to the incoming streams during blending.
Audio can be included in the input videos or ingested from an external source. Blended audio output can be muxed into the output panoramic video or output as a separate audio stream.
Ambisonic Audio is a technique to record, mix and playback 3D 360-degree audio. The new pipeline enables 3D, omnidirectional audio such that the perceived direction of sound sources change when viewers modify their orientation resulting in a much more immersive 360-video experience.
Warp 360 provides highly-optimized image warping and distortion removal by converting images between a number of projection types, including perspective, fisheye and equirectangular. It can transform equirectangular stitched output into projection format such as cubemap to reduce streaming bandwidth leading to increased performance.