Real-time AI for Face Tracking, Virtual Greenscreen, and Style Transfer Filters


NVIDIA’s RTX Broadcast Engine is a set of SDKs that use GPU acceleration to deliver AI-driven broadcast features. Live streaming app-makers can provide their customers with advanced face tracking, virtual greenscreen, and style transfer filters, all without the cost and inconvenience of traditional studio setups. These features are possible in real-time - without special cameras or other hardware - due to the power of RTX GPUs and dedicated Tensor Cores.


Sign Up for NVIDIA RTX Broadcast Engine News


The RTX Broadcast Engine is currently available in Early Access to select partners, but we will soon open it up to other developers. To request Early Access to the RTX Broadcast Engine or stay up to date with its latest developments.

RTX Broadcast Engine SDKs

RTX AR SDK

Enables real-time 3D tracking of a person’s face using a standard web camera. Developers can create fun, engaging augmented reality effects, such as overlaying 3D content on your face, allowing you to control 3D characters, or adding immersive effects that make you look like you are in the game.

The SDK features include:

  • Face Tracking: AI based technology that detects human faces in images and video feed. It determines the location of the face and specifies the size using a bounding box. It is accelerated on tensor cores.
  • Face Landmark Tracking: AI based technology that identifies the facial features and facial contour of a person, (including face shape, lips, eyes and eyelids,) using 68 key points. It tracks 3 degrees of freedom head pose and facial deformation due to head movement and expressions. It is accelerated on tensor cores.
  • Face Mesh: GPU accelerated optimization using a 3D morphable model to create a 3D mesh representation of a human face with up to 3000 vertices and 6 degrees of freedom head pose in real-time.
face tracking

Video Effects SDK

Gives content creators control over the look and feel of their streams, without requiring special cameras or other hardware.

The SDK features include:

  • RTX Greenscreen: RTX Greenscreen delivers real-time background removal of a webcam feed, so only your face and body show up on your livestream. It uses AI to separate the talent from the background so you can enjoy the benefits of a physical greenscreen, without the hassle of installing it.
  • RTX Style Filters: RTX Style Filters uses AI to augment the look and feel of your webcam feed based on the style of another image. Imagine styling your webcam feed in the look of Van Gogh’s Starry Night.
greenscreen

stylized

Video Codec SDK

Unlock the full potential of streaming apps with dedicated hardware encoding and decoding, reducing the FPS impact of streaming by up to 50%! With RTX GPUs, NVENC delivers up to 15% better efficiency for H.264 encoding, and 25% for HEVC encoding compared to prior generations.


Minimum System Requirements
Hardware: Compatible with NVIDIA GeForce RTX 2060 / Quadro RTX 3000 or higher
Software: Windows 10 64-bit
NVIDIA graphics driver version 410.48 or later
Microsoft Visual Studio 2017 (MSVC14.1) or later
CMake 3.12 or above
CUDA 10.0 or above
cuDNN 7.3.1 or above

Industry Support


OBS



"NVIDIA has been at the top of my list when it comes to streaming and recording equipment. I’m continually impressed with what they’re doing,” said Hugh Bailey, author, OBS. “And their technology is impressive with RTX features like AI Greenscreen."

Xsplit


"We have collaborated with NVIDIA over the years on many projects and the introduction of the NVIDIA RTX Broadcast Engine is by far the most exciting. This new development represents a huge leap forward for creators using supported RTX-enabled GPUs, providing them with new avenues to enhance their productions. For the XSplit team, we are excited to integrate these new tools into our suite of apps, enabling our users to create better content by maximizing the potential of NVIDIA GeForce RTX,” said Miguel Molina, Director of Developer Relations, XSplit."

elgato



"By integrating NVIDIA’s Video Codec SDK into our software and pairing it with our new 4K60 Pro MK.2 capture card, we’re able to significantly improve the quality of recorded content,” says Philipp Eggebrecht, Technical Marketing Specialist, Elgato. “Gamers can play and capture their favorite titles in state-of-the-art 4K with HDR turned on in real-time, without impacting PC performance."

streamlabs


"The new RTX Broadcast Engine is an exciting advancement that will allow developers in our app store to create powerful new tools for streamers with NVIDIA RTX GPUs," said Ali Moiz, CEO of Streamlabs. "We're thrilled to continue working with NVIDIA as they introduce new features to the Streamlabs developer community, and look forward to implementing this new technology."