The world of live streaming has expanded rapidly this year: Twitch has seen an 89% increase in the number of streamers, while viewership is up 56%. Meanwhile more employees are working from home and collaborating by video conference. Background noise in the home and messy rooms can be an inconvenience for those hosting video meetings, and distracting to viewers tuning into a livestream. Fortunately, AI can help.
Earlier this month, we announced the NVIDIA Broadcast app, an end-user application that provides noise removal, virtual backgrounds and webcam auto-frame features to end-users. Now, we are announcing the release of the SDKs that powers this app – the NVIDIA Broadcast Engine – so livestream, video conference, voice chat, and video editing developers can integrate these technologies into their own applications.
The NVIDIA Broadcast Engine leverages NVIDIA RTX GPUs and their dedicated Tensor Cores to enhance broadcasts and video conferences through a set of AI-powered SDKs:
Audio Effects SDK enables developers to apply AI-based noise removal to any audio feed, while preserving the quality of the speaker’s voice.
Video Effects SDK applies AI effects to videos. The first filter in the SDK is Video Super Resolution – an effect that improves the quality of a video (live or recorded) in real time by using AI to remove video compression artifacts and upscaling it.
NVIDIA AR SDK is getting new feature optimizations, and doubles the points it can track on a person’s face to 126. This allows developers to track features with more detail, enabling immersive effects.
Video Codecs SDK recently updated with version 10.0, with an updated and optimized set of encoding profiles, access to Optical Flow 2.0, and more.
XSplit, NDI, Notch, and Xaymar Integrate NVIDIA Broadcast Engine
- XSplit has integrated the Audio Effects SDK into XSplit Broadcaster, the popular live streaming client, to enable their users to apply noise removal per audio source.
- NDI, a division of Vizrt Group, has integrated the Video Codec SDK into NDI, their popular solution to transmit video between PCs through a local network, replacing the need for a capture card. NDI now provides NVIDIA GPU acceleration for screen capture and encoding, providing higher performance and quality, enabling capture of up to 4K120.
- Notch is using the AR SDK to enable advanced Face Tracking effects to animate and create advanced real-time effects, greatly reducing the processing time while improving quality and stability of the tracking versus the old CPU-based solution.
- Xaymar, the creator of the popular StreamFX plugin for OBS Studio, has ported NVIDIA’s noise removal into a VST3 filter – VoiceFX – allowing editors to load it as a filter in VST3 compatible apps, including Adobe Premiere Pro and Adobe Audition.
To learn more or to download the SDKs, please visit NVIDIA Broadcast Engine.
If you want to test one of these integrations, download the updated version of the app, and make sure to download the redistributable files. You can also test out some of these effects in the NVIDIA Broadcast App.
