NVIDIA NGX is a new deep learning powered technology stack bringing AI-based features that accelerate and enhance graphics, photos imaging and video processing directly into applications. NVIDIA NGX features utilize Tensor Cores to maximize the efficiency of their operation, and require an RTX-capable GPU. The NGX SDK makes it easy for developers to integrate AI features into their application with pre-trained networks.
The following are examples of how Deep Learning and NGX can be used to improve creativity, performance and image quality for gamers and content creators.
InPainting allows the removal existing content from images, and uses NGX to replace the removed content with a realistic computer-generated alternative. For example, InPainting could be used to automatically remove power lines from a landscape image, replacing them seamlessly with the existing sky background.
While content aware replacement methods currently exist, these solutions utilize more basic methods of replacing image data. Traditional content aware fill algorithms search the image to find other areas of the image which best match the area to be replaced based on color data similarity. Because the replaced data is a copy of data already existing within the image, the matching process can introduce semantically meaningless data which exists within this copied data into the replacement if the algorithm feels that the content offers the best match.
InPainting uses a fundamentally different method for replacing content within an image. InPainting is based on a deep learning model that has been trained on a large scale image dataset. The training and the deep learning based algorithms allow InPainting to generate new content which may not exist in the original image. This method can help to minimize the inclusion of extraneous image information and result in more visually meaningful replacement results.
AI Slow-Mo inserts interpolated frames into a video stream to provide smooth, slow-motion video. The application analyzes frames for features and objects and identifies object and camera movement to create new video frames between the existing video frames.
AI Up-Res increases the resolution of an image or video by 2x, 4x or 8x. Unlike traditional filtering methods which stretch out the existing pixels and filter between them, AI Up-Res creates new pixels by interpreting the image and intelligently placing data. This results in a sharper enlargement that correctly preserves the depth of field and other artistic aspects.
Distracting jagged lines on the edges of objects in a scene, or aliasing, is a common artifact in real-time computer graphics. A common way to remove the jagged lines is to increase the number of pixels, or samples, on the line which helps to smooth it. Increasing the resolution of the entire image is not always practical, so many techniques have been developed which intelligently blend the colors of the jagged edges with the colors of nearby pixels but most of these often remove fine detail.
NVIDIA researchers have successfully trained a neural network to find these jagged edges and perform high-quality anti-aliasing by determining the best color for each pixel, and then apply proper colors to create smoother edges and improve image quality. This technique is known as Deep Learning Super Sample (DLSS). DLSS is like an “Ultra AA” mode-- it provides the highest quality anti-aliasing with fewer artifacts than other types of anti-aliasing.
DLSS requires a training set of full resolution frames of the aliased images that use one sample per pixel to act as a baseline for training. Another full resolution set of frames with at least 64 samples per pixel acts as the reference that DLSS aims to achieve. At runtime, a full resolution frame with motion vectors is required in addition to the input image to be anti-aliased. The motion vectors are used to temporally transform the previously anti-aliased frame to align with the current frame, enabling the algorithm to use temporal information. Training also requires a variety of scenes from the game in order to generate the best results.
Many games today support Temporal Anti-Aliasing (TAA), an anti-aliasing technology that combines high-quality Multi-Sample Anti-Aliasing (MSAA), post processes, and temporal filters. Unfortunately, the temporal aspect of TAA can deliver a blurry-looking image, especially in static scenes. TAA also suffers from flickering and ghosting that is inherent in the technology. DLSS by comparison provides sharp images and is temporally stable. The difference in clarity is dramatic.
- NGX software: The RTX feature is integrated into any game, application or plugin through the use of the NGX SDK. This is the main code for the AI-accelerated functionality, but it also requires a trained neural network in order to function. NGX has been designed to be thin - a header file that points to a DLL in the NVIDIA driver, making it simple to augment any application with these RTX features.The NGX SDK provides access to a number of RTX technologies and features for games and digital content creation and editing applications.The capabilities of NGX are tightly coupled to the NVIDIA driver and hardware and make use of Tensor Cores found in RTX-capable GPUs.
- NGX features: The trained deep neural networks (DNNs) are included in the driver download. These neural networks are updatable as they get more accurate and powerful allowing the performance and capabilities of these features to evolve over time.