NVIDIA NGX is a new deep learning powered technology stack bringing AI-based features that accelerate and enhance graphics, photos imaging and video processing directly into applications. NVIDIA NGX features utilize Tensor Cores to maximize the efficiency of their operation, and require an RTX-capable GPU. The NGX SDK makes it easy for developers to integrate AI features into their application with pre-trained networks.
The following are examples of how Deep Learning and NGX can be used to improve creativity, performance and image quality for gamers and content creators.
InPainting allows the removal existing content from images, and uses NGX to replace the removed content with a realistic computer-generated alternative. For example, InPainting could be used to automatically remove power lines from a landscape image, replacing them seamlessly with the existing sky background.
- Details
While content aware replacement methods currently exist, these solutions utilize more basic methods of replacing image data. Traditional content aware fill algorithms search the image to find other areas of the image which best match the area to be replaced based on color data similarity. Because the replaced data is a copy of data already existing within the image, the matching process can introduce semantically meaningless data which exists within this copied data into the replacement if the algorithm feels that the content offers the best match.
InPainting uses a fundamentally different method for replacing content within an image. InPainting is based on a deep learning model that has been trained on a large scale image dataset. The training and the deep learning based algorithms allow InPainting to generate new content which may not exist in the original image. This method can help to minimize the inclusion of extraneous image information and result in more visually meaningful replacement results.
Image above: the areas in white are intelligently filled in.AI Slow-Mo inserts interpolated frames into a video stream to provide smooth, slow-motion video. The application analyzes frames for features and objects and identifies object and camera movement to create new video frames between the existing video frames.
AI Up-Res increases the resolution of an image or video by 2x, 4x or 8x. Unlike traditional filtering methods which stretch out the existing pixels and filter between them, AI Up-Res creates new pixels by interpreting the image and intelligently placing data. This results in a sharper enlargement that correctly preserves the depth of field and other artistic aspects.
NGX Architecture
- NGX software: The RTX feature is integrated into any game, application or plugin through the use of the NGX SDK. This is the main code for the AI-accelerated functionality, but it also requires a trained neural network in order to function. NGX has been designed to be thin - a header file that points to a DLL in the NVIDIA driver, making it simple to augment any application with these RTX features.The NGX SDK provides access to a number of RTX technologies and features for games and digital content creation and editing applications.The capabilities of NGX are tightly coupled to the NVIDIA driver and hardware and make use of Tensor Cores found in RTX-capable GPUs.
- NGX features: The trained deep neural networks (DNNs) are included in the driver download. These neural networks are updatable as they get more accurate and powerful allowing the performance and capabilities of these features to evolve over time.
If you would like to try out the NGX SDK please submit your request to join our Early Access Program by clicking the button below:
Join Early Access Program