The NVIDIA® NGX neural graphics framework is a new deep learning-based SDK that will bring powerful new previously unrealizable AI-based features that accelerate and enhance graphics, image, and video processing to innovative applications from ISVs ranging from industry heavyweights to disruptive startups. NVIDIA NGX utilizes the Tensor Cores found in NVIDIA Quadro® RTX™ series products to implement previously unheard-of features while making it easier for developers to integrate AI features into their application(s) with pre-trained neural networks.
NVIDIA NGX implements AI delivered and enhanced features into graphics or video applications by creating a “ground truth”, which is used to train a DNN (Deep Neural Network). The source material utilized to train the DNN to create the ground truth is typically a large set of very high-resolution images or videos. Once trained the AI model can infer (in the language of AI) what to do when presented with graphics or video that needs to be processed by the AI to deliver the results required based on AI-powered features creative professionals need to accomplish within the NGX enhanced application. AI training and inferencing both depend on the specialized matrix math operations provided by Tensor Cores, which NVIDIA assists developers to access through the NGX API.
NVIDIA delivers three major NGX features: AI Inpainting, AI-Slow-Mo, and AI Up-Res. ISVs or in-house developers can implement additional capabilities by creating and training their models.
AI Inpainting
Inpainting removes existing content from images, for example distracting elements in a still image, or can restore missing or damaged portions of an image (or video). NGX s used to replace the undesirable or removed content with a realistic computer-generated alternative. Examples would include using Inpainting to automatically remove pipes from a landscape image, replacing them seamlessly with the ground or landscape background. Other examples would include reconstruction of faces where portions are missing due to damage or deliberate creative intent, for example changing hairstyle or de-aging.
AI Slow-Mo
AI Slow-Mo inserts AI interpolated frames into a video stream to provide smooth, slow-motion video. The application analyzes frames for features and objects and identifies object and camera movement to create new video frames between the existing video frames. A video shot at 30 FPS could be changed to 60 FPS for smoother motion, even to 120 FPS or 240 FPS for smooth slow motion, without changing the overall length of the video. With AI Slow-Mo video shot at standard FPS rates can be quickly turned into extraordinarily smooth slow-motion sequences – without the hassle, expense, or potentially overheating problems found in some video cameras that offer this feature. One live on-air broadcast example would be the ability to view instant replay results in enhanced slow motion to allow officials or referees to make better calls.
AI Up-Res
AI Up-Res increases the resolutions of images or video by 2x, 4x, or 8x. Unlike scaling based on traditional filtering methods like the nearest neighbor or bicubic filtering, AI Up-Res (also referred to as AI Super-Res) doesn’t stretch out existing pixels and then filter between them, AU Up-Res creates new pixels by interpreting the image and intelligently placing data. This results if a far sharper enlargement that correctly preserves the depth of field and other artistic and aesthetic aspects of the source material.
To summarize, NGX utilized deep neural networks (DNNs) and a set of Neural Services to perform AI-based functions that accelerate graphics, rendering, and other client-side applications. NGX employs the Tensor Cores found in NVIDIA Quadro RTX products for deep learning-based operations and accelerates the delivery of NVIDIA deep learning research directly to end-users. NGX does not work on GPU architectures before Turing, on which NVIDIA Quadro RTX is based. NVIDIA NGX is tightly integrated with the NVIDIA Quadro driver and hardware, and the NGX API is thin and easy for developers to use, which provides access to multiple AI-based features 9described above), pre-trained by NVIDIA.
All NVIDIA NGX features can be managed by NVIDIA’s new – check it out – NVIDIA Quadro Experience if you have an NVIDIA Quadro RTX GPU installed and, upon finding one in the system, proceeds to download the NVIDIA NGX Core package as well as the deep neural network models available for installed applications (or games). As NVIDIA continues to train and enhance its DNNs included as part of NGX the enhancements automatically find their way into the features of ISV applications that utilize NGX – a win-win for all parties (NVIDIA, developers, and most importantly customers) involved!
For additional information, albeit with a bias towards developers, visit https://developer.nvidia.com/rtx/ngx. Watch the embedded videos to see the amazing capabilities of NVIDIA NGX in action. NGX enhanced applications will be coming to the market in 2020.
