NVIDIA TensorRT

NVIDIA TensorRT-based applications perform up to 36X faster than CPU-only platforms during inference, enabling developers to optimize neural network models trained on all major frameworks, calibrate for lower precision with high accuracy, and deploy to hyperscale data centers, embedded platforms, or automotive product platforms.

TensorRT, built on the NVIDIA CUDA® parallel programming model, enables developers to optimize inference by leveraging libraries, development tools, and technologies in CUDA-X™ for AI, autonomous machines, high performance computing, and graphics. With new NVIDIA Ampere Architecture GPUs, TensorRT also uses sparse tensor cores for an additional performance boost.

TensorRT provides INT8 using quantization aware training and post-raining quantization, and FP16 optimizations for production deployments of deep learning inference applications, such as video streaming, speech recognition, recommendation, fraud detection, text generation, and natural language processing. Reduced precision inference significantly minimizes application latency, which is a requirement for many real-time services, as well as autonomous and embedded applications.

With TensorRT, developers can focus on creating novel AI-powered applications rather than inference optimization. TensorRT optimized models can then be deployed with NVIDIA Triton™, an open-source inference serving software that includes TensorRT as one of its backends.