TensorRT: What’s New
NVIDIA® TensorRT™ 8.4 includes new tools to explore TensorRT optimized engines and quantize the TensorFlow models with QAT.
Torch-TensorRT is now an official part of PyTorch, read more about the announcement here.
Highlights:
- New tool to visualize optimized graphs and debug model performance easily. Learn more
- Reduce engine size by up to 50x, allowing for easier application package distribution
- New TF-QAT toolkit for improved INT8 accuracy on TensorFlow models. Learn more
Torch-TensorRT is available today in the PyTorch container from the NVIDIA NGC™ catalog.
TensorFlow-TensorRT is available today in the TensorFlow container from the NGC catalog.
How to Get Started with TensorRT
Intermediate
- Documentation
- Sample codes
- BERT, EfficientDet inference using TensorRT [Jupyter Notebook]
Expert
- Quantization Aware Training (QAT) with TensorRT [Blog]
- PyTorch-Quantization (QAT) Toolkit [Python Code]
- Sparsity with TensorRT [Blog]
Intermediate
Beginner
Intermediate
- Documentation [Sample Code]
- Object Detection with TF-TRT [Sample Code]
- Image Classifcation with TF-TRT [Video]
Find other TensorRT developers on the NVIDIA Developer Forum or additional resources on the NVIDIA Developer Blog.
Additional TensorRT Resources
Conversational AI
- Real-Time Natural Language Understanding with BERT Using TensorRT (Blog)
- Real-Time Inference T5 & GPT-2 using NVIDIA TensorRT (Blog)
- Quantize BERT with PTQ and QAT for INT8 Inference (Sample)
- Automatic Speech Recognition with TensorRT (Notebook)
- Accelerating Real-Time Text-to-Speech with TensorRT (Blog)
- NLU with BERT (Notebook)
- Real Time Text-to-Speech (Sample)
- Neural Machine Translation (NMT) Using A Sequence To Sequence (seq2seq) Model (Sample Code)
- Building An RNN Network Layer By Layer (Sample Code)
Image and Vision
- Optimize Object Detection with EfficientDet and TensorRT 8 (Notebook)
- Estimating Depth with ONNX Models and Custom Layers Using NVIDIA TensorRT (Blog)
- Speeding up Deep Learning Inference using TensorFlow, ONNX, and TensorRT (Semantic Segmentation Blog)
- Object detection with SSD, Faster R-CNN networks (C++ Code Samples)
- Accelerating Inference with Sparsity using Ampere Architecture and TensorRT (Blog)
- Achieving FP32 Accuracy in INT8 using Quantization Aware Training with TensorRT (Blog)
NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.