NVIDIA Triton Inference Server

Support for Multiple Frameworks

Triton Inference Server supports all major frameworks like TensorFlow, TensorRT, PyTorch, ONNX Runtime, and even custom framework backends. It provides AI researchers and data scientists the freedom to choose the right framework for their projects.

High-Performance Inference

Triton runs models concurrently on GPUs to maximize utilization, supports CPU-based inferencing, and offers advanced features like model ensemble and streaming inferencing. It helps developers bring models to production rapidly.

Designed for IT, DevOps, and MLOps

Available as a Docker container, Triton integrates with Kubernetes for orchestration and scaling, is part of Kubeflow, exports Prometheus metrics for monitoring, and can be used with cloud AI platforms like Azure ML and Google CAIP. It helps IT/DevOps streamline model deployment in production.

Triton can load models from local storage or cloud platforms. As models are retrained with new data, developers can easily make updates without restarting the inference server or disrupting the application.

Triton runs multiple models from the same or different frameworks concurrently on a single GPU or CPU. In a multi-GPU server, it automatically creates an instance of each model on each GPU to increase utilization without extra coding.

It supports real-time inferencing, batch inferencing to maximize GPU/CPU utilization, and streaming inference with built-in support for audio streaming input. It also supports model ensemble for use cases that require multiple models to perform end-to-end inference, such as conversational AI.

Users can also use shared memory. The Inputs and outputs that pass to and from Triton are stored in shared memory, reducing HTTP/gRPC overhead and increasing performance.

Also available as a Docker container, Triton integrates with Kubernetes for orchestration, metrics, and auto-scaling. It also integrates with Kubeflow and Kubeflow pipelines for an end-to-end AI workflow. It exports Prometheus metrics for monitoring GPU utilization, latency, memory usage, and inference throughput. It supports the standard HTTP/gRPC interface to connect with other applications like load balancers. And it can easily scale to any number of servers to handle increasing inference loads for any model.

Triton can serve tens or hundreds of models through model control API. Models can be loaded and unloaded into and out of the inference server based on changes to fit in GPU or CPU memory. Supporting a heterogeneous cluster with both GPUs and CPUs helps standardize inference across platforms and dynamically scale out to any CPU or GPU to handle peak loads.

View the latest Triton release notes on GitHub.

Simplify AI Deployment at Scale

Simplify the deployment of AI models at scale in production. Learn how Triton meets the challenges of deploying AI models and review the steps to getting started with Triton.

Deep Learning Inference Platform

Achieve the performance, efficiency, and responsiveness critical to powering the next generation of AI products and services—in the cloud, in the data center and edge.

Deploy AI Deep Learning Models

Get the latest news and updates, release note highlights, and key benefits of using Triton Inference Server and NGC™ open-source software on the NVIDIA Developer Blog.