AI Inference

An illustration showing the steps "LLM" then "Optimize" then "Deploy."
Dec 04, 2023

NVIDIA TensorRT-LLM Enhancements Deliver Massive Large Language Model Speedups on NVIDIA H200

Large language models (LLMs) have seen dramatic growth over the last year, and the challenge of delivering great user experiences depends on both high-compute...
5 MIN READ
Nov 28, 2023

One Giant Superchip for LLMs, Recommenders, and GNNs: Introducing NVIDIA GH200 NVL32

At AWS re:Invent 2023, AWS and NVIDIA announced that AWS will be the first cloud provider to offer NVIDIA GH200 Grace Hopper Superchips interconnected with...
9 MIN READ
Nov 27, 2023

Announcing HelpSteer: An Open-Source Dataset for Building Helpful LLMs

NVIDIA recently announced the NVIDIA NeMo SteerLM technique as part of the NVIDIA NeMo framework. This technique enables users to control large language model...
6 MIN READ
Nov 17, 2023

Mastering LLM Techniques: Inference Optimization

Stacking transformer layers to create large models results in better accuracies, few-shot learning capabilities, and even near-human emergent abilities on a...
25 MIN READ
Nov 13, 2023

Upcoming Webinar Series: How to Get Started With AI Inference

Join us for a series of expert-led talks where we'll explore a full-stack approach to AI inference and how to optimize the AI-inferencing workflow to lower...
1 MIN READ
Stylized image of a workflow, with nodes labelled LLM, Optimize, and Deploy.
Oct 19, 2023

Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM, Now Publicly Available

Today, NVIDIA announces the public release of TensorRT-LLM to accelerate and optimize inference performance for the latest LLMs on NVIDIA GPUs. This open-source...
10 MIN READ
Sep 09, 2023

NVIDIA TensorRT-LLM Supercharges Large Language Model Inference on NVIDIA H100 GPUs

Large language models (LLMs) offer incredible new capabilities, expanding the frontier of what is possible with AI. However, their large size and unique...
9 MIN READ
NVIDIA Jetson Orin modules.
Sep 09, 2023

Leading MLPerf Inference v3.1 Results with NVIDIA GH200 Grace Hopper Superchip Debut

AI is transforming computing, and inference is how the capabilities of AI are deployed in the world’s applications. Intelligent chatbots, image and video...
13 MIN READ
Aug 31, 2023

Deploying YOLOv5 on NVIDIA Jetson Orin with cuDLA: Quantization-Aware Training to Inference

NVIDIA Jetson Orin is the best-in-class embedded platform for AI workloads. One of the key components of the Orin platform is the second-generation Deep...
11 MIN READ
Aug 30, 2023

How to Build a Distributed Inference Cache with NVIDIA Triton and Redis

Caching is as fundamental to computing as arrays, symbols, or strings. Various layers of caching throughout the stack hold instructions from memory while...
13 MIN READ
Decorative image of structured sparsity
Jul 03, 2023

Structured Sparsity in the NVIDIA Ampere Architecture and Applications in Search Engines

Deep learning is achieving significant success in various fields and areas, as it has revolutionized the way we analyze, understand, and manipulate data. There...
13 MIN READ
Jun 28, 2023

How to Deploy an AI Model in Python with PyTriton

AI models are everywhere, in the form of chatbots, classification and summarization tools, image models for segmentation and detection, recommendation models,...
6 MIN READ
Deep learning abstract
Jun 12, 2023

Distributed Deep Learning Made Easy with Spark 3.4

Apache Spark is an industry-leading platform for distributed extract, transform, and load (ETL) workloads on large-scale data. However, with the advent of deep...
7 MIN READ
May 16, 2023

Sparsity in INT8: Training Workflow and Best Practices for NVIDIA TensorRT Acceleration

The training stage of deep learning (DL) models consists of learning numerous dense floating-point weight matrices, which results in a massive amount of...
12 MIN READ
May 04, 2023

Increasing Throughput and Reducing Costs for AI-Based Computer Vision with CV-CUDA

Real-time cloud-scale applications that involve AI-based computer vision are growing rapidly. The use cases include image understanding, content creation,...
11 MIN READ
Apr 25, 2023

Increasing Inference Acceleration of KoGPT with NVIDIA FasterTransformer

Transformers are one of the most influential AI model architectures today and are shaping the direction of future AI R&D. First invented as a tool for...
6 MIN READ