TensorRT-LLM

Feb 14, 2025

Optimizing Qwen2.5-Coder Throughput with NVIDIA TensorRT-LLM Lookahead Decoding

Large language models (LLMs) that specialize in coding have been steadily adopted into developer workflows. From pair programming to self-improving AI agents,...
7 MIN READ
Feb 05, 2025

Improving Translation Quality with Domain-Specific Fine-Tuning and NVIDIA NIM

Translation plays an essential role in enabling companies to expand across borders, with requirements varying significantly in terms of tone, accuracy, and...
8 MIN READ
Jan 24, 2025

Optimize AI Inference Performance with NVIDIA Full-Stack Solutions

The explosion of AI-driven applications has placed unprecedented demands on both developers, who must balance delivering cutting-edge performance with managing...
9 MIN READ
Jan 16, 2025

Introducing New KV Cache Reuse Optimizations in NVIDIA TensorRT-LLM

Language models generate text by predicting the next token, given all the previous tokens including the input text tokens. Key and value elements of the...
7 MIN READ
Jan 15, 2025

GPU Memory Essentials for AI Performance

Generative AI has revolutionized how people bring ideas to life, and agentic AI represents the next leap forward in this technological evolution. By leveraging...
6 MIN READ
Dec 18, 2024

NVIDIA TensorRT-LLM Now Supports Recurrent Drafting for Optimizing LLM Inference

Recurrent drafting (referred as ReDrafter) is a novel speculative decoding technique developed and open-sourced by Apple for large language model (LLM)...
6 MIN READ
Dec 17, 2024

Boost Llama 3.3 70B Inference Throughput 3x with NVIDIA TensorRT-LLM Speculative Decoding

Meta's Llama collection of open large language models (LLMs) continues to grow with the recent addition of Llama 3.3 70B, a text-only...
8 MIN READ
Dec 16, 2024

Top Posts of 2024 Highlight NVIDIA NIM, LLM Breakthroughs, and Data Science Optimization

2024 was another landmark year for developers, researchers, and innovators working with NVIDIA technologies. From groundbreaking developments in AI inference to...
4 MIN READ
Chat avatar between tiles with computer activity icons, on a black background.
Dec 11, 2024

NVIDIA TensorRT-LLM Now Accelerates Encoder-Decoder Models with In-Flight Batching

NVIDIA recently announced that NVIDIA TensorRT-LLM now accelerates encoder-decoder model architectures. TensorRT-LLM is an open-source library that optimizes...
4 MIN READ
Dec 05, 2024

Spotlight: Perplexity AI Serves 400 Million Search Queries a Month Using NVIDIA Inference Stack

The demand for AI-enabled services continues to grow rapidly, placing increasing pressure on IT and infrastructure teams. These teams are tasked with...
7 MIN READ
Image of the TensorRT-LLM icon next to multiple other icons of computer activities.
Dec 02, 2024

TensorRT-LLM Speculative Decoding Boosts Inference Throughput by up to 3.6x

NVIDIA TensorRT-LLM support for speculative decoding now provides over 3x the speedup in total token throughput. TensorRT-LLM is an open-source library that...
9 MIN READ
Nov 22, 2024

Spotlight: TCS Increases Automotive Software Testing Speeds by 2x Using NVIDIA Generative AI

Generative AI is transforming every aspect of the automotive industry, including software development, testing, user experience, personalization, and safety....
8 MIN READ
Image of an HGX H200
Nov 21, 2024

NVIDIA TensorRT-LLM Multiblock Attention Boosts Throughput by More Than 3x for Long Sequence Lengths on NVIDIA HGX H200

Generative AI models are advancing rapidly. Every generation of models comes with a larger number of parameters and longer context windows. The Llama 2 series...
5 MIN READ
Nov 19, 2024

Llama 3.2 Full-Stack Optimizations Unlock High Performance on NVIDIA GPUs

Meta recently released its Llama 3.2 series of vision language models (VLMs), which come in 11B parameter and 90B parameter variants. These models are...
6 MIN READ
Nov 15, 2024

NVIDIA NIM 1.4 Ready to Deploy with 2.4x Faster Inference

The demand for ready-to-deploy high-performance inference is growing as generative AI reshapes industries. NVIDIA NIM provides production-ready microservice...
3 MIN READ
Nov 15, 2024

Streamlining AI Inference Performance and Deployment with NVIDIA TensorRT-LLM Chunked Prefill

In this blog post, we take a closer look at chunked prefill, a feature of NVIDIA TensorRT-LLM that increases GPU utilization and simplifies the deployment...
4 MIN READ