TensorRT

Nov 21, 2024

NVIDIA TensorRT-LLM Multiblock Attention Boosts Throughput by More Than 3x for Long Sequence Lengths on NVIDIA HGX H200

Generative AI models are advancing rapidly. Every generation of models comes with a larger number of parameters and longer context windows. The Llama 2 series...

5 MIN READ

Nov 19, 2024

Llama 3.2 Full-Stack Optimizations Unlock High Performance on NVIDIA GPUs

Meta recently released its Llama 3.2 series of vision language models (VLMs), which come in 11B parameter and 90B parameter variants. These models are...

6 MIN READ

Nov 15, 2024

NVIDIA NIM 1.4 Ready to Deploy with 2.4x Faster Inference

The demand for ready-to-deploy high-performance inference is growing as generative AI reshapes industries. NVIDIA NIM provides production-ready microservice...

3 MIN READ

Nov 08, 2024

5x Faster Time to First Token with NVIDIA TensorRT-LLM KV Cache Early Reuse

In our previous blog post, we demonstrated how reusing the key-value (KV) cache by offloading it to CPU memory can accelerate time to first token (TTFT) by up...

5 MIN READ

Nov 06, 2024

Spotlight: Fourier Trains Humanoid Robots for Real-World Roles Using NVIDIA Isaac Gym

This post was written in partnership with the Fourier research team. Training humanoid robots to operate in fields that demand high levels of interaction and...

4 MIN READ

Nov 01, 2024

3x Faster AllReduce with NVSwitch and TensorRT-LLM MultiShot

Deploying generative AI workloads in production environments where user numbers can fluctuate from hundreds to hundreds of thousands – and where input...

5 MIN READ

Oct 07, 2024

Optimizing Microsoft Bing Visual Search with NVIDIA Accelerated Libraries

Microsoft Bing Visual Search enables people around the world to find content using photographs as queries. The heart of this capability is Microsoft's TuringMM...

11 MIN READ

Sep 25, 2024

Deploying Accelerated Llama 3.2 from the Edge to the Cloud

Expanding the open-source Meta Llama collection of models, the Llama 3.2 collection includes vision language models (VLMs), small language models (SLMs), and an...

6 MIN READ

Sep 24, 2024

Developing Next-Generation Wireless Networks with NVIDIA Aerial Omniverse Digital Twin

The journey to 6G has begun, offering opportunities to deliver a network infrastructure that is performant, efficient, resilient, and adaptable. 6G networks...

9 MIN READ

Sep 23, 2024

Using Generative AI to Enable Robots to Reason and Act with ReMEmbR

Vision-language models (VLMs) combine the powerful language understanding of foundational LLMs with the vision capabilities of vision transformers (ViTs) by...

10 MIN READ

Sep 19, 2024

Just Released: Torch-TensorRT v2.4.0

Includes C++ runtime support in Windows Support, Enhanced Dynamic Shape support in Converters, PyTorch 2.4, CUDA 12.4, TensorRT 10.1, Python 3.12.

1 MIN READ

An image of a person working in a greenhouse.

Sep 11, 2024

AI Tool Helps Farmers Combat Crop Loss and Climate Change

Machine Learning algorithms are beginning to revolutionize modern agriculture. Enabling farmers to combat pests and diseases in real time, the technology is...

3 MIN READ

Sep 10, 2024

Post-Training Quantization of LLMs with NVIDIA NeMo and NVIDIA TensorRT Model Optimizer

As large language models (LLMs) are becoming even bigger, it is increasingly important to provide easy-to-use and efficient deployment paths because the cost of...

10 MIN READ

Sep 05, 2024

Low Latency Inference Chapter 1: Up to 1.9x Higher Llama 3.1 Performance with Medusa on NVIDIA HGX H200 with NVLink Switch

As large language models (LLMs) continue to grow in size and complexity, multi-GPU compute is a must-have to deliver the low latency and high throughput that...

5 MIN READ

Aug 28, 2024

Boosting Llama 3.1 405B Performance up to 1.44x with NVIDIA TensorRT Model Optimizer on NVIDIA H200 GPUs

The Llama 3.1 405B large language model (LLM), developed by Meta, is an open-source community model that delivers state-of-the-art performance and supports a...

7 MIN READ

Aug 28, 2024

NVIDIA Blackwell Platform Sets New LLM Inference Records in MLPerf Inference v4.1

Large language model (LLM) inference is a full-stack challenge. Powerful GPUs, high-bandwidth GPU-to-GPU interconnects, efficient acceleration libraries, and a...

13 MIN READ