Triton Inference Server

Decorative image.
Aug 28, 2024

NVIDIA Triton Inference Server Achieves Outstanding Performance in MLPerf Inference 4.1 Benchmarks

Six years ago, we embarked on a journey to develop an AI inference serving solution specifically designed for high-throughput and time-sensitive production use...
8 MIN READ
Aug 28, 2024

NVIDIA Blackwell Platform Sets New LLM Inference Records in MLPerf Inference v4.1

Large language model (LLM) inference is a full-stack challenge. Powerful GPUs, high-bandwidth GPU-to-GPU interconnects, efficient acceleration libraries, and a...
13 MIN READ
An illustration of a chatbot.
Aug 21, 2024

Practical Strategies for Optimizing LLM Inference Sizing and Performance

As the use of large language models (LLMs) grows across many applications, such as chatbots and content creation, it's important to understand the process of...
2 MIN READ
Aug 06, 2024

Accelerating Hebrew LLM Performance with NVIDIA TensorRT-LLM

Developing a high-performing Hebrew large language model (LLM) presents distinct challenges stemming from the rich and complex nature of the Hebrew language...
8 MIN READ
Decorative image.
Aug 01, 2024

Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API

NVIDIA offers tools like Perf Analyzer and Model Analyzer to assist machine learning engineers with measuring and balancing the trade-off between latency and...
6 MIN READ
Jul 16, 2024

New Workshops: Customize LLMs, Build and Deploy Large Neural Networks

Register now for an instructor-led public workshop in July, August or September. Space is limited.
1 MIN READ
An illustration of a NIM use case.
Jul 08, 2024

Deploy Multilingual LLMs with NVIDIA NIM

Multilingual large language models (LLMs) are increasingly important for enterprises operating in today's globalized business landscape. As businesses expand...
9 MIN READ
Jul 02, 2024

Achieving High Mixtral 8x7B Performance with NVIDIA H100 Tensor Core GPUs and NVIDIA TensorRT-LLM

As large language models (LLMs) continue to grow in size and complexity, the performance requirements for serving them quickly and cost-effectively continue to...
9 MIN READ
Jun 14, 2024

Level Up Your Skills with Five New NVIDIA Technical Courses

With AI introducing an unprecedented pace of technological innovation, staying ahead means keeping your skills up to date. The NVIDIA Developer Program gives...
4 MIN READ
Picture of a clothing store.
May 17, 2024

Enhancing the Apparel Shopping Experience with AI, Emoji-Aware OCR, and Snapchat's Screenshop

Ever spotted someone in a photo wearing a cool shirt or some unique apparel and wondered where they got it? How much did it cost? Maybe you've even thought...
8 MIN READ
nearly 100 training labs from GTC available on demand
May 07, 2024

NVIDIA GTC Training Labs On Demand Available Now

Missed GTC or want to replay your favorite training labs? Find it on demand with the NVIDIA GTC Training Labs playlist.
1 MIN READ
Llama 3 Performance with NVIDIA TensorRT-LLM and NVIDIA Triton Inference Server
Apr 28, 2024

Turbocharging Meta Llama 3 Performance with NVIDIA TensorRT-LLM and NVIDIA Triton Inference Server

We're excited to announce support for the Meta Llama 3 family of models in NVIDIA TensorRT-LLM, accelerating and optimizing your LLM inference performance. You...
9 MIN READ
Apr 02, 2024

Tune and Deploy LoRA LLMs with NVIDIA TensorRT-LLM

Large language models (LLMs) have revolutionized natural language processing (NLP) with their ability to learn from massive amounts of text and generate fluent...
15 MIN READ
Mar 18, 2024

Translate Your Enterprise Data into Actionable Insights with NVIDIA NeMo Retriever

Across every industry, and every job function, generative AI is activating the potential within organizations—turning data into knowledge and empowering...
9 MIN READ
Four images of products against enhanced backgrounds.
Mar 07, 2024

Generate Stunning Images with Stable Diffusion XL on the NVIDIA AI Inference Platform

Diffusion models are transforming creative workflows across industries. These models generate stunning images based on simple text or image inputs by...
14 MIN READ
Decorative image of inference steps: LLM, optimize, deploy. The GTC logo is in one corner.
Feb 13, 2024

Top Inference for Large Language Models Sessions at NVIDIA GTC 2024

Learn how inference for LLMs is driving breakthrough performance for AI-enabled applications and services.
1 MIN READ