LLM Inference Benchmarking: How Much Does Your LLM Inference Cost?

Learn how to calculate LLM inference costs using NVIDIA GenAI-Perf benchmarking tools and TCO formulas. This guide covers performance metrics (TTFT, latency-throughput trade-offs), infrastructure provisioning, and cost calculations per token to optimize deployment ROI. This is the fourth post in the large language model latency-throughput benchmarking series, which aims to instruct developers on how to … Continued

Profiling LLM Training Workflows on NVIDIA Grace Hopper

The rapid advancements in AI have resulted in an era of exponential growth in model sizes, particularly in the domain of large language models (LLMs). These models, with their transformative capabilities, are driving innovation across industries. However, the increasing complexity and computational demands of training such models necessitate a meticulous approach to optimization and profiling.   … Continued

How to Streamline Complex LLM Workflows Using NVIDIA NeMo-Skills

A typical recipe for improving LLMs involves multiple stages: synthetic data generation (SDG), model training through supervised fine-tuning (SFT) or reinforcement learning (RL), and model evaluation. Each stage requires using different libraries, which are often challenging to set up and difficult to use together.  For example, you might use NVIDIA TensorRT-LLM or vLLM for SDG … Continued

Turbocharge LLM Training Across Long-Haul Data Center Networks with NVIDIA Nemo Framework

Multi-data center training is becoming essential for AI factories as pretraining scaling fuels the creation of even larger models, leading the demand for computing performance to outpace the capabilities of a single facility. By distributing workloads across multiple data centers, organizations can overcome limitations in power, cooling, and space, enabling the training of even larger, … Continued

Train a Reasoning-Capable LLM in One Weekend with NVIDIA NeMo

Have you ever wanted to build your own reasoning models such as the open NVIDIA Nemotron, but thought it was too complicated or required massive resources? Think again. With NVIDIA’s powerful tools and datasets, you can train a small, effective reasoning model in about 48 hours, all on a single GPU. Even better, we’ve made … Continued

Streamline LLM Deployment for Autonomous Vehicle Applications with NVIDIA DriveOS LLM SDK

Large language models (LLMs) have shown remarkable generalization capabilities in natural language processing (NLP). They are used in a wide range of applications, including translation, digital assistants, recommendation systems, context analysis, code generation, cybersecurity, and more. In automotive applications, there is growing demand for LLM-based solutions for both autonomous driving and in-cabin features. Deploying LLMs … Continued

Automating Inference Optimizations with NVIDIA TensorRT LLM AutoDeploy

NVIDIA TensorRT LLM enables developers to build high-performance inference engines for large language models (LLMs), but deploying a new architecture traditionally requires significant manual effort. To address this challenge, today we are announcing the availability of AutoDeploy as a beta feature in TensorRT LLM.  AutoDeploy compiles off-the-shelf PyTorch models into inference-optimized graphs. This avoids the … Continued

LLM Model Pruning and Knowledge Distillation with NVIDIA NeMo Framework

Model pruning and knowledge distillation are powerful cost-effective strategies for obtaining smaller language models from an initial larger sibling.  Pruning: Either drop layers (depth-pruning) or drop neurons, attention heads, and embedding channels (width-pruning).  Knowledge distillation: Transfer knowledge from a large teacher model to a smaller student model, with the goal of creating a more efficient, … Continued

A Fine-tuning–Free Approach for Rapidly Recovering LLM Compression Errors with EoRA

Model compression techniques have been extensively explored to reduce the computational resource demands of serving large language models (LLMs) or other large-size neural networks.  However, most existing methods either incur significant accuracy degradation compared to uncompressed models or have long training times. Also, their adaptability is often constrained by a limited range of hardware-supported compression … Continued

Building Your First LLM Agent Application

When building a large language model (LLM) agent application, there are four key components you need: an agent core, a memory module, agent tools, and a planning module. Whether you are designing a question-answering agent, multi-modal agent, or swarm of agents, you can consider many implementation frameworks—from open-source to production-ready. For more information, see Introduction … Continued

Accelerating Hebrew LLM Performance with NVIDIA TensorRT-LLM

Developing a high-performing Hebrew large language model (LLM) presents distinct challenges stemming from the rich and complex nature of the Hebrew language itself. The intricate structure of Hebrew, with words formed through root and pattern combinations, demands sophisticated modeling approaches. Moreover, the lack of capitalization and the frequent absence of punctuation like periods and commas … Continued

NVIDIA NeMo Accelerates LLM Innovation with Hybrid State Space Model Support

Today’s large language models (LLMs) are based on the transformer model architecture introduced in 2017. Since then, rapid advances in AI compute performance have enabled the creation of even larger transformer-based LLMs, dramatically improving their capabilities. Advanced transformer-based LLMs are enabling many exciting applications such as intelligent chatbots, computer code generation, and even chip design. … Continued

Scalable Federated Learning with NVIDIA FLARE for Enhanced LLM Performance

In the ever-evolving landscape of large language models (LLMs), effective data management is a key challenge. Data is at the heart of model performance. While most advanced machine learning algorithms are data-centric, necessary data can’t always be centralized. This is due to various factors such as privacy, regulation, geopolitics, copyright issues, and the sheer effort … Continued

Curating Non-English Datasets for LLM Training with NVIDIA NeMo Curator

Data curation plays a crucial role in the development of effective and fair large language models (LLMs). High-quality, diverse training data directly impacts LLM performance, addressing issues like bias, inconsistencies, and redundancy. By curating high-quality datasets,  we can ensure that  LLMs are accurate, reliable, and generalizable.   When training a localized multilingual LLM, especially for low-resourced languages, … Continued

Curating Custom Datasets for LLM Training with NVIDIA NeMo Curator

Data curation is the first, and arguably the most important, step in the pretraining and continuous training of large language models (LLMs) and small language models (SLMs). NVIDIA recently announced the open-source release of NVIDIA NeMo Curator, a data curation framework that prepares large-scale, high-quality datasets for pretraining generative AI models.  NeMo Curator, which is … Continued