Every byte moved has a cost. As model checkpoints grow to hundreds of gigabytes or even a terabyte, that cost adds up quickly. To make things even worse, moving these model weights around the cluster is extremely common. For instance, a cold start may pull weights from remote storage into GPU memory; autoscaling and rolling updates must populate each new replica; and RL post-training continuously moves updated weights from trainers to roll out workers. These may look like different workflows, but they impose the same recurring tax: time spent moving weights before useful work can begin.
ModelExpress: Accelerating the model weight lifecycle
NVIDIA ModelExpress (MX) is built around a simple idea: Before loading a model, first ask where a compatible copy of its weights already lives. Rather than treating every replica as an independent cold start, MX chooses the fastest available source and transfer path.
When a serving peer already holds compatible weights in GPU, MX transfers them directly from GPU to GPU over P2P RDMA via NVIDIA Inference Xfer Library (NIXL), bypassing redundant access to object storage, local disk, and host memory. When no peer is available, MX bootstraps from the fastest supported path by streaming from an object store without landing on disk or reading local files directly into GPU memory.
MX transfers DeepSeek-V4 Pro weights and JIT Kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing the total startup time to 1 minute 44 seconds from 8 minutes. The rest of the post shows how MX selects the fastest available path to GPU memory, prioritizing P2P RDMA from a serving replica and eliminating redundant downloads and copies along the way. It then extends the same approach to reusing kernel caches and distributing RL weight updates.
Accelerating every stage from remote storage to GPU memory
Every new worker must get its weights from one of three places: remote storages (e.g. HF or S3), local storage, or another worker already serving the model. For the first worker, there is no peer yet, so it must bootstrap from storage. MX can stream the checkpoints from object storage or load it from fast local storage, removing avoidable copies along either path.
Once that first worker is serving, the preferred source changes. Its weights are already resident, post-processed, and laid out in GPU memory, so every compatible worker after it should load directly from that peer over P2P RDMA. MX makes this transition automatically: bootstrap once from storage, then scale out GPU to GPU, falling back to storage only when no compatible peer is available.
Starting the first worker: Bootstrap from storage
Remote object storage to GPU: Avoiding local disk
When the checkpoint lives in a cloud bucket and you would rather not provision and manage a disk cache tier, MX uses the Model Streamer to pull safetensors through a reusable CPU staging buffer and into GPU. The checkpoint never lands on local disk, eliminating the intermediate download, reload, and storage volume.
Model Streamer uses a multithreaded tensor reader to fetch tensor ranges concurrently across checkpoint shards. As tensors arrive, it pipelines remote reads with GPU placement: completed tensors are passed to the inference engine while later tensors are still being fetched. This keeps the storage, network, and GPU copy paths busy while reusing a bounded amount of host memory.
In tensor-parallel deployments, the participating ranks divide the remote reads and share the results, typically over NCCL, instead of having every rank download the full checkpoint independently. MX connects this distributed stream directly to the inference engine’s weight loader, preparing the first worker to become the P2P source for every compatible replica that follows.
Cluster ingress: Download once, not N times
When a cluster maintains a shared disk cache tier (e.g. persistent volumes in K8s), MX ensures that the fleet populates it only once. If 10 replicas concurrently want to fetch the 806 GiB DeepSeek-V4 Pro model, they will need to pull roughly 8 TiB of identical data across the network while competing for the same ingress bandwidth. The MX Model Cache Service collapses those requests into one coordinated download: an atomic claim in Metadata Store selects a downloader, while the remaining replicas track its progress and reuse the cached copy. The cluster pays the external download cost once, then every replica can begin from the same cached checkpoint.
Local storage to GPU: Bypassing host-memory staging
When GPUDirect Storage (GDS) is supported in the system, MX reads checkpoint files directly from local storage into GPU memory through NIXL’s multithreaded GDS backend. NIXL executes batched tensor reads in parallel directly into GPU memory, bypassing host memory and the staging copy required by a conventional loader. Users don’t need to enable GDS explicitly: MX detects the capability automatically and falls back to another loading strategy when it is unavailable.
Local storage to GPU: Pipelining local reads with ModelStreamer
MX can also load local checkpoints through ModelStreamer. Multiple OS threads read safetensors concurrently into a configurable CPU buffer while completed tensors move to the GPU and later reads continue in parallel. Unlike GDS, this path still stages through host memory, but it overlaps disk I/O with GPU placement, benefits from the OS page cache, and provides a portable fast path when direct storage-to-GPU access is unavailable.
Starting every worker after the first: Fetch from a serving peer
This is the key feature of MX. Once another replica is already serving the same model, the weights have completed most of their journey: they are resident in GPU memory, post-processed, and laid out for the inference engine. MX treats that replica as a live weight source. After confirming compatibility, it transfers the tensors directly from the source GPU to the target GPU. Once its weights are loaded, the new replica joins the source pool, giving subsequent replicas another peer to load from. With every successful transfer, that pool grows alongside the deployment, turning scale-out into GPU-to-GPU fan-out instead of repeated cold loads.
The MX control plane discovers compatible peers, exchanges transfer metadata, and tracks source readiness, but never handles the weight bytes themselves. On the data plane, MX uses NIXL as a default transfer engine whose pluggable backends allow for peak performance across a variety of networks, such as Infiniband, RoCE, NVLink, EFA, etc. MX has a first-class transport interface that allows libraries such as fabric-lib and standalone Mooncake to integrate with MX.
Before any transfer begins, MX computes an mx_source_id from the model and runtime settings that determine tensor layout, then considers only peers with a matching ID. The control plane discovers those peers through Redis, Kubernetes CRDs, or k8s-service (serverless) metadata backends.Â
Optimizing NIXL memory registration overhead
Before NIXL can RDMA a tensor, the GPU memory backing it has to be registered: an ibv_reg_mr call that returns the Remote Key (rkey) used for remote access. A large model has tens of thousands of tensors, and registering them one at a time is slow enough to show up in the budget. By default, MX registers each tensor individually. Two opt-in strategies reduce that registration cost:
- Pool registration registers each underlying
cudaMallocallocation once instead of each tensor, cutting registration count by 80 to 99 percent on typical models with no change to transfer semantics. - VMM arena registration goes further. It installs a
CUDAPluggableAllocatorthat routes every load-time allocation into a single 16 TiB virtual-address arena, then registers the whole used range as one dmabuf-backed memory region at end of load. Registration collapses from one call per tensor to one call, total; each tensor descriptor simply carries an offset into that single region.
Using DeepSeek-V4-Pro TP=8 on the vLLM engine, as shown in Figure 3, below, we measured the average NIXL registration time for each approach.
Runtime path selection and safe fallback
At startup, MX probes the available capabilities, automatically skipping any path the environment does not support. The first applicable strategy runs in the current priority order: P2P RDMA -> ModelStreamer -> GDS -> default loader (host-staged POSIX I/O). If a path is unavailable or fails before modifying the model state, MX falls through automatically. If a failure occurs after weights have begun landing, it reinitializes the model before continuing, so partially written weights are never served.Â
P2P retries alternate peers only for metadata failures before transfer begins, and the native loader remains the final fallback. This capability-driven design keeps the MX core hardware and software agnostic, with platform-specific fast paths enabled only where supported.
End-to-end results
We ran DeepSeek-V4-Pro on an 8xB200 GPU node with NVIDIA ConnectX-7 NICs and compared the total model loading time across different cold start scenarios. Each replica used vLLM 0.23.0 with TP=8 and --enable-flashinfer-autotune. See Figure 4, below.Â
Warm, not just loaded: Inheriting the compiled kernels
Getting weights into GPU memory is critical, but a loaded model is not yet ready to serve. During its first forward passes, the engine JIT-compiles and autotunes kernels (e.g. torch.compile, Triton, DeepGEMM, TileLang, and etc.) and captures CUDA graphs for the exact model, dtype, quantization, and GPU. For models such as DeepSeek-V4 Pro, this can take several minutes and can become the dominant startup cost once MX reduces weight-loading latency (see Figure 5, below).
That repeated warmup is avoidable. When the model, software stack, and GPU architecture match, one replica can pay the compilation cost and the rest can inherit the resulting caches.
MX’s Artifact Transfer API packages these file-backed artifacts, transfers them directly between registered host-memory buffers over NIXL’s CPU-to-CPU RDMA path, then verifies and installs them in the target engine’s cache directory. This eliminates the need for a shared ReadWriteMany (RWX) volume in Kubernetes, while an artifact-specific mx_source_id prevents reuse across incompatible replicas. MX detects standard cache locations automatically when a Redis or Kubernetes metadata backend is configured.
We ran using the same setup to measure how much the kernel artifact transfer can reduce the startup time. The artifact-enabled run transferred the Triton/DeepGEMM/TileLang/CuTe DSL/FlashInfer caches. The chart compares the major startup stages and total wall-clock time from process start until the API was ready.
When the weights change every Step: RL post-training
Everything so far assumes a model’s weights are fixed once loaded. RL post-training breaks that assumption. A trainer updates the policy every step, and the inference actors generating rollouts must pick up those weights before the next round of generation. As with inference startup, weight movement is on the critical path in RL: rollout workers wait while updated weights move from the trainer’s distributed layout (whether FSDP/DTensor shards or Megatron TP, PP, and EP partitions) into the inference engine’s layout.
MX drives the refit through the following four stages:
- Publish: Each trainer rank advertises the tensors or shards it already owns, together with metadata describing their shape, dtype, placement, and parameter mapping to MX.
- Discover: A rollout worker looks up the requested weight version and its available sources through MX.
- Plan: The receiver maps the published ownership information onto its own target layout and identifies which sources contain the required tensors or ranges.
- Pull, convert, and load: The receiver issues one-sided reads directly against those sources.
MX includes the core building blocks for receiver-driven refit, and customers are evaluating them in active integrations. We are also testing delta weight diff refits for cross-cluster weight transfer, a technique used by Fireworks/Cursor, Cognition, and more in recent RL runs.
Contributing to Dynamo and our roadmap
MX has native integrations with vLLM and SGLang and supports serving frameworks including Dynamo and llm-d.
The Dynamo open source community is actively working toward deeper TensorRT-LLM integration and broader inference capabilities. Explore the current Dynamo documentation and roadmap, try the available workflows in your own environment, and contribute feedback to help shape the project’s direction.
Acknowledgments
ModelExpress is a team effort. Thank you to the rest of the MX team, Zhongdongming Dai and Tanushriya Singh for their core work on the project. We’re grateful to Itay Neeman, Anish Maddipoti, Istvan Haller, and Omri Kahalon for their guidance on the project’s technical direction, and Will Eaton at Red Hat for his support on the llm-d integration.
