If you’re an application developer or a cluster administrator, you’ve likely seen how non-uniform memory access (NUMA) can impact system performance. When an application is not fully NUMA-aware, performance can be inconsistent and unpredictable.

Because of these challenges, NVIDIA released the Coherent Driver-based Memory Management (CDMM) mode for the NVIDIA Driver on hardware-coherent platforms, such as NVIDIA GH200 Grace Hopper Superchip, NVIDIA GB200 Grace Blackwell Superchip, NVIDIA GB300 Grace Blackwell Ultra Superchip, and NVIDIA Vera Rubin Superchip. 

With CDMM, the NVIDIA driver, not the OS, controls and manages GPU memory. This gives applications finer-grained control over data placement, helping maximize performance.

This blog post describes the differences between NUMA and CDMM, how they can impact application performance, and how to configure either mode. An accompanying whitepaper provides additional information. 

What is NUMA?

NUMA exposes the entire CPU (host) memory and GPU (device) memory to the OS. This means that standard Linux APIs such as malloc and mmap as well as NVIDIA CUDA APIs, can allocate memory on both the CPU and GPU. It also facilitates dynamic memory migration between CPU and GPU via user-space APIs, or automatically by the kernel to optimize resource utilization.  

One important side effect is that NUMA mode treats GPU memory as a generic memory pool, limiting strict isolation of GPU memory from general OS functions. In typical NUMA behavior, memory may spill onto the GPU, which can reduce application performance.  

That’s why NVIDIA provides an alternative: Coherent Driver-based Memory Management (CDMM) mode.

What are hardware-coherent platforms?

Several NVIDIA systems, including the GH200, GB200, GB300, and Vera Rubin, contain direct NVIDIA NVLink-C2C connections between the CPU and the GPU. That introduces a powerful capability not present on PCIe-connected systems: hardware-coherent memory. It enables CPU and GPU memory to be directly addressed from either processor. 

This behavior can have unintended consequences for applications that rely on specific NUMA behavior. In particular, the operating system may use GPU memory unexpectedly, such as for file caching or to satisfy an allocation request that would otherwise cause an out-of-memory (OOM) condition. These differences may be undesirable for Kubernetes-based workloads optimized for a particular layout of CPU and GPU memory. 

CDMM mode addresses these challenges and is particularly useful for Kubernetes-based workloads.

How NUMA impacts Kubernetes

Kubernetes is widely used to operate large GPU clusters, but running it in NUMA mode can produce unexpected behavior. These issues may hurt performance and application functionality.

• Memory over-reporting: Kubernetes incorrectly includes GPU memory in its system memory count, leading to pods requesting more memory than available and causing OOM failures.
• Pod memory limits apply to GPU memory, not just system memory: Kubernetes pod memory limits, designed for system memory, incorrectly apply to both system and GPU memory when system-allocated memory is used, as each GPU is exposed as a NUMA node. This breaks the intended Pod spec API contract. 
• Isolating GPU memory among pods: Kubernetes pods, by default, can access all memory across NUMA nodes, including GPU memory. This allows containers to allocate memory on GPUs that have not been assigned to them, breaking isolation.

For these reasons, CDMM mode is recommended when using Kubernetes.

What is CDMM?

CDMM is an alternative operating mode for the NVIDIA driver that does not expose GPU memory to the operating system as a software NUMA node. Instead, the NVIDIA device driver directly manages GPU memory, separating it from system memory. This approach is inspired by the PCIe-attached GPU model, where GPU memory is distinct from system memory.

In CDMM mode, the Linux kernel manages CPU memory, while the NVIDIA driver manages GPU memory. This separation gives the driver full control over GPU memory, often improving application performance.

How CDMM affects CUDA developers

CDMM primarily affects codes that use Unified Memory. Codes using explicit device allocation and explicit copies are functionally unaffected, though performance may differ.

Under CDMM, system-allocated memory is not migrated to the GPU. The GPU can still access it across the C2C link, but pages will not be migrated. Hints intended to encourage migration, including cudaMemPrefetchAsync() and cudaMemAdvise(SetPreferredLocation), will not move system-allocated pages.

Tuning Unified Memory applications for CDMM

If an application using Unified Memory shows a performance difference under CDMM, the following options may help:

• Compile with -gpu=mem:unified
• Avoid -gpu=mem:unified:nomanagedalloc
• Set NVCOMPILER_ACC_MEMHINTS=DISABLE at runtime

Please report any performance degradation to NVIDIA. For more information, see the CDMM section of the NVIDIA HPC Compilers User’s Guide.

How CDMM affects system administration

When the system is in CDMM mode, there will still be NUMA nodes corresponding to the GPUs, but they will not present any memory to the OS. Using tools such as numactl or mbind won’t have any effect when applied to GPU memory. Do not use these tools in CDMM mode for GPU memory management. They can still be used to manage system memory.

CDMM is currently the default mode for Kubernetes-based NVIDIA GPU Operator deployments beginning with NVIDIA Linux driver 580.65.06 or later. To select a memory management mode, set the appropriate kernel module parameter when loading the driver.  

Guidelines for CDMM and NUMA usage

The following sections highlight the main differences between CDMM and NUMA modes and explain when to consider using each.

Application-specific memory management

• NUMA mode: Best for applications that use OS NUMA APIs and rely on OS management of total system memory (CPU memory and GPU memory).
• CDMM mode: Ideal for applications that need direct GPU memory control and bypass the OS.

Memory pooling

• NUMA mode: Enables GPU and CPU memory to form a larger pool. Workloads benefit from aggregated memory and bandwidth management.
• CDMM mode: The driver manages GPU memory, preventing the OS from incorporating it into a larger pool. GPU memory is dedicated to GPU-specific data.

GPU memory usage: visibility and measurement

• NUMA mode: Standard tools report GPU memory use within the integrated pool and can filter it by NUMA node, providing an overall view of system memory.
• CDMM mode: Provides fine-grained control and visibility into GPU memory. Driver management gives administrators and developers a clear view of GPU memory consumption for performance diagnosis and optimization.

CDMM is the recommended mode for most workloads, including containerized and Kubernetes deployments, on coherent platforms. NUMA mode may be preferred if the workload relies on any of the following behaviors:

• Applications using OS NUMA APIs (numactl, mbind, set_mempolicy) to manage GPU memory placement
• Applications expecting GPU memory to be visible as a NUMA node to the OS
• Unified Memory codes that depend on pages migrating into GPU memory

Beginning with the R615 driver in NVIDIA CUDA Toolkit 13.4, CDMM becomes the default memory management mode on hardware-coherent platforms. NUMA mode remains fully supported and can be configured using NVreg_CoherentGPUMemoryMode.

How to configure memory management mode 

This node-wide administrative setting is controlled through a kernel module parameter. Any change requires either a full driver reload or a reboot to take effect.

Check which mode you are on

grep Coherent /proc/driver/nvidia/params

The memory management mode is set through a kernel module parameter. Write a single option to /etc/modprobe.d/nvidia-openrm.conf:

Enable CDMM mode

echo 'options nvidia NVreg_CoherentGPUMemoryMode=driver' \
  | sudo tee /etc/modprobe.d/nvidia-openrm.conf

Enable NUMA mode

# Loading NVIDIA module instructions

echo 'options nvidia NVreg_CoherentGPUMemoryMode=numa' \
| sudo tee /etc/modprobe.d/nvidia-openrm.conf

# NVIDIA-UVM module parameter

echo 'options nvidia-uvm uvm_disable_sam_migration=false' \
| sudo tee -a /etc/modprobe.d/nvidia-openrm.conf

After setting the desired mode, reload the driver or reboot.

Reload the driver without rebooting

Ensure no GPU workloads are running before proceeding.

# Stop services
sudo systemctl stop nvidia-dcgm.service nvidia-imex.service nvidia-persistenced.service

#Unload the driver
sudo modprobe -rf nvidia_uvm nvidia_drm nvidia_modeset nvidia_fs nvidia

# Reload the driver, the defaults are assumed to come from /etc/modprobe.d via options or the user can specify options if needed
sudo modprobe nvidia <optional arguments>

# Restart the service
sudo systemctl start nvidia-persistenced.service nvidia-imex.service nvidia-dcgm.service

If any module reports “in use”, reboot instead:

sudo reboot

Rebuild initramfs

If the system includes NVIDIA module configuration in initramfs, rebuild after changing the modprobe configuration.

#Ubuntu/Debian-style systems
sudo update-initramfs -u -k all

#Dracut-based systems
sudo dracut --force --regenerate-all

Verify the active mode

grep Coherent /proc/driver/nvidia/params

Expected CDMM output: CoherentGPUMemoryMode: "driver"

Expected NUMA output: CoherentGPUMemoryMode: "numa"

Check for SAM migration

cat /sys/module/nvidia_uvm/parameters/uvm_disable_sam_migration

Expected output “N”

Stuck on CDMM after switching

If a previous CDMM setting persists after removing the modprobe file and rebuilding initramfs:

1. Set NVreg_CoherentGPUMemoryMode=numa explicitly in /etc/modprobe.d/nvidia.conf. This option is present in nvidia.conf, but commented out by default.
2. Rebuild initramfs again, using the commands above.
3. Reboot the node.

Summary

The following table highlights the major differences in how memory is handled between NUMA and CDMM modes.

By understanding and strategically implementing CDMM, developers and administrators can unlock the full potential of NVIDIA hardware-coherent memory architectures, ensuring optimal performance and control for their GPU-accelerated workloads.

If you’re using a hardware-coherent platform such as GH200, GB200, GB300, or Vera Rubin, read the whitepaper for deeper technical details.