NVIDIA CUDA Profiling Tools Interface (CUPTI) - CUDA Toolkit

The NVIDIA CUDA Profiling Tools Interface (CUPTI) is a library that enables the creation of profiling and tracing tools that target CUDA applications. CUPTI provides a set of APIs targeted at ISVs creating profilers and other performance optimization tools:

the Activity API,
the Callback API,
the Host Profiling API,
the Range Profiling API,
the PC Sampling API,
the SASS Metric API,
the PM Sampling API,
the Checkpoint API,
the Profiling API,
the Python API (available separately)

Using these CUPTI APIs, independent software developers can create profiling tools that provide low and deterministic profiling overhead on the target system, while giving insight into the CPU and GPU behavior of CUDA applications. Normally packaged with the CUDA Toolkit, NVIDIA occasionally uses this page to provide CUPTI improvements and bug fixes between toolkit releases.

There is currently no CUPTI update to the CUDA Toolkit 13.4. You may obtain the latest version of CUPTI by Downloading the CUDA Toolkit 13.4.1

Download the CUDA Toolkit 13.4 Now
Download the CUPTI Python API 13.3.0 Now

Revision History

Key Features

Trace CUDA API by registering callbacks for API calls of interest

Full support for entry and exit points in the CUDA C Runtime (CUDART) and CUDA Driver

GPU workload trace for the activities happening on the GPU, which includes kernel executions, memory operations (e.g., Host-to-Device memory copies) and memset operations.
CUDA Unified Memory trace for transfers from host to device, device to host, device to device and page faults on CPU and GPU etc.
Normalized timestamps for CPU and GPU trace
Profile hardware and software event counters, including:

Utilization metrics for various hardware units
Instruction count and throughput
Memory load/store events and throughput
Cache hits/misses
Branches and divergent branches
Many more

Enables automated bottleneck identification based on metrics such as instruction throughput, memory throughput, and more
Range profiling to enable metric collection over concurrent kernel launches within a range
Metrics attribution at the high-level source code and the executed assembly instructions.
Device-wide sampling of the program counter (PC). The PC Sampling gives the number of samples for each source and assembly line with various stall reasons.

See the CUPTI User Guide for a complete listing of hardware and software event counters available for performance analysis tools.

Updates in CUDA Toolkit 13.4

New Features

Added support for the Rubin GPU architecture.
Fixed Clock Rate - Added APIs cuptiClockControlLock() and cuptiClockControlUnlock() with CUPTI_CLOCK_CONTROL_MODE_BASE or CUPTI_CLOCK_CONTROL_MODE_BOOST to lock GPU clocks and eliminate clock-rate variability during profiling. Refer to the section Fixed Clock Rate for more details.
Cache State Reset - Range Profiler can now reset L2 (or all) cache before each replay pass via the cacheControlMode field in CUpti_RangeProfiler_Enable_Params. Refer to the section Cache State Reset for more details.
Added sourceGraphId to provide the unique graph ID of the node from which the kernel node was instantiated or last updated and sourceGraphNodeId which provides unique graph node ID of the node from which the kernel node was instantiated or last updated. The activity record CUpti_ActivityKernel12 is deprecated and replaced by CUpti_ActivityKernel13.
To ensure tracing and profiling accuracy, live vGPU migration is blocked while CUPTI is active and resumes only after cuptiFinalize() is called or the process exits.
Added workqueueChannelCount and workqueueChannelIds fields to the green context activity record to report the number of HW channels assigned to a green context s work queue and their channel IDs, respectively. The activity record CUpti_ActivityGreenContext2 is deprecated and replaced by CUpti_ActivityGreenContext3.
CUPTI library installation path: For Windows, libraries are now placed under architecture-specific directories instead of the generic lib64 folder to support proper cross-compile support. For ex: on a x86_64 system, the CUPTI libraries are installed under C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA<version>\extras\CUPTI\lib\x64.

Resolved Issues

Per-subscriber timestamp callback registration via CUPTI_ACTIVITY_ATTR_TIMESTAMP_CALLBACK using cuptiActivitySetAttribute_v2 is now supported in multi-subscriber mode, allowing tools to register their own callbacks rather than sharing a common callback.
Fixed issues that could occur after disabling user-defined records via the CUPTI_ACTIVITY_ATTR_USER_DEFINED_RECORDS attribute.
Fixed an issue that sometimes causes a brief (~2 s) CUPTI block when disabling the CUPTI_ACTIVITY_KIND_UNIFIED_MEMORY_COUNTER activity.
Fixed a host heap corruption issue that could occur when using CUPTI HES trace (cuptiActivityEnableHWTrace).
Fixed CUPTI HES activity so that graphNodeId field retains the full graphId (upper 32-bits) in CUDA-graph memory copy records.