NVIDIA CUDA Profiling Tools Interface (CUPTI) - CUDA Toolkit
The NVIDIA CUDA Profiling Tools Interface (CUPTI), distributed as part of the CUDA Toolkit, is a library that enables the creation of profiling and tracing tools that target CUDA applications. CUPTI provides a set of APIs targeted at ISVs creating profilers and other performance optimization tools:
- the Activity API,
- the Callback API,
- the Host Profiling API,
- the Range Profiling API,
- the PC Sampling API,
- the SASS Metric API,
- the PM Sampling API,
- the Checkpoint API,
- the Profiling API,
- the Event API,
- the Metric API, and
- the Python API (available separately)
Using these CUPTI APIs, independent software developers can create profiling tools that provide low and deterministic profiling overhead on the target system, while giving insight into the CPU and GPU behavior of CUDA applications. Normally packaged with the CUDA Toolkit, NVIDIA occasionally uses this page to provide CUPTI improvements and bug fixes between toolkit releases.
Revision History
Key Features
- Trace CUDA API by registering callbacks for API calls of interest
- Full support for entry and exit points in the CUDA C Runtime (CUDART) and CUDA Driver
- GPU workload trace for the activities happening on the GPU, which includes kernel executions, memory operations (e.g., Host-to-Device memory copies) and memset operations.
- CUDA Unified Memory trace for transfers from host to device, device to host, device to device and page faults on CPU and GPU etc.
- Normalized timestamps for CPU and GPU trace
- Profile hardware and software event counters, including:
- Utilization metrics for various hardware units
- Instruction count and throughput
- Memory load/store events and throughput
- Cache hits/misses
- Branches and divergent branches
- Many more
- Enables automated bottleneck identification based on metrics such as instruction throughput, memory throughput, and more
- Range profiling to enable metric collection over concurrent kernel launches within a range
- Metrics attribution at the high-level source code and the executed assembly instructions.
- Device-wide sampling of the program counter (PC). The PC Sampling gives the number of samples for each source and assembly line with various stall reasons.
Updates in CUDA Toolkit 12.8
- Added support for the Blackwell architecture.
- Blackwell architecture introduces a new hardware method called as HW events for collecting CUDA kernel timestamps.
This method provides more accurate timestamps compared to the traditional SW instrumentation and semaphore based approach,
and it is expected to add minimal overhead to the application. The new API
cuptiActivityEnableHWTrace
enables the collection of kernel timestamps through HW events. In CUDA 12.8, this feature is experimental and subject to certain limitations:- It is not supported on MPS, WSL, MCDM, MIG, or vGPU setups.
- On Windows, only a single context can be profiled at a time, while Linux allows tracing multiple contexts simultaneously.
- Late attach functionality is not supported.
- Added support to provide GPU timestamps for the CUDA event record completion. The activity record
CUpti_ActivityCudaEvent
is deprecated and has been replaced by the new activity recordCUpti_ActivityCudaEvent2
. The new fielddeviceTimestamp
provides the device-side timestamp and the fieldcudaEventSyncId
provides a unique ID to associate event synchronization records with the latest CUDA Event record. - The activity kind
CUPTI_ACTIVITY_KIND_SYNCHRONIZATION
doesn t provide records for CUDA synchronization operations which return non-zero CUDA status. The APIcuptiActivityEnableAllSyncRecords
can be used to enable/disable collection of all CUDA synchronization operations which return non-zero CUDA status. - Profiling of NVLINK (nvl*), C2C and PCIe metrics are now supported using Profiler Host API and Range Profiler APIs. These metrics are collected at the device-level. These can t be collected with the metrics which are collected at the context-level.
- Added a new field
copyCount
in the memcpy activity record to provide the number of copies performed by the batched memcpy operation. The activity recordCUpti_ActivityMemcpy5
is replaced by a new activity recordCUpti_ActivityMemcpy6
. - For PM Sampling, users can configure the buffer append mode to either retain the latest or the oldest records in the event of a buffer overflow.
Refer to the
CUpti_PmSampling_HardwareBuffer_AppendMode
enum for additional details. - For unified memory profiling, the bitmask of devices involved in the operation is provided using the field
processors
. The activity recordCUpti_ActivityUnifiedMemoryCounter2
is deprecated and has been replaced by the new activity recordCUpti_ActivityUnifiedMemoryCounter3
. - To provide additional information for graph level trace for device launched graphs, a new activity record
CUpti_ActivityDeviceGraphTrace
and activity kindCUPTI_ACTIVITY_KIND_DEVICE_GRAPH_TRACE
are introduced. The activity kindCUPTI_ACTIVITY_KIND_DEVICE_GRAPH_TRACE
can not be directly enabled or disabled. It is enabled when activity kindCUPTI_ACTIVITY_KIND_GRAPH_TRACE
is enabled and device graph trace is enabled through the APIcuptiActivityEnableDeviceGraph()
. - Added a new overhead kind
CUPTI_ACTIVITY_OVERHEAD_UVM_ACTIVITY_INIT
to report the overhead incurred while initializing the Unified Memory profiling. - Enum
CUpti_PcieGen
is extended to include PCIe Gen 6. - The CUPTI Event API from the header
cupti_events.h
and the CUPTI Metric API from the headercupti_metrics.h
are deprecated in the CUDA 12.8 release and will be removed in a future CUDA release. It is recommended to use the CUPTI Range Profiling API as an alternative. For more information, refer to the section Evolution of the profiling APIs. - The PC Sampling Activity API and the source/SASS level metrics from the header
cupti_activity.h
will not be supported on Blackwell and future GPU architectures. These features were deprecated in earlier releases. - Resolved an issue where PM Sampling data was being collected on a different GPU than the intended one in multi-GPU systems.
- Fixed the issue that activity records for conditional nodes in the graph were not generated for Ampere and later architectures.
- If the device is in MIG mode, the
uuid
field in the device and NVLINK records returns the MIG UUID which uniquely identifies the subscribed MIG compute instance. This needs a matching driver from the 12.8 toolkit. - CUPTI generates external correlation records for CUDA APIs even when CUDA API tracing is disabled using the
cuptiActivityEnableRuntimeApi
orcuptiActivityEnableDriverApi
API.
New Features
Deprecated and dropped features
Resolved Issues
Requirements
- Linux x86_64[1]
- Windows x86_64[1]
- Linux aarch64 SBSA[1]
- DRIVE OS QNX aarch64[2]
- DRIVE OS Linux aarch64[2]
- Activity and Callback APIs
- All architectures supported by CUDA Toolkit
- Event and Metric APIs (this API and associated GPUs have been deprecated as of CUDA 12.8, and will be dropped in a future release)
- Volta
- Pascal
- Maxwell
- Profiling and PC Sampling APIs
- Blackwell: B100, GB10x
- Hopper: GH100
- Ada: AD10x
- Ampere: A100 with Multi-Instance GPU, GA10x
- Turing
- Volta: GV100, GV10B (Volta GPUs have been deprecated as of CUDA 12.8, and will be dropped in a future release)
- CUPTI can be found in the CUDA Toolkit 12.8 production release
- 571.96 (Windows)
- 570.86.10 (Linux)
Supported platforms
[2] available in the Embedded or Drive toolkits only
Supported NVIDIA GPU architectures
CUDA Toolkit
Drivers
-
Please use the following drivers
Documentation
Support
To provide feedback, request additional features, or report issues, please use the Developer Forums.
Installation Overview
When installing CUDA Toolkit 12.8 and specifying options, be sure to select CUDA > Development > Tools > CUPTI.