NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: nvprof Permission issue with Performance Counters

Overview

This page provides an explanation and solutions for the following error message from the NVIDIA nvprof command line profiler:
   The user does not have permission to profile on the target device.
For general information about this error including Solutions and Administration, please see NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters.

Solutions for this issue

To get the desired behavior (see chart below) when profiling and tracing,
  1. Have a system administrator enable access to the NVIDIA performance counters using the Administration instructions
  2. Run the application being profiled with administrative privileges
  • On Windows targets: Launch the target application by right-clicking on the application and selecting "Run as administrator"
  • On Linux targets: Launch the target application with 'sudo' or as a user with the CAP_SYS_ADMIN capability set

Behavior

For the nvprof command line profiler found in the CUDA Toolkit version 10.2

Mode[1] User[2] Expected Behavior
Restricted Profiling Admin[3] Profiling and tracing produce normal results
Restricted Profiling non-Admin Tracing produces normal results
No Profiling is possible. All attempts to profile will result in output with only the following error message:

The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM"
Unrestricted Profiling Admin[3] Profiling and tracing produce normal results
Unrestricted Profiling non-Admin Profiling and tracing produce normal results
[1] Mode set by system administrator using the Administration instructions
[2] Privilege level of user running target application being profiled on target device
[3] Admin user privilege achieved by
  • On Windows targets: Launching the target application by right-clicking on the application and selecting "Run as administrator"
  • On Linux targets: Launching the target application with 'sudo' or as a user with the CAP_SYS_ADMIN capability set
For the nvprof command line profiler found in the CUDA Toolkit version 10.1

Mode[1] User[2] Expected Behavior
Restricted Profiling Admin[3] Profiling and tracing produce normal results
Restricted Profiling non-Admin No Profiling or Tracing is possible. All attempts to profile will result in output with only the following error message:

The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM"
Unrestricted Profiling Admin[3] Profiling and tracing produce normal results
Unrestricted Profiling non-Admin Profiling and tracing produce normal results
[1] Mode set by system administrator using the Administration instructions
[2] Privilege level of user running target application being profiled on target device
[3] Admin user privilege achieved by
  • On Windows targets: Launching the target application by right-clicking on the application and selecting "Run as administrator"
  • On Linux targets: Launching the target application with 'sudo' or as a user with the CAP_SYS_ADMIN capability set