NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: nvprof Permission issue with Performance Counters

Overview

This page provides an explanation and solutions for the following error message from the NVIDIA nvprof command line profiler:
   The user does not have permission to profile on the target device.
For general information about this error including Solutions and Administration, please see NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters.

Solutions for this issue

To get the desired behavior (see chart below) when profiling and tracing,
  1. Have a system administrator enable access to the NVIDIA performance counters using the Administration instructions
  2. Run the application being profiled with administrative privileges
    • On Windows targets: Launch the target application by right-clicking on the application and selecting "Run as administrator"
    • On Linux targets: Launch the target application with 'sudo' or as a user with the CAP_SYS_ADMIN capability set

Behavior

For the nvprof command line profiler found in the CUDA Toolkit version 10.2

Mode[1] User[2] Expected Behavior
Restricted Profiling Admin[3] Profiling and tracing produce normal results
Restricted Profiling non-Admin Tracing produces normal results
No Profiling is possible. All attempts to profile will result in output with only the following error message:

The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM"

    Unrestricted Profiling Admin[3] Profiling and tracing produce normal results
    Unrestricted Profiling non-Admin Profiling and tracing produce normal results
    [1] Mode set by system administrator using the Administration instructions
    [2] Privilege level of user running target application being profiled on target device
    [3] Admin user privilege achieved by
    • On Windows targets: Launching the target application by right-clicking on the application and selecting "Run as administrator"
    • On Linux targets: Launching the target application with 'sudo' or as a user with the CAP_SYS_ADMIN capability set

    For the nvprof command line profiler found in the CUDA Toolkit version 10.1

    Mode[1] User[2] Expected Behavior
    Restricted Profiling Admin[3] Profiling and tracing produce normal results
    Restricted Profiling non-Admin No Profiling or Tracing is possible. All attempts to profile will result in output with only the following error message:

    The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM"

      Unrestricted Profiling Admin[3] Profiling and tracing produce normal results
      Unrestricted Profiling non-Admin Profiling and tracing produce normal results
      [1] Mode set by system administrator using the Administration instructions
      [2] Privilege level of user running target application being profiled on target device
      [3] Admin user privilege achieved by
      • On Windows targets: Launching the target application by right-clicking on the application and selecting "Run as administrator"
      • On Linux targets: Launching the target application with 'sudo' or as a user with the CAP_SYS_ADMIN capability set