Short URL: https://developer.nvidia.com/ERR_NVGPUCTRPERM

Overview

This page provides an explanation and solutions for the following error message from various NVIDIA development tools:
   ERR_NVGPUCTRPERM The user running <tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters on the target device.

If you are receiving this error then

1. You are using a tool that accesses the GPU Performance Counters. These metrics are a developer-specific feature that provide low level access to GPU hardware.
2. Your Administrator or a recent NVIDIA driver installation has disabled access to GPU Performance due to Security Notice: NVIDIA Response to “Rendered Insecure: GPU Side Channel Attacks are Practical” - November 2018.

Solutions for this issue

Your tool's profiling is affected by the administrative restriction on access to the NVIDIA GPU performance counters when using a windows 419.17 or Linux 418.43 or later driver. To gain access to these counters when profiling,

1. Run the tool or application being profiled with administrative privileges. See tool specific information, but in general,
   • On Windows targets: Launch the tool by right clicking on the tool and selecting “Run as administrator” or running the full command from an Administrator command prompt
   • On Linux Desktop targets: Launch the tool with 'sudo' or as a user with the CAP_SYS_ADMIN capability set
   • On Linux and QNX Mobile targets: Launch the tool with 'sudo' or as the root user
2. Have a system administrator enable access to the NVIDIA performance counters using the Administration instructions

Administration

To control access to the NVIDIA performance counters, a system administrator use NVIDIA Control Panel or Command Line Control, as follows:

NVIDIA Control Panel Control (Windows^[1] and Mac only)

The NVIDIA Control Panel is installed with your display driver. GPU Performance Counter control requires display driver 419.17 or later. You must launch the control panel as a system administrator to manage the GPU Performance Counters.
(right-click on your desktop desktop for quick access to the NVIDIA Control Panel)[1] Note: The 425.25 windows driver control panel for Tesla family GPUs may not respect the performance counter access setting. If you encounter this issue, please see the Tesla on Windows Control Panel Issue page.

Command Line Control - Linux Desktop Only

GPU Performance Counter control requires Linux display driver 418.43 or later.

1. Unload the old modules.
Before you can insert the module with the key set/unset, you first need to stop the window manager, and unload all nvidia-based modules. So as root, or with sudo:
• systemctl isolate multi-user^[1] # Stop the window manager.
• modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia^[2] # Unload dependent modules


[1] Assumes a systemd-based distribution. For non-systemd-based distributions, a different procedure is required.
[2] Assumes no processes are using these modules, for sucessful unloading.



2. Specify the access you desire: Insert the modules with the registry key set/unset.:
   

   To require administrative privileges (CAP_SYS_ADMIN capability set) to profile using NVIDIA GPU performance counters

   • modprobe nvidia NVreg_RestrictProfilingToAdminUsers=1 ^[1][4][7][8]

   Or, to allow any user to profile using NVIDIA GPU performance counters

   • modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0 ^[2][4][7][9]

   
   [1] Alternatively, A file with the .conf extension containing 'options nvidia "NVreg_RestrictProfilingToAdminUsers=1"' may be saved to /etc/modprobe.d ^[3][5]
   [2] Alternatively, A file with the .conf extension containing 'options nvidia "NVreg_RestrictProfilingToAdminUsers=0"' may be saved to /etc/modprobe.d ^[3][5]
   [3] A reboot may be required in order for this to take effect, but these settings will be persisted
   [4] These settings will not be persisted after a reboot.
   [5] On some systems (or when using a deb to install), it may be necessary to rebuild the initrd after writing a configuration file to /etc/modprobe.d[6]
   [6] When rebuilding the initrd, running “update-initramfs -u” is also required.
   [7] On Ubuntu systems, when installing via the distro-native packages, the kernel module gets
          renamed from nvidia to nvidia-xxx, and then nvidia is aliased to nvidia-xxx
          (where xxx is the major number of the driver. So a 418.67 driver would use nvidia-418)
   For Container/Docker users:
         If 'CTRL+F "container" or "docker"' comes up empty,

   • [8] The 1st solution of Launching the tool with 'sudo' or as a user with the CAP_SYS_ADMIN capability set should work.
   • [9] The 2nd solution of using the nvidia mod flag should work the same in or outside containers and don't require any flags for Docker.
   
   

3. Restart the window manager (if desired).
   • systemctl isolate graphical^[1] # Restart the window manager.

   
   [1] Assumes a systemd-based distribution. For non-systemd-based distributions, a different procedure is required.
   
   

4. If the steps above fail to work, a system reboot may also be required.

Also see the "Restricting access to GPU performance counters" section of the README.txt in the Linux driver

Linux and QNX Mobile Only

On Tegra and QNX platforms, you must profile as root/sudo user in order to allow access to GPU performance counters.

Note that on DRIVE OS environments, support-gpu-tools must be enabled:

Steps to re-enable GPU Debugger and Profiler support (if needed) :
• Set the support-gpu-tools device tree property in the GPU device node to "1".
• Recompile the Device Tree following the instructions in the appropriate DRIVE OS SDK Developement Guide:
• DRIVE OS Linux SDK Development Guide
• DRIVE OS QNX SDK Development Guide
• Flash updated DTB.
• GPU Debugger and Profiler support should be enabled now.

Tool Specific Solutions

The following tools may encounter this issue and may have tool-specific information on the associated pages:

• Nsight Graphics
• Nsight Compute
• Nsight Visual Studio Edition - Graphics Range Profiler
• Nsight Visual Studio Edition - Legacy CUDA Profiler
• Nsight Visual Studio Edition - Nsight Compute CUDA Profiler
• nvprof
• Visual Profiler
• CUPTI

REF:DTSHARED-118