NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters
Overview
This page provides an explanation and solutions for the following error message from various NVIDIA development tools:ERR_NVGPUCTRPERM The user running <tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters on the target device.
If you are receiving this error then
- You are using a tool that accesses the GPU Performance Counters. These metrics are a developer-specific feature that provide low level access to GPU hardware.
- Your Administrator or a recent NVIDIA driver installation has disabled access to GPU Performance due to Security Notice: NVIDIA Response to “Rendered Insecure: GPU Side Channel Attacks are Practical” - November 2018.
Solutions for this issue
Your tool's profiling is affected by the administrative restriction on access to the NVIDIA GPU performance counters when using a windows 419.17 or Linux 418.43 or later driver. To gain access to these counters when profiling,- Run the tool or application being profiled with administrative privileges. See tool specific information, but in general,
- On Windows targets: Launch the tool "As Administrator"
- On Linux targets: Launch the tool with 'sudo' or as a user with the CAP_SYS_ADMIN capability set
- Have a system administrator enable access to the NVIDIA performance counters using the Administration instructions
Administration
To control access to the NVIDIA performance counters, a system administrator use NVIDIA Control Panel or Command Line Control, as follows:NVIDIA Control Panel Control (Windows[1] and Mac only)
The NVIDIA Control Panel is installed with your display driver. GPU Performance Counter control requires display driver 419.17 or later. You must launch the control panel as a system administrator to manage the GPU Performance Counters.(right-click on your desktop desktop for quick access to the NVIDIA Control Panel)
Windows Step 1: Open the NVIDIA Control Panel, select 'Desktop', and ensure 'Enable Developer Settings' is checked.
Windows Step 2: Under 'Developer' > 'Manage GPU Performance Counters', select 'Allow access to the GPU performance counter to all users' to enable unrestricted profiling[1]
MacOS: Under 'Developer' > 'GPU Performance Counters', select 'Allow access for all users' to enable unrestricted profiling
Command Line Control - Linux Only
GPU Performance Counter control requires Linux display driver 418.43 or later.- Unload the old modules. Before you can insert the module with the key set/unset, you first need to stop the window manager, and unload all nvidia-based modules. So as root, or with sudo:
- # The following assumes a systemd-based distribution. For non-systemd-based distributions, a different procedure is required.
systemctl isolate multi-user # Stop the window manager. - modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia # Unload dependent modules
- Specify the access you desire: Insert the modules with the registry key set/unset.:
To require administrative privileges (CAP_SYS_ADMIN capability set) to profile using NVIDIA GPU performance counters
- modprobe nvidia NVreg_RestrictProfilingToAdminUsers=1 [1][4][7]
Or, to allow any user to profile using NVIDIA GPU performance counters
- modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0 [2][4][7]
[1] Alternatively, A file containing 'options nvidia "NVreg_RestrictProfilingToAdminUsers=1"' may be saved to /etc/modprobe.d [3][5]
[2] Alternatively, A file containing 'options nvidia "NVreg_RestrictProfilingToAdminUsers=0"' may be saved to /etc/modprobe.d [3][5]
[3] A reboot may be required in order for this to take effect, but these settings will be persisted
[4] These settings will not be persisted after a reboot.
[5] On some systems (or when using a deb to install), it may be necessary to rebuild the initrd after writing a configuration file to /etc/modprobe.d[6]
[6] When rebuilding the initrd, running “update-initramfs -u” is also required.
[7] On Ubuntu systems, when installing via the distro-native packages, the kernel module gets
renamed from nvidia to nvidia-xxx, and then nvidia is aliased to nvidia-xxx
(where xxx is the major number of the driver. So a 418.67 driver would use nvidia-418)
- Restart the window manager (if desired).
- # The following assumes a systemd-based distribution. For non-systemd-based distributions, a different procedure is required.
systemctl isolate graphical # Restart the window manager.
- # The following assumes a systemd-based distribution. For non-systemd-based distributions, a different procedure is required.
Also see the "Restricting access to GPU performance counters" section of the README.txt in the Linux driver