NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters

Short URL: https://developer.nvidia.com/ERR_NVGPUCTRPERM

Overview

You may encounter the following error message when using NVIDIA tools:
   ERR_NVGPUCTRPERM The user running <tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters on the target device.

If you are receiving this error, then

  1. You are using a tool that accesses the GPU Performance Counters. These counters are a developer-specific feature that provides low-level access to GPU hardware, and
  2. Your system administrator or a recent NVIDIA driver installation has disabled access to GPU Performance Counters for regular users due to Security Notice: NVIDIA Response to “Rendered Insecure: GPU Side Channel Attacks are Practical” - November 2018. Your tool is affected by this restriction when using driver versions 419.17+ (Windows) or 418.43+ (Linux).
To avoid this error, run the tool or application with elevated privileges or enable access for all users. For problems following these instructions, see the troubleshooting guide.

Run with elevated privileges

Run the tool or application being profiled with administrative privileges on the target device:

  • Windows: Launch the tool by right clicking on the tool and selecting “Run as administrator” or running the full command from an Administrator command prompt.
  • Mac OS: Launch the tool with sudo or as an administrative user.
  • Linux Desktop: Launch the tool with sudo or as a user with the CAP_SYS_ADMIN capability set. When profiling within a container, access must be enabled on the host, or the container must be started with the appropriate permissions by passing --cap-add=SYS_ADMIN as an admin user.
  • DRIVE, Tegra and QNX: Launch the tool with sudo or as the root user.
  • See also tool specific information for further details.

Enable access for all users

Windows

The NVIDIA Control Panel is installed with your display driver. You must launch the control panel as a system administrator to manage the GPU Performance Counters. The relevant option in the control panel requires display driver 419.17 or later.

Right-click on your desktop for quick access to the NVIDIA Control Panel or launch it from the Windows Control Panel.

Windows Step 1: Open the NVIDIA Control Panel, select 'Desktop', and ensure 'Enable Developer Settings' is checked.

Windows Step 2: Under 'Developer' > 'Manage GPU Performance Counters', select 'Allow access to the GPU performance counter to all users' to enable unrestricted profiling[1]

[1] Note: The 425.25 windows driver control panel for Tesla family GPUs may not respect the performance counter access setting. If you encounter this issue, please see the Tesla on Windows Control Panel Issue page.

MacOS

The NVIDIA Control Panel is installed with your display driver. You must launch the control panel as a system administrator to manage the GPU Performance Counters. The relevant option in the control panel requires display driver 419.17 or later.

MacOS: Under 'Developer' > 'GPU Performance Counters', select 'Allow access for all users' to enable unrestricted profiling

Linux Desktop

GPU Performance Counter control requires Linux display driver 418.43 or later. Also see the "Restricting access to GPU Performance Counters" section of the README.txt in the Linux driver.
Enable access permanently
  • To allow access for any user, create a file the .conf extension containing options nvidia "NVreg_RestrictProfilingToAdminUsers=0" in /etc/modprobe.d.
  • To restrict access to admin users (CAP_SYS_ADMIN capability set), create a file with the .conf extension containing options nvidia "NVreg_RestrictProfilingToAdminUsers=1" in /etc/modprobe.d.
      Notes:
  • A reboot may be required for the change to take effect.
  • On some systems (or when using a package manager to install), it may be necessary to rebuild the initrd after writing a configuration file to /etc/modprobe.d.
    • For RedHat-based distributions, rebuild the initrd with dracut -–regenerate-all -f
    • For Debian-based distributions, rebuild the initrd with update-initramfs -u -k all
Enable access temporarily
  1. Before you can insert the kernel module with the required key set/unset, you first need to stop the window manager and unload all NVIDIA kernel modules. As root, or with sudo:
    • Stop the window manager with systemctl isolate multi-user (or your system-specific solution).
    • Unload modules with modprobe -rf nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia
  2. To allow access for any user, run modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0
  3. To restrict access to admin users (CAP_SYS_ADMIN capability set), run modprove nvidia NVreg_RestrictProfilingToAdminUsers=1
  4. If desired, restart the window manager with systemctl isolate graphical (or your system-specific solution).
      Notes:
  • Instructions are for systemd-based distributions. For non-systemd-based distributions, a different procedure is required.
  • For successful unloading, no processes may be using these modules.
  • On Ubuntu systems, when installing via the distro-native packages, the kernel module is renamed from nvidia to nvidia-xxx, and then nvidia is aliased to nvidia-xxx, where xxx is the major number of the driver. So, a 418.67 driver would use nvidia-418.
  • In case of problems with the above instructions or for non-systemd-based distributions, see the troubleshooting guide at the end of this page.

DRIVE, Tegra and QNX

You must enable GPU profiler support and profile as sudo or the root user for access to GPU Performance Counters. To enable GPU profiler (and debugger) and support:
  • Set the support-gpu-tools device tree property in the GPU device node to "1".
  • Recompile the Device Tree following the instructions in the appropriate DRIVE OS SDK Development Guide:
    • DRIVE OS Linux SDK Development Guide
    • DRIVE OS QNX SDK Development Guide
  • Flash updated DTB.

Troubleshooting

Linux Desktop

If you encounter errors when unloading kernel modules that indicate they are still in use, it means that processes still have handles to the relevant devices in /dev. To identify which process is causing the error, use sudo lsof /dev/nvidia* which will list all processes holding handles to GPU device nodes. To be able to unload the kernel modules, no processes should be listed here.

It is possible to use the procfs file system of your Linux machine to check whether access to GPU Performance Counters currently requires administrative permissions. The currently loaded parameters for the nvidia kernel module can generally be viewed in the file /proc/driver/nvidia/params. The boolean flag RmProfilingAdminOnly describes if profiling is restricted to administrative users. When set to 1, only administrative users are allowed to access GPU Performance Counters. When set to 0, all users have access.

In order to check whether the modprobe.d file was correctly included into your initrd file, you can run one of the following commands:

  • Debian-based distributions: sudo lsinitramfs /boot/initrd.img | grep etc/modprobe.d
  • RedHat-based distributions: sudo lsinitrd | grep etc/modprobe.d

You should see the .conf file you created being listed. Both these commands will search through the default initrd file. To specify which initrd to search, see the manual pages for these commands.

Tool Specific Solutions

The following tools may encounter this issue and may have tool-specific information on the associated pages: REF:DTCNSC-5090