Updating the CUDA Linux GPG Repository Key

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by the apt, dnf/yum, and zypper package managers beginning April 27, 2022.

If you don’t update your repository signing keys, expect package management errors when attempting to access or install packages from CUDA repositories.

To ensure continued access to the latest NVIDIA software, complete the following steps.

Remove the outdated signing key

Debian, Ubuntu, WSL

$ sudo apt-key del 7fa2af80

Fedora, RHEL, openSUSE, SLES

$ sudo rpm --erase gpg-pubkey-7fa2af80*

Install the new key

For Debian-based distributions, including Ubuntu, you must also install the new package or manually install the new signing key.

Install the new cuda-keyring package

To avoid the need for manual key installation steps, NVIDIA is providing a new helper package to automate the installation of new signing keys for NVIDIA repositories. 

Replace $distro/$arch in the following commands with values appropriate for your OS; for example:

• debian10/x86_64
• debian11/x86_64
• ubuntu1604/x86_64
• ubuntu1804/cross-linux-sbsa
• ubuntu1804/ppc64el
• ubuntu1804/sbsa
• ubuntu1804/x86_64
• ubuntu2004/cross-linux-sbsa
• ubuntu2004/sbsa
• ubuntu2004/x86_64
• ubuntu2204/sbsa
• ubuntu2204/x86_64
• wsl-ubuntu/x86_64

Debian, Ubuntu, WSL

$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb

Alternate method: Manually install the new signing key

If you can’t install the cuda-keyring package, you can install the new signing key manually (not the recommended method).

Debian, Ubuntu, WSL

$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub

RPM distros

On a fresh installation, Fedora, RHEL, openSUSE, or SLES as dnf/yum/zypper prompt you to accept new keys when the repository signing key changes. Accept the change when prompted.

Replace $distro/$arch in the following commands with values appropriate for your OS; for example:

• fedora32/x86_64
• fedora33/x86_64
• fedora34/x86_64
• fedora35/x86_64
• opensuse15/x86_64
• rhel7/ppc64le
• rhel7/x86_64
• rhel8/cross-linux-sbsa
• rhel8/ppc64le
• rhel8/sbsa
• rhel8/x86_64
• sles15/cross-linux-sbsa
• sles15/sbsa
• sles15/x86_64

For upgrades on RPM-based distros including Fedora, RHEL, and SUSE, you must also run the following command.

Fedora and RHEL 8

$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

RHEL 7

$ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/$arch/cuda-rhel7.repo

openSUSE and SLES

$ sudo zypper removerepo cuda-$distro-$arch
$ sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

Working with containers

CUDA applications built using older NGC base containers may contain outdated repository keys. If you build Docker containers using these images as a base and update the package manager or install additional NVIDIA packages as part of your Dockerfile, these commands may fail as they would on a non-container system. To work around this, integrate the earlier commands into the Dockerfile you use to build the container.

Existing containers in which the package manager is not used to install updates are not affected by this key rotation.

Working with the NVIDIA GPU Operator

If you are a current user of the GPU Operator on Ubuntu distributions, you may be affected by the rotation of the CUDA GPG keys, where some of the containers managed by the GPU Operator may fail to start with the following error:

Stopping NVIDIA persistence daemon... Unloading NVIDIA driver kernel modules... Unmounting NVIDIA driver rootfs... Checking NVIDIA driver packages... Updating the package cache... W: GPG error:  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is no longer signed.

NVIDIA is publishing new images for the driver containers by overwriting existing image tags. You can work around this error by updating the existing clusterPolicy to pull new images:

$ kubectl edit clusterpolicy
...
set  driver.imagePullPolicy=Always

This step results in the GPU Operator pulling the updated images.

New installations of the GPU Operator should be unaffected by this change and do not require any clusterPolicy updates. If you use the GPU Operator on RHEL or OpenShift, you are also not affected by this change.

Common issues and solutions on Debian-based distros

Here are some common errors that we’ve helped people with. If you see an error not listed here, please comment below.

Duplicate .list entries

{{E: Conflicting values set for option Signed-By regarding source
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: 
/usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.}}

Solution: If you previously used add-apt-repository to enable the CUDA repository, then remove the duplicate entry.

sudo sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list

Also check for and remove cuda*.list files under the /etc/apt/sources.d/ directory.

New GPG key is not enrolled

{{Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY A4B46996 3BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease' is no longer signed.}}

Solution: See “Duplicate .list entries” notice to install cuda-keyring package OR one of the manual enrollment methods for the 3bf863cc public key.

Machine Learning repository

{{W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used.
GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64
Release: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY F60F4B3D 7FA2AF80}}

Solution: Remove the NVIDIA machine learning repository entry, as it is no longer updated. Newer versions of cuDNN, NCCL, and TensorRT are available in the CUDA repository.

File has unexpected size

{{Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: XXX.XXX.XXX.XXX 443]
Hashes of expected file:

* Filesize:481481 [weak]
* SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
* SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
* MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
* Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
* E: Some index files failed to download. They have been ignored, or old ones used instead.}}

Solution: Report CDN issue to NVIDIA.