NVIDIA JetPack provides a full development environment for hardware-accelerated AI-at-the-edge on Jetson platforms. Previously, a standalone version of NVIDIA JetPack supports a single release of CUDA, and you did not have the ability to upgrade CUDA on a given NVIDIA JetPack version. NVIDIA JetPack is released on a rolling cadence with a single version of CUDA, typically being supported throughout each major release cycle (for example, NVIDIA JetPack 4.x or NVIDIA JetPack 5.x).
Starting with CUDA Toolkit 11.8, Jetson users on NVIDIA JetPack 5.0 and later can upgrade to the latest CUDA release without updating the NVIDIA JetPack version or Jetson Linux BSP (Board Support Package). You can stay on par with the CUDA Desktop releases.
CUDA on Jetson compared with CUDA on desktop
To understand why the CUDA support model has been different between the desktop with discrete-GPU (dGPU) and Jetson with integrated-GPU (iGPU), it helps to understand the following:
- How CUDA is packaged on Jetson
- How CUDA is packaged on desktop
- The differences between them
Figure 1 shows the Jetson software architecture, with a core of the Jetson Linux BSP and layers of the various software components that make up the NVIDIA JetPack SDK. For more information, see Jetson Software Architecture.
Figure 2 shows where CUDA resides in the overall NVIDIA JetPack SDK packaging structure and how it interacts with all other components of the Jetson Linux BSP. As you can see in Figure 2, the CUDA driver is part of the Jetson Linux BSP, along with other components. All these components update as per the release cadence and frequency of the Jetson Linux BSP, which has been different from the quarterly CUDA release cadence. The CUDA toolkit is separate from the BSP and does not package the CUDA driver.
When you install the NVIDIA JetPack SDK, the Jetson Linux BSP (containing the CUDA driver) and the CUDA toolkit get installed by default.
Due to this packaging structure, CUDA developers on desktop have the flexibility to stay up to date with the latest CUDA releases aligning with the CUDA quarterly release cadence. Moreover, features such as forward compatibility and minor version compatibility help you pick up combinations of driver and toolkit, and tailor it per your application needs.
CUDA upgradable package on Jetson
Starting from CUDA 11.8, CUDA has introduced an upgrade path that provides Jetson developers with an option to update the CUDA driver and the CUDA toolkit to the latest versions.
Figure 4 shows blue boxes that depict components that are present by default in the NVIDIA JetPack 5.0 SDK. The dotted line separates Jetson Linux BSP from the other components that are part of the NVIDIA JetPack SDK. The green boxes indicate the CUDA components that you can upgrade to through this feature.
These upgrades are made possible by the introduction of the CUDA driver upgrade (also referred to as the CUDA compatibility package), as shown in Figure 5.
This upgrade package mainly contains the CUDA driver (libcuda.so.*) and its dependencies that enable you to access the latest and greatest CUDA functionalities that come with every quarterly CUDA release.
Without this package, you were previously limited to the functionality provided by the default CUDA driver that was packaged in the Jetson Linux BSP. You had no mechanism to upgrade to the latest CUDA driver and toolkit.
With this package, Jetson users who have invested in long and thorough validation cycles for the existing Jetson Linux BSP can upgrade to the latest CUDA versions. This upgrade is done over the existing Jetson Linux BSP, keeping it unchanged.
How to upgrade CUDA on Jetson
With CUDA 11.8, the CUDA Downloads page now displays a new architecture, aarch64-Jetson, as shown in Figure 6, with the associated aarch64-Jetson CUDA installer and provides step-by-step instructions on how to download and use the local installer, or CUDA network repositories, to install the latest CUDA release.
The new aarch64-Jetson CUDA installer packages both the CUDA Toolkit and the upgrade package together. The step-by-step installation instructions provided ensure that the CUDA upgrade package gets downloaded and installed along with the corresponding CUDA toolkit for Jetson devices.
The installed upgrade package is available in the versioned toolkit file directory. For example, you can find 11.8 in the following directory:
/usr/local/cuda-11.8/
The upgrade package consists of the following files:
libcuda.so.*: The CUDA driver.libnvidia-nvvm.so.*: Just-in-time link-time optimization (CUDA 11.8 and later only).libnvidia-ptxjitcompiler.so.*: The JIT (just-in-time) compiler for PTX files.
These files together implement the CUDA driver interface. This package only provides the files and does not configure the system.
If you are working on an x86 host and cross-compiling to the aarch64-Jetson target, the U20.04 CUDA host installer can be found on the CUDA Downloads page. The cross-compile bits can be found in the following directory:
aarch64-jetson/cross/Ubuntu/20.04/deb installer
Example
The following code example shows how the CUDA Upgrade package can be installed and used to run the applications.
$ sudo apt-get -y install cuda
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
cuda-11-8 cuda-cccl-11-8 cuda-command-line-tools-11-8 cuda-compat-11-8
...<snip>…
The following NEW packages will be installed:
cuda cuda-11-8 cuda-cccl-11-8 cuda-command-line-tools-11-8 cuda-compat-11-8
...<snip>…
0 upgraded, 48 newly installed, 0 to remove and 38 not upgraded.
Need to get 15.7 MB/1,294 MB of archives.
After this operation, 4,375 MB of additional disk space will be used.
Get:1 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/arm64 cuda-compat-11-8 11.8.31058490-1 [15.8 MB]
Fetched 15.7 MB in 12s (1,338 kB/s)
Selecting previously unselected package cuda-compat-11-8.
(Reading database ...
...<snip>…
(Reading database ... 100%
(Reading database ... 148682 files and directories currently installed.)
Preparing to unpack .../00-cuda-compat-11-8_11.8.31058490-1_arm64.deb ...
Unpacking cuda-compat-11-8 (11.8.31058490-1) ...
...<snip>…
Unpacking cuda-11-8 (11.8.0-1) ...
Selecting previously unselected package cuda.
Preparing to unpack .../47-cuda_11.8.0-1_arm64.deb ...
Unpacking cuda (11.8.0-1) ...
Setting up cuda-toolkit-config-common (11.8.56-1) ...
Setting up cuda-compat-11-8 (11.8.31058490-1) ...
$ ls -l /usr/local/cuda-11.8/compat
total 55300
lrwxrwxrwx 1 root root 12 Jan 6 19:14 libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root 14 Jan 6 19:14 libcuda.so.1 -> libcuda.so.1.1
-rw-r--r-- 1 root root 21702832 Jan 6 19:14 libcuda.so.1.1
lrwxrwxrwx 1 root root 19 Jan 6 19:14 libnvidia-nvvm.so -> libnvidia-nvvm.so.4
lrwxrwxrwx 1 root root 23 Jan 6 19:14 libnvidia-nvvm.so.4 -> libnvidia-nvvm.so.4.0.0
-rw-r--r-- 1 root root 24255256 Jan 6 19:14 libnvidia-nvvm.so.4.0.0
-rw-r--r-- 1 root root 10665608 Jan 6 19:14 libnvidia-ptxjitcompiler.so
lrwxrwxrwx 1 root root 27 Jan 6 19:14 libnvidia-ptxjitcompiler.so.1 -> libnvidia-ptxjitcompiler.so
The user can set LD_LIBRARY_PATH to include the libraries installed by upgrade package before running the CUDA 11.8 application:
$ LD_LIBRARY_PATH=/usr/local/cuda-11.8/compat:$LD_LIBRARY_PATH ~/Samples/1_Utilities/deviceQuery
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Orin"
CUDA Driver Version / Runtime Version 11.8 / 11.8
CUDA Capability Major/Minor version number: 8.7
...<snip>...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.8, CUDA Runtime Version = 11.8, NumDevs = 1
Result = PASS
The default drivers (originally installed with NVIDIA JetPack and part of the Jetson Linux BSP) are retained by the installer. The application can use either the default version of CUDA (originally installed with NVIDIA JetPack) or the one installed by the upgrade package. Use the LD_LIBRARY_PATH variable to choose the required version.
Only a single CUDA upgrade package can be installed at any point in time on a given system. While installing a new CUDA upgrade package, the previous version of the installed upgrade package is removed and replaced with the new one. Installation of the upgrade package fails if it is not compatible with the NVIDIA JetPack version.
For example, applications that were previously compiled with CUDA 11.4 continue to work with the CUDA 11.8 upgrade package due to backward compatibility in the CUDA driver.
Table 1 shows the CUDA user-mode driver (UMD) and CUDA Toolkit version compatibility for the NVIDIA JetPack 5.0 release.
| CUDA UMD | CUDA Toolkit | |
| 11.4 (default; part of NVIDIA JetPack) |
11.8 | |
| 11.4 (default; part of NVIDIA JetPack) |
C | C (minor version compatibility) |
| 11.8 (with the upgrade package) |
C | C |
C = Compatible; X = Not compatible
Points to note
- This feature is available from CUDA 11.8 and NVIDIA JetPack 5.0 onwards and will be supported on the latest Jetson Linux releases.
- CUDA upgrade package only updates the CUDA driver interfaces while leaving the rest of the NVIDIA JetPack SDK components unchanged. If a new feature in the latest CUDA driver needs an updated NVIDIA JetPack SDK component or interface, it might return an error when called. For more information about feature compatibility, see the CUDA release notes.
- Users are requested to check for compatibility of new CUDA versions with the NVIDIA JetPack SDK version being used, as not all NVIDIA JetPack SDKs support all versions of CUDA. For more information about compatible versions, see CUDA for Tegra App Note.
On Jetson, the compute stack of CUDA, cuDNN, TensorRT, and so on, was tightly tied to a particular version of Jetson Linux (L4T). To upgrade to a newer version of the compute stack, you also had to deal with upgrading to Jetson Linux.
We are working towards a future where Jetson developers can migrate to newer versions of the compute libraries without upgrading Jetson Linux. This CUDA feature that enables upgrading CUDA is a step in that direction.
Upgrade to the latest CUDA release on your Jetson today!
- On the CUDA 11.8 Downloads page, download the CUDA installer for
aarch64-Jetsonand follow the installation instructions to upgrade your Jetson device to CUDA 11.8. - For more information about the CUDA upgradable package on Jetson, see CUDA for Tegra App Note.
- For information about all the new features that CUDA 11.8 brings in, see CUDA 11.8 Omnibus.
- If you have any questions or require support, post your questions on the Jetson forum.
Do register for the NVIDIA JetPack 5 deep-dive webinar. The CUDA and Jetson team walk you through details on this new feature and you get an opportunity to ask questions live!
