Containers are now ubiquitous, and for good reason; the portability and productivity enhancements they provide have made them a standard component in HPC and many other computing fields. The NVIDIA Nsight family of developer tools for analyzing performance of CUDA applications are supported in container environments. For more information about the environmental landscape and Nsight Systems, see Using NVIDIA Nsight Systems in Containers and the Cloud.
This post contains similar information for using Nsight Compute in container environments. Nsight Compute is specifically designed to provide detailed performance analysis of CUDA kernels running on the GPU.
Setting up a system for profiling
Profiling with Nsight Compute requires access to the performance counters on the GPU. Whether or not you are running in a container, you may run into a permissions issue if they are not available. For more information about this common issue, see Permission issue with Performance Counters.
These counters need to be accessible on the machine where the application is running and being profiled. This should be configured on the machine outside of the container.
Installing Nsight Compute inside a container
If your container does not have Nsight Compute installed already, you can install it just like you would outside of a container. You must have the NVIDIA repositories set up, which should be the default in any NGC container. For more information, see the following sections for sample commands to add the repositories manually to your existing containers.
When you have the repos set up, use a command like these to install Nsight Compute:
# apt-get update -y # apt-get install -y nsight-compute-2020.1.0
If you’re unsure of the versions available, use a command like the following:
# apt-cache search nsight-compute
Adding Nsight Compute to an existing Docker container
Adding Nsight Compute to your existing Docker container image is straightforward. The following code examples provide example lines to add to the existing Dockerfile. These examples assume Nsight Compute version 2020.1.0, which was the most recent at the time of publication.
The available versions can be found using a command like apt-cache search nsight-compute, provided that the NVIDIA repositories are set up. Use the appropriate code example depending on whether your container image is RHEL-based or Debian-based.
Debian-based Docker
RUN apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
gnupg \
wget && \
rm -rf /var/lib/apt/lists/*
RUN echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - && \
apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
nsight-compute-2020.1.0 && \
rm -rf /var/lib/apt/lists/*
RHEL-based Docker
RUN rpm --import https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub && \
yum install -y yum-utils && \
yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64 && \
yum list available | grep nsight && \
yum install -y \
nsight-compute-2020.1.0 && \
rm -rf /var/cache/yum/*
Adding Nsight Compute to an existing Singularity container
Adding Nsight Compute to your existing Singularity container image is almost identical to Docker. The following code examples provide example lines to add to the existing Singularity file. These code examples assume Nsight Compute version 2020.1.0, which was the most recent at the time of publication.
The available versions can be found using a command like apt-cache search nsight-compute. Use the appropriate code example depending on whether your container image is RHEL-based or Debian-based.
Debian-based Singularity
%post
apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
gnupg \
wget && \
rm -rf /var/lib/apt/lists/*
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - && \
apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
nsight-compute-2020.1.0 && \
rm -rf /var/lib/apt/lists/*
RHEL-based Singularity
%post
rpm --import https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub && \
yum install -y yum-utils && \
yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64 && \
yum list available | grep nsight && \
yum install -y \
nsight-compute-2020.1.0 && \
rm -rf /var/cache/yum/*
Mapping an Nsight Compute host installation into a container
So far, I’ve described how to add Nsight Compute to a Docker or Singularity container image. Sometimes, it may be easier to use the existing Nsight Compute installation from the host inside the container.
An Nsight Compute installation on the host can be mounted into a container at runtime but this may not work properly if the required Nsight Compute dependencies available on the host are not available inside the container.
For example, if the host Nsight Compute is installed on the host at /opt/nvidia/nsight-compute/2020.1.0/, you can add the following command-line options to temporarily make Nsight Compute available inside a container.
$ sudo docker run --gpus all -it -v /opt/nvidia/nsight-compute/2020.1.0/:/opt/nvidia/nsight-compute/2020.1.0:ro … $ singularity run --nv -B /opt/nvidia/nsight-compute/2020.1.0/:/opt/nvidia/nsight-compute/2020.1.0/ ...
Using the GUI from a container (Singularity only)
The previous sections described how to use the Nsight Compute command-line interface with containers. In those cases, the generated report file could be opened and visualized on another machine with the GUI. However, it is also possible to use the Nsight Compute GUI inside the container.
The container image must have several X11 and related packages installed to satisfy the Nsight Compute GUI requirements. The following is a reference Singularity definition file. There may be additional or duplicate requirements listed here depending on your container. It is provided to be instructive but can be modified as needed.
Bootstrap: docker
From: nvidia/cudagl:10.1-base-ubuntu18.04
%post
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
dbus \
fontconfig \
gnupg \
libasound2 \
libfreetype6 \
libglib2.0-0 \
libnss3 \
libsqlite3-0 \
libxkbcommon-x11-0 \
libx11-xcb1 \
libxcb-glx0 \
libxcb-xkb1 \
libxcomposite1 \
libxcursor1 \
libxdamage1 \
libxi6 \
libxml2 \
libxrandr2 \
libxrender1 \
libxtst6 \
openssh-client \
wget \
xcb \
xkb-data
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - && \
apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
nsight-compute-2020.1.0 && \
rm -rf /var/lib/apt/lists/*
%environment
export LC_ALL=C
export PATH=/opt/nvidia/nsight-compute/2020.1.0/:$PATH
%runscript
exec ncu-ui "$@"
To create the Nsight Compute GUI container image, first create a file named Singularity.def with the earlier content and then build the image.
$ sudo singularity build ncu-gui.sif Singularity.def
Running the container automatically starts the Nsight Compute GUI:
$ singularity run --nv ncu-gui.sif
Call to action
If you’re writing CUDA kernels and you’re concerned about performance, Nsight Compute should be a tool in your toolbox. With support for containers, it’s easy to try and easy to integrate into your existing development environment.
To learn more, see the following resources:
- CUDA Developer Tools: Overview and Exciting New Features
- Optimizing CUDA Kernels in HPC Simulation and Visualization Codes Using NVIDIA Nsight Compute
- Modern CUDA Programming Hazards and the Linux Nsight Toolbox to Fix Them
- OptiX profiling with Nsight Compute
- CUDA Kernel Profiling Using NVIDIA Nsight Compute
Have a question? Post it to the NVIDIA forums using NVIDIA Nsight Compute or drop a message.
