Gone are the days when it was expected that a programmer would “own” all the systems that they needed. Modern computational work frequently happens in shared systems, in the cloud, or otherwise on hardware not owned by the user or even their employer.
This is good for developers. It can save time and money by allowing for testing and development on multiple architectures or OSs without prohibitive expense. But it can present real challenges as well, including handling set up, negotiating security constraints, and navigating access requirements for resources.
Trying to use the right tools in those environments is sometimes complicated, so here is a handy guide to help you use the NVIDIA Nsight family of tools wherever you are doing your development.
VMs and Containers and Clouds, Oh My!
First, here’s a breakdown of some options.
As shown in Figure 1, both virtual machines (VMs) and containers are used to create a self-contained environment that allows you to run your application in isolation on a shared system. The main difference is that when you run a container, the underlying platform provides an operating system for basic services with a limited degree of isolation using Linux namespaces. With VMs, the VM must provide the operating system itself, and VM support in the hardware is used to enforce isolation.
For this reason, VMs have higher overhead than containers, so containers are usually used when hardware is shared by many users. “Bare metal” instances in the cloud are a thin hypervisor layer, basically a VM.
Containers are a widely adopted method of taming the complexity of deploying HPC and AI software. The entire software environment, from the deep learning framework or HPC application down to the math and communication libraries necessary for performance, is packaged into a single, easily deployed bundle.
Because workloads inside a container always use the same environment, the performance is reproducible and portable across a wide variety of systems. It is possible to easily deploy the same containerized software environment on your local workstation, a DGX server in your datacenter, your site’s shared HPC cluster, and the cloud.
There are several container runtimes available for use, including Docker, Singularity, and Podman. The best container technology for you depends on the administrative requirements of the cluster that you are working on, particularly security and job scheduling requirements. In the examples below, we focus on Docker and Singularity as two of the most commonly used container technologies.
Likewise, there are dozens of cloud service providers (CSPs) to choose from. If your organization does not already have a designated CSP, consider aspects of service, pricing, and GPU availability when making your decision.
Container runtimes often limit the access of software running inside the container. This can be a challenge for profiling tools like NVIDIA Nsight Systems or NVIDIA Nsight Compute. For this post, we are focusing on considerations when using Nsight Systems. Look for a future post covering Nsight Compute.
Setting Up and Using Nsight Systems Inside Containers (Docker/Singularity)
There are different ways to work with Nsight Systems in a container environment. You can map it into an existing container or build a new container image that includes it. After it’s installed, you can run the tool inside the container or remote from the tool into the container for analysis.
This post assumes that you have experience with Docker or Singularity containers. If you have general questions, see the Docker or Singularity documentation.
Enable Sampling in Containers
Nsight Systems samples CPU activity and gets backtraces using the Linux kernel’s perf subsystem. To collect thread scheduling data and instruction pointer (IP) samples, the perf paranoid level on the target system must be ≤2. Run the following command to check the level:
cat /proc/sys/kernel/perf_event_paranoid
If the output is >2, then run the following command to temporarily adjust the paranoid level (after each reboot):
sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid'
To make the change permanent, run the following command:
sudo sh -c 'echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/local.conf'
When performing a Nsight Systems collection with sampling in a Docker container, additional steps are required to enable the perf_event_open system call to enable Linux perf. These steps are not required if you are using Singularity.
There are two ways to enable the perf_event_open syscall. You can enable it by using the --cap-add=SYS_ADMIN switch. If your system meets the requirements, you can also enable it by setting the seccomp security profile.
Secure computing mode (seccomp) is a feature of the Linux kernel that can be used to restrict an application’s access. This feature is available only if the kernel is enabled with seccomp support. To check for seccomp support, use the following command:
$ grep CONFIG_SECCOMP= /boot/config-$(uname -r)
The result should contain the following line:
CONFIG_SECCOMP=y
Seccomp profiles require seccomp 2.2.1, which is not available on some older distributions (for example, Ubuntu 14.04, Debian Wheezy, or Debian Jessie). To use seccomp on older distributions, you must download the latest static Linux binaries rather than packages.
Download the default seccomp profile file, default.json, relevant to your Docker version. If perf_event_open is already listed in the file as guarded by CAP_SYS_ADMIN, then remove the perf_event_open line. Add the following lines under “syscalls” and save the resulting file as default_with_perf.json:
{
"name": "perf_event_open",
"action": "SCMP_ACT_ALLOW",
"args": []
},
To apply the new seccomp profile, use the following switch when starting the Docker container.
--security-opt seccomp=default_with_perf.json
To make sure that your container is set up properly for Nsight Systems, run the status command inside the container to check your environment:
$ nsys status -e
This lets you know whether features are limited in the current environment.
Installing Nsight Systems from NGC and the CUDA Toolkit
The simplest way to profile with Nsight Systems in a container is to download one of the containers from the NVIDIA GPU Cloud (NGC) catalog. Many of these containers, such as the NGC 19.11 TensorFlow container, already include Nsight Systems and just work out of the box.
If a NGC container does not have Nsight Systems pre-installed, install it using one of the following commands, depending on the CUDA toolkit version used in the container. If you are not sure which CUDA toolkit version is included in your container, the output of nvidia-smi shows the CUDA toolkit version.
CUDA 10.1:
$ apt-get update -y $ apt-get install -y cuda-nsight-systems-10-1 nsight-systems-2019.3.7
CUDA 10.2:
$ apt-get update -y $ apt-get install -y cuda-nsight-systems-10-2 nsight-systems-2019.5.2
However, the web release of Nsight Systems frequently has more features and fixes than the version available with the CUDA toolkit. To use the most recent version, build or modify your own container image.
Adding Nsight Systems to your Existing Docker Container
Adding the latest Nsight Systems to your existing Docker container image is a simple process. Add the following Dockerfile code example to your existing Dockerfile. Use the appropriate code example depending on whether your container image is RHEL-based or Debian-based. If you are not sure what Linux distribution your container image is based on, try looking at /etc/os-release in the container image. Most NGC images are based on Ubuntu.
NVIDIA Nsight Systems 2020.2.1
RUN rpm --import https://developer.download.nvidia.com/devtools/repos/rhel8/x86_64/nvidia.pub && \
yum install -y yum-utils && \
yum-config-manager --add-repo https://developer.download.nvidia.com/devtools/repos/rhel8/x86_64/ && \
yum install -y \
nsight-systems-2020.2.1 && \
rm -rf /var/cache/yum/*
Dockerfile code example to add Nsight Systems to a RHEL-based container image
NVIDIA Nsight Systems 2020.2.1
RUN apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
gnupg \
wget && \
rm -rf /var/lib/apt/lists/*
RUN wget -qO - https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/nvidia.pub | apt-key add - && \
echo "deb https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/ /" >> /etc/apt/sources.list.d/nsight.list && \
apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
nsight-systems-2020.2.1 && \
rm -rf /var/lib/apt/lists/*
Dockerfile code example to add Nsight Systems to a Debian-based container image
At publication time, the latest Nsight Systems version is 2020.2.1.
If you downloaded the image from NGC or Docker Hub, you may not have the original Dockerfile used to generate your container image. Create a Dockerfile and use your container image as the base image. For example, if you were using the 0.9 RAPIDS image from NGC, use the following:
ARG IMAGE=nvcr.io/nvidia/rapidsai/rapidsai:0.9-cuda9.2-base-ubuntu18.04 FROM $IMAGE # NVIDIA Nsight Systems 2020.2.1 RUN apt-get update -y && \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ apt-transport-https \ ca-certificates \ gnupg \ wget && \ rm -rf /var/lib/apt/lists/* RUN wget -qO - https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/nvidia.pub | apt-key add - && \ echo "deb https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/ /" >> /etc/apt/sources.list.d/nsight.list && \ apt-get update -y && \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ nsight-systems-2020.2.1 && \ rm -rf /var/lib/apt/lists/*
Complete Dockerfile example to add Nsight Systems to the NGC RAPIDS container image
The code example to install Nsight Systems is identical to the preceding example.
HPC Container Maker also has an nsight_systems building block to generate the Dockerfile instructions for installing and configuring Nsight Systems.
After the Docker container has been started, use the Nsight Systems CLI (nsys) to launch a collection within the Docker container. For more information, see the previous section about running Nsight Systems inside a container. The result file can be copied from the container to the host using docker cp and then opened or imported into the NVIDIA Nsight Systems GUI on the host, like any other CLI result.
Adding Nsight Systems to your Existing Singularity Container
The steps to add Nsight Systems to your existing Singularity container image are essentially the same as for Docker, although the syntax differs.
Add the following Singularity definition example to your existing Singularity definition file. Use the appropriate example depending on whether your container image is RHEL-based or Debian-based. If you are not sure what Linux distribution your container image is based on, try looking at /etc/os-release in the container image. Most NGC images are based on Ubuntu.
%post
# NVIDIA Nsight Systems 2020.2.1
rpm --import https://developer.download.nvidia.com/devtools/repos/rhel8/x86_64/nvidia.pub
yum install -y yum-utils
yum-config-manager --add-repo https://developer.download.nvidia.com/devtools/repos/rhel8/x86_64/
yum install -y \
nsight-systems-2020.2.1
rm -rf /var/cache/yum/*
Singularity definition code example to add Nsight Systems to a RHEL-based container image
%post # NVIDIA Nsight Systems 2020.2.1 apt-get update -y DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ apt-transport-https \ ca-certificates \ gnupg \ wget rm -rf /var/lib/apt/lists/* wget -qO - https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/nvidia.pub | apt-key add - echo "deb https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/ /" >> /etc/apt/sources.list.d/nsight.list apt-get update -y DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ nsight-systems-2020.2.1 rm -rf /var/lib/apt/lists/*
Singularity definition code example to add Nsight Systems to a Debian-based container image
At the time of post publication, the latest Nsight Systems version is 2020.2.1. To use a different version package, modify the code example and change the place where the NSight Systems package name appears.
If you downloaded the image from NGC or the Singularity container library, you do not have the original Singularity definition file used to generate your container image. In that case, create a Singularity definition file and use your container image as the base image. For example, if you were using a local NAMD image downloaded from NGC named namd-2.13.sif, use the following code example:
Bootstrap: localimage From: namd-2.13.sif %post # NVIDIA Nsight Systems 2020.2.1 apt-get update -y DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ apt-transport-https \ ca-certificates \ gnupg \ wget rm -rf /var/lib/apt/lists/* wget -qO - https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/nvidia.pub | apt-key add - echo "deb https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/ /" >> /etc/apt/sources.list.d/nsight.list apt-get update -y DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ nsight-systems-2020.2.1 rm -rf /var/lib/apt/lists/*
The code example to install Nsight Systems is identical to the preceding example.
HPC Container Maker also has an nsight_systems building block to generate the Singularity definition file instructions for installing and configuring Nsight Systems.
After the Singularity container has been started, use the Nsight Systems CLI (nsys) to launch a collection within the Singularity container. The result file can be opened or imported into the Nsight Systems GUI on the host like any other CLI result.
Mapping an Nsight Systems Host Installation into a Container
The previous two sections covered how to add Nsight Systems to a Docker or Singularity container image. To only profile a single run, creating a new container image may be overkill.
A Nsight Systems installation on the host can also be mounted into a container at runtime. However, note that a few bare-bones container images may not satisfy the Nsight Systems CLI dependencies, so this method may have issues for some containers where adding Nsight Systems to the container image does not have issues.
For example, if the host Nsight Systems installation is located at /opt/nvidia/nsight-systems/2020.2.1 on the host, you can add the following command line options to temporarily make Nsight Systems available inside a container.
$ sudo docker run --runtime=nvidia -v /opt/nvidia/nsight-systems/2020.2.1:/opt/nvidia/nsight-systems/2020.2.1:ro ...
$ singularity run --nv -B /opt/nvidia/nsight-systems/2020.2.1:/opt/nvidia/nsight-systems/2020.2.1 ...
You can now use the Nsight Systems CLI (nsys) inside the container, although you must specify the full path to the nsys executable.
Using the Nsight Systems GUI from a Container
The presumed workflow in the previous sections is to use the Nsight Systems CLI to collect a profile of a workload running in a container and to visualize it in the GUI on the host or on another system. However, the Nsight Systems GUI can also be run from a container. This is the most straightforward approach when using Singularity, as the host X11 environment is automatically accessible from inside the container.
The container image must have several X11 and related packages installed to satisfy the Nsight Systems GUI requirements. The following is a reference Singularity definition file for CUDA 10.1, Ubuntu 16.04, and Nsight Systems 2020.2.1. The Nsight Systems code example is functionally identical to the example in the section on adding Nsight Systems to a Singularity container image.
BootStrap: docker
From: nvidia/cudagl:10.1-base-ubuntu16.04
%post apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
dbus \
fontconfig \
gnupg \
libasound2 \
libfreetype6 \
libglib2.0-0 \
libnss3 \
libsqlite3-0 \
libx11-xcb1 \
libxcb-glx0 \
libxcb-xkb1 \
libxcomposite1 \
libxcursor1 \
libxdamage1 \
libxi6 \
libxml2 \
libxrandr2 \
libxrender1 \
libxtst6 \
openssh-client \
wget \
xcb \
xkb-data
wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub | apt-key add -
echo "deb https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/ /" >> /etc/apt/sources.list.d/nsight.list
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
nsight-systems-2020.2.1
rm -rf /var/lib/apt/lists/*
%environment
export LC_ALL=C
export __GLX_VENDOR_LIBRARY_NAME=nvidia
%runscript
exec nsight-sys "$@"
To create the Nsight Systems GUI container image, first create a file named Singularity.def with the above content, and then build the image.
$ sudo singularity build nsys-gui.sif Singularity.def
Running the container automatically starts the Nsight Systems GUI.
$ singularity run --nv nsys-gui.sif
A very cool feature of the Singularity Nsight Systems GUI container is that it can be used “remotely” to profile a workload running the host.
Configure a new remote target, using “localhost” for the hostname, your normal username for the username, and select Password-based authentication. Select the remote target for profiling and you are prompted for your password. Nsight Systems then injects all the target components onto the host from the container.
Because Singularity automatically mounts the user home directory inside the container, the Nsight Systems report is retained and is available even when the container exits.
Of course, you can also use the GUI container to visualize previously collected profiles. Choose File, Load or File, Import just as you would on a bare metal installation of Nsight Systems.
Using Nsight Systems in the Cloud
The process (and price) for setting yourself up with a GPU- enabled cloud instance varies by CSP, and is beyond the scope of this post. When choosing your instance, remember that Nsight Systems supports GPUs from Pascal forward.
NVIDIA partners with many cloud service providers to ensure quality service. To find out more about that program and find a full list of current partners and offerings, see NVIDIA GPU Cloud Computing.
Consult the service manual for your CSP to determine how to select a GPU enhanced instance.
After you have your instance, installing Nsight Systems is basically the same as installing it anywhere else. For more information, see the Getting Started Guide. To make sure that your instance is set up properly for Nsight Systems, run the status command to check your environment:
$ nsys status -e
This lets you know whether features are limited in the current environment.
A Call to Action
This is the fourth in a series of posts that describe the new Nsight family of tools, show the functionality, and explain how to move your development to the new tools. Check the NVIDIA Developer Blog for future posts covering these topics in greater depth.
Previous entries:
- Migrating to NVIDIA Nsight Tools from NVVP and Nvprof
- Transitioning to Nsight Systems from NVIDIA Visual Profiler / nvprof
- Using Nsight Compute to Inspect your Kernels
Get Nsight Systems and Nsight Compute from the NVIDIA CUDA ToolKit public download. You can also obtain the most recent, updated Nsight tools with enhancements and fixes beyond the version shipping in the NVIDIA CUDA Toolkit at the Nsight Systems page, Nsight Compute page, or Nsight Graphics page.
To see the tools in action, check out the following links featuring videos from recent GPU Technology Conferences (GTC):
NVIDIA Nsight Systems
- Blue Waters Webinar Series – Nov. 2018 – Introduction to NVIDIA Nsight Systems
- Optimizing HPC Simulation and Visualization Codes Using the NVIDIA Nsight Systems
- Using Nsight Tools to Optimize the NAMD Molecular Dynamic Simulation Program
- Optimizing Facebook AI Workloads for NVIDIA GPUs
- Boost DNN Training Performance using NVIDIA Tools
NVIDIA Nsight Compute
- OptiX profiling with Nsight Compute
- CUDA Kernel Profiling Using NVIDIA Nsight Compute
- Using Nsight Compute or Nvprof to Show Mixed Precision Use in Deep Learning Models
NVIDIA Nsight Graphics
We’ve also covered NVIDIA Nsight tools in older posts, including Nsight Systems Exposes New GPU Optimization Opportunities and What is Limiting Your Rendering Performance.
Have a question? Post it to the NVIDIA forums using either NVIDIA Nsight Systems or NVIDIA Nsight Compute. Drop a message at nsight-systems-feedback@nvidia.com or nsight-compute-feedback@nvidia.com. Or just choose Feedback in the application to let us know what you are seeing and what you think.
