Red Hat OpenShift is an enterprise-grade Kubernetes platform for managing Kubernetes clusters at scale, developed and supported by Red Hat. It offers a path to transform how organizations manage complex infrastructures on-premises as well as across the hybrid cloud.
AI computing brings far-reaching transformations to modern business, including fraud detection in financial services and recommendations engines for entertainment and e-commerce. A 2018 CTA Market Research study shows that companies using AI technology at the core of their corporate strategy boosted their profit margin by 15% compared to companies not embracing AI.
The responsibility of providing infrastructure for these new, massively compute-intensive workload falls on the shoulders of IT. Many organizations struggle with the complexity, time, and cost associated with deploying an IT-approved infrastructure for AI. Up to 40% of organizations wishing to deploy AI see infrastructure as a major blocker. Deploying a cluster for AI workloads often raises questions in areas of network topology and sizing of compute and storage resources. NVIDIA has therefore created reference architectures for typical applications to alleviate the guesswork.
The NVIDIA DGX POD, for example, consists of multiple DGX-1 systems and storage from named vendors. NVIDIA developed DGX POD from the experience of deploying thousands of nodes of leading-edge accelerated computing deployed in the world’s largest research and enterprise environments. Ensuring AI success at-scale, however, necessitates a software platform, such as KubernetesTM, that ensures manageability of the AI infrastructure.
Red Hat OpenShift 4 is a major release incorporating technology from Red Hat’s acquisition of CoreOS. At its core (no pun intended) are immutable system images based on Red Hat Enterprise Linux CoreOS (RHCOS). It follows a new paradigm where installations are never modified or updated after they are deployed but replaced with updated version of the entire system image. This provides higher reliability and consistency with a more predictable deployment process.
This post takes a first look at OpenShift 4 and GPU Operators for AI workloads on NVIDIA reference architectures. We base this post on a software preview requiring some manual steps, which will be resolved in the final version.
Installing and running OpenShift requires a Red Hat account and additional subscriptions. The official installation instructions are available under Installing a cluster on bare metal.
Test Setup Overview
The minimum configuration for an OpenShift cluster consists of three master nodes and two worker nodes (also called compute nodes). Initial setup of the cluster requires an additional bootstrap node, which can be removed or repurposed during the installation process. The requirement of having three master nodes ensures high-availability (avoiding split-brain situations) and allows for uninterrupted upgrades of the master nodes.
We used virtual machines on a single x86 machine for the bootstrap and master nodes and two DGX-1 systems for the compute nodes (bare-metal). A load-balancer ran in a separate VM to distribute requests to the nodes. Using round-robin DNS might have also worked, but getting the configurations correct turned out to be tricky. The virsh network needs to be set in bridged mode instead of NAT so that the nodes can communicate with each other (see Libvirt Networking for more details).
Red Hat OpenShift 4 does not yet provide a fully automated installation method for bare-metal systems but requires external infrastructure for provisioning and performing the initial installation (the OpenShift documentation refers to it as User Provisioned Infrastructure – UPI). In our case, we used the x86 server for provisioning and for booting nodes through PXE boot. Once installed, nodes perform upgrades automatically.
Creating the Systems Configurations
Red Hat Enterprise Linux CoreOS uses Ignition for the system configuration. Ignition provides a similar functionality to cloud-init and allows to configure systems during the first boot.
The Ignition files are generated by the OpenShift installer from the configuration file install-config.yaml. It describes the cluster through various parameters and also includes an SSH key and credentials for pulling containers from the Red Hat container repository. The OpenShift tools and the Pull Secret can be downloaded from the OpenShift Start Page.
apiVersion: v1
baseDomain: nvidia.com
compute:
- hyperthreading: Enabled
name: worker
platform: {}
replicas: 2
controlPlane:
hyperthreading: Enabled
name: master
platform: {}
replicas: 3
metadata:
creationTimestamp: null
name: dgxpod
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OpenShiftSDN
machineCIDR: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '{"auths": ….}'
sshKey: ssh-rsa ...
The parameters baseDomain and metadata:name form the domain name of the cluster (dgxpod.nvidia.com). The network parameters describe the internal network of the OpenShift cluster and only need to be modified if they conflict with the external network.
The following commands create the Ignition files for the nodes and the authentication file for the cluster. Because these commands delete install-config.yam, we kept a copy of it outside the ignition directory. The generated authentication file (ignition/auth/kubeconfig) should be renamed and copied to $USERHOME/.kube/config
mkdir ignition cp install-config.yaml ignition openshift-install --dir ignition create ignition-configs
DHCP and PXE Boot
Setting up PXE boot is certainly not an easy feat; providing detailed instructions is beyond the scope of this post. Readers should have knowledge of setting up PXE booting and DHCP. The following snippets only cover the DNS configuration for dnsmasq.
The address directive in the dnsmasq configuration file allows for using a wildcard to resolve any *.apps request with the address of the load balancer. The SRV entries allow the cluster to access the etcd service.
# Add hosts file addn-hosts=/etc/hosts.dnsmasq # Forward all *.apps.dgxpod.nvidia.com to the load balancer address=/apps.dgxpod.nvidia.com/10.33.3.54/ # SRV DNS records srv-host=_etcd-server-ssl._tcp.dgxpod.nvidia.com,etcd-0.dgxpod.nvidia.com,2380,0,10 srv-host=_etcd-server-ssl._tcp.dgxpod.nvidia.com,etcd-1.dgxpod.nvidia.com,2380,0,10 srv-host=_etcd-server-ssl._tcp.dgxpod.nvidia.com,etcd-2.dgxpod.nvidia.com,2380,0,10
The corresponding /etc/hosts.dnsmasq file lists the IP address and host names. Note that OpenShift requires that the first entry for each host to be the node name, such as master-0. The api-int and api entries point to the load balancer.
10.33.3.44 worker-0.dgxpod.nvidia.com 10.33.3.46 worker-1.dgxpod.nvidia.com 10.33.3.50 master-0.dgxpod.nvidia.com etcd-0.dgxpod.nvidia.com 10.33.3.51 master-1.dgxpod.nvidia.com etcd-1.dgxpod.nvidia.com 10.33.3.52 master-2.dgxpod.nvidia.com etcd-2.dgxpod.nvidia.com 10.33.3.53 bootstrap.dgxpod.nvidia.com 10.33.3.54 api-int.dgxpod.nvidia.com api.dgxpod.nvidia.com
The following pxelinux.cfg file is an example of a non-EFI PXE boot configuration. It defines the kernel and initial ramdisk, and provides additional command line arguments. Note that the arguments prefixed with coreos are passed to the CoreOS installer.
DEFAULT rhcos PROMPT 0 TIMEOUT 0 LABEL rhcos kernel rhcos/rhcos-410.8.20190425.1-installer-kernel initrd rhcos/rhcos-410.8.20190425.1-installer-initramfs.img append ip=dhcp rd.neednet=1 console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=vda coreos.inst.image_url=http://10.33.3.18/rhcos/rhcos-410.8.20190412.1-metal-bios.raw coreos.inst.ignition_url=http://10.33.3.18/rhcos/ignition/master.ign
Kernel, initramfs, and raw images are available from the OpenShift Mirror. The installation instructions, Installing a cluster on bare metal, provide the latest version and download path. The image files and the ignition configurations from the previous step should be copied to the http directory. Ensure you have the proper http SELinux label set for all these files. Note that the DGX-1 systems only support UEFI for network booting and thus requires a different set of files.
Load Balancer
The Load balancer handles distributing requests across the online nodes. We ran and instance of CentOS in a separate virtual machine and used HAProxy with the following configuration.
listen ingress-http
bind *:80
mode tcp
server worker-0 worker-0.dgxpod.nvidia.com:80 check
server worker-1 worker-1.dgxpod.nvidia.com:80 check
listen ingress-https
bind *:443
mode tcp
server worker-0 worker-0.dgxpod.nvidia.com:443 check
server worker-1 worker-1.dgxpod.nvidia.com:443 check
listen api
bind *:6443
mode tcp
server bootstrap bootstrap.dgxpod.nvidia.com:6443 check
server master-0 master-0.dgxpod.nvidia.com:6443 check
server master-1 master-1.dgxpod.nvidia.com:6443 check
server master-2 master-2.dgxpod.nvidia.com:6443 check
listen machine-config-server
bind *:22623
mode tcp
server bootstrap bootstrap.dgxpod.nvidia.com:22623 check
server master-0 master-0.dgxpod.nvidia.com:22623 check
server master-1 master-1.dgxpod.nvidia.com:22623 check
server master-2 master-2.dgxpod.nvidia.com:22623 check
Creating the Bootstrap and Master Nodes
The virt-install command allows for easy deployment of the bootstrap and master nodes. NODE-NAME should be replaced with the actual name of the node and NODE-MAC with the corresponding network address (MAC) for the node.
virt-install --connect qemu:///system --name --ram 8192 --vcpus 4 --os-type=linux --os-variant=virtio26 --disk path=/var/lib/libvirt/images/.qcow2,device=disk,bus=virtio,format=qcow2,size=20 --pxe --network bridge=virbr0 -m --graphics vnc,listen=0.0.0.0 --noautoconsole
After the initial installation completes, the VM exits and must be restarted manually. The state of the VMs can be monitored using sudo virsh list which prints all active VMs. Once they disappear, they can be restarted with virsh start <NODE-NAME>. In the prerelease version of OpenShift, resolv.conf wasn’t updated and required rebooting the nodes one more time using virsh reset for each node to reboot the nodes again.
The entire installation process of the cluster should take less than an hour, assuming correct settings and configurations. The initial bootstrapping progress can be monitored using the following command.
openshift-install --dir wait-for bootstrap-complete
The bootstrap node can be deleted when the bootstrapping completes. Next, wait for the entire installation process to complete, use:
openshift-install --dir wait-for install-complete
The pre-release version of the installer reported errors at times but eventually completed successfully. Because it also didn’t automatically approve pending certifications (CSR), we added the following crontab entry that ran every 10 minutes.
*/10 * * * * dgxuser oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve
GPU Support
NVIDIA and Red Hat continue to work together to provide a straightforward mechanism for deploying and managing GPU drivers. The Node Feature Discovery Operator (NFD) and GPU Operator build the foundation for this improved mechanism and will be available from the Red Hat Operator Hub. This allows having and optimized software stack deployed at all times. The following instructions describe the manual steps to install these operators.
The NFD detects hardware features and configurations in the OpenShift cluster, such as CPU type and extensions or, in our case, NVIDIA GPUs.
git clone https://github.com/openshift/cluster-nfd-operator cd cluster-nfd-operator/manifests oc create -f . oc create -f cr/nfd_cr.yaml
After the installation completes, the NVIDIA GPU should show up in the feature list for the worker nodes; the final software will provide a human readable name instead of the vendor ID (0x10de for NVIDIA).
oc describe node worker-0|grep 10de
feature.node.kubernetes.io/pci-10de.present=true
The Special Resource Operator (SRO) provides a template for accelerated cards. It activates when a component is detected and installs the correct drivers and other software components.
The development version of the Special Resource Operator already includes support for NVIDIA GPUs and will merge into the NVIDIA GPU Operator when it becomes available. It manages the installation process of all required NVIDIA drivers and software components.
git clone https://github.com/zvonkok/special-resource-operator cd special-resource-operator/manifests oc create -f . cd cr oc create -f sro_cr_sched_none.yaml
The following nvidia-smi.yaml file defines a Kubernetes Pod that can be used for a quick validation. It allocates a single GPU and runs the nvidia-smi command.
apiVersion: v1
kind: Pod
metadata:
name: nvidia-smi
spec:
containers:
- image: nvidia/cuda
name: nvidia-smi
command: [ nvidia-smi ]
resources:
limits:
nvidia.com/gpu: 1
requests:
nvidia.com/gpu: 1
The oc create -f nvidia-smi.yaml script creates and runs the pod. To monitor the progress of the pod creation use oc describe pod nvidia-smi. When completed, the output of the nvidia-smi command can be viewed with oc logs nvidia-smi:
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:86:00.0 Off | 0 | | N/A 36C P0 41W / 300W | 0MiB / 16130MiB | 1% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
Finally, the pod can be deleted with oc delete pod nvidia-smi.
Conclusion
Introducing Operators and an immutable infrastructure built on top of Red Hat Enterprise Linux CoreOS brings exciting improvements to OpenShift 4. It simplifies deploying and managing an optimized software stack for multi-node large scale GPU-accelerated data centers. These new features look pretty solid now and we think customers will be pleased to use them going forward.
