GTC Silicon Valley-2019: Exascale Deep Learning for Climate Analytics
Note: This video may require joining the NVIDIA Developer Program or login
GTC Silicon Valley-2019 ID:S9412:Exascale Deep Learning for Climate Analytics
Thorsten Kurth(Lawrence Berkeley National Laboratory),Josh Romero(NVIDIA)
We'll discuss how we scaled the training of a single deep learning model to 27,360 V100 GPUs (4,560 nodes) on the OLCF Summit HPC System using the high-productivity TensorFlow framework. We discuss how the neural network was tweaked to achieve good performance on the NVIDIA Volta GPUs with Tensor Cores and what further optimizations were necessary to provide excellent scalability, including data input pipeline and communication optimizations, as well as gradient boosting for SGD-type solvers. Scalable deep learning becomes more and more important as datasets and deep learning models grow and become more complicated. This talk is targeted at deep learning practitioners who are interested in learning what optimizations are necessary for training their models efficiently at massive scale.