To enable deep learning, Yahoo added GPU nodes into their Hadoop clusters with each node having 4 Nvidia Tesla K80 cards, each card with two GK210 GPUs. These nodes have 10x processing power than the traditional commodity CPU nodes they generally use in their Hadoop clusters.
Yahoo has progressively invested in building and scaling Apache Hadoop clusters with a current footprint of more than 40,000 servers and 600 petabytes of storage spread across 19 clusters.
Hadoop clusters are the preferred platform for large-scale machine learning at Yahoo. Deep learning is used in many of Yahoo’s products, such as Flickr’s Magic View feature which automatically tags all user photos, enabling Flickr end users to organize and find photos easily. Read “Picture This: NVIDIA GPUs Sort Through Tens of Millions of Flickr Photos” for more information on the feature.
To enable deep learning on these enhanced clusters, the Yahoo Big Data and Machine Learning team developed a comprehensive distributed solution based upon open source software libraries, Apache Spark and Caffe. Caffe-on-Spark enables multiple GPUs, and multiple machines to be used for deep learning.
Related resources
- GTC session: Accelerating Data Analytics on GPUs with the RAPIDS Accelerator for Apache Spark
- NGC Containers: Caffe2
- NGC Containers: NVCaffe
- SDK: RAPIDS Accelerator for Spark
- SDK: NVIDIA PyTorch
- SDK: DGL Container
About the Authors
Their example talks about training neural networks, is there any information on if this same technique can be applied for detection on a distributed platform like AWS to do a detection using a trained caffemodel file?