GTC Silicon Valley-2019: Determinism in Deep Learning
GTC Silicon Valley-2019 ID:S9911:Determinism in Deep Learning
Until recently, deep learning on GPUs has been characterized by non-exact results. Training and inference have been approximate due to non-deterministic effects beyond pseudo-random choices such as mini-batch processing used to implement stochastic gradient descent. This non-determinism has created insurmountable challenges in traceability, debugging, experimentation, and regression testing. We'll discuss our work to eliminate non-determinism from deep learning when using TensorFlow. We'll explain how we were motivated by a need to make our processes reproducible with a primary focus on auditing and traceability in safety-critical applications. Beneficial side effects are simplified debugging, more effective experimentation, and the ability to accurately regression-test changes. Our talk summarizes discoveries and solutions in this area.