Distributed deep learning depends on fast, reliable GPU-to-GPU communication using the NVIDIA Collective Communication Library (NCCL). When training slows down, it becomes challenging to determine why and what to do next. A problem can span computation, communication, a specific rank, or underlying hardware.  NVIDIA NCCL Inspector accelerates triaging by providing a lightweight and continuous report … Continue reading Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus