Kai Sun

Kai Sun is a DevTech Engineer at NVIDIA. He joined NVIDIA in 2020. He has expertise in network communication technologies and NCCL implementation. He is one of the developers of the Hybrid-EP communication library. Kai holds a master's degree from the College of Engineering, Peking University.
Avatar photo

Posts by Kai Sun

Networking / Communications Feb 02, 2026

Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert Parallel

In LLM training, Expert Parallel (EP) communication for hyperscale mixture-of-experts (MoE) models is challenging. EP communication is essentially all-to-all,... 11 MIN READ