Tong Liu

Tong Liu is a DevTech engineer at NVIDIA, specializing in optimizing Mixture-of-Experts (MoE) large language model training and CUDA kernel development. He has contributed to key features in the optimization of Megatron-Core and Transformer-Engine frameworks. He holds a master's degree from the Institute of Computing Technology, Chinese Academy of Sciences.
Avatar photo

Posts by Tong Liu

Networking / Communications Feb 02, 2026

Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert Parallel

In LLM training, Expert Parallel (EP) communication for hyperscale mixture-of-experts (MoE) models is challenging. EP communication is essentially all-to-all,... 11 MIN READ