So you think you can dance? Earlier this month, the Dallas Mavericks of the NBA showed off a new deep learning in-game entertainment application that synthesized the dance moves of one of their star players on the team’s jumbotron.
The Texas-based startup Xpire AI who was co-founded by the team’s owner Mark Cuban, based their new technology on work from NVIDIA Research called pix2pixHD, a conditional GAN framework for image-to-image translation.
In order to create the training data for their Deep Convolutional Generative Adversarial Network (DCGAN), the engineers recorded one of the team’s players, Jalen Brunson, performing basic body movements. With this short training video, they were then able to extract the pose from every frame to generate a skeleton image of each pose.
“Now that we have pairs of (skeleton image, realistic image), we can train our network to map the input images to the output images,” the startup explained in a technical blog about the process. “After training for many hours on a modern (GeForce RTX 2080 Ti) GPU, we are left with a model that can effectively map input pose images to realistic output images!
With the trained model, the team was able to generate realistic videos of Jalen doing never-before-seen dance moves.
