backflip process thread or how we helped my avatar to make a backflip 👥 2x in VR <3 to gerhard funk for the set up in Unity3D

The idea was inspired by YouTube and some ridiculous goals that we love to set for ourselves.

About AI. In 2015, I saw Flexible Muscle-Based Locomotion for Bipedal Creatures (2013) by @tgeijten, @Mvandepanne, Frank van der Stappen. AI is learning to walk.

In 2018, @xbpeng4, @pabbeel, @svlevine & @Mvandepanne released “DeepMimic”. Here, agents copy human motion from YT videos and reapply the skills in different settings. DeepMimic was our base. <3 to @m_schneider for setting it up for the avatar.

Once the system was running, it was time to train our avatar to stand up and balance (around a day for each task). We simulated the process connecting the movements from different stages. The info text indicates the current sim task and skill level.

Training/sim were set up in arg files and then initiated in prompt. The sets were photographed and 3D scanned at different times of day/night. With & without furniture. Everything needed to be remodeled in low poly for the simulation sim. <3 to Leonhard Gläser for modeling/tex

Voice was cloned in Descript. wav2lip was used to generate lip/face expressions. These were connected to our physical face rig. <3 to @any_user and C. Dorstewitz for input. Finally, @kamptweets and I generated and recorded most of the foley. It was edited/synched in the studio.

There are many more stories around the film but I stop here. Watch backflip on YT if you haven’t:

@gyfunk How did you approach the sound design?

@gyfunk Controlling your avatar in vr added even one more layer to hilarity of it all xD

@gyfunk ⚡️breakdancing2.3tera_LOL⚡️