📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

Some details and designs for our work: (1/5) We tackle the exciting challenge of generating scene-aware interaction motions for virtual characters based on text instructions and target locations within a 3D environment. There are some beautiful results showing how the generated motion interacts with the 3D scenes instructed by input text.

（2/5) Our motion generation method handles both locomotion and interaction motions. We leverage an auto-regressive conditional diffusion model that takes language guidance, the goal location for the current segment, and the scene voxel as input.

(3/5) The character's scene awareness comes from a local occupancy grid. Each voxel in the grid indicates whether the corresponding location is occupied by a scene object. Such representation enhances the understanding of 3D space and interaction.

(4/5) Given the same trajectory and scene, our model generates characters that actively avoid penetrating the scene and exhibit natural cues of scene awareness.

(5/5) Our model is supercharged by LINGO, a comprehensive motion-captured dataset. We employ a synthetic vision approach, where scene objects are projected into the virtual view displayed in a VR headset worn by the motion actor.

nice work 👏

🔥🔥🔥🔥