Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Check out Christian Diller's CG-HOI :) We generate realistic 3D human-object interactions, from object geometry and text description. A key ingredient is explicit modeling of contact, during training and as guidance during inference.

Angela Dai

9,758 subscribers

20,514 views • 2 years ago •via X (Twitter)

Science & Technology Education Arts

Anya Rossi• Live Now

Private livecam show

5 Comments

Michael Black2 years ago

@chrdiller Wow. Nice work!

Behzad Tabari2 years ago

@chrdiller Revolutionizing the animation industry and Hollywood 👍☄️

D. hans2 years ago

@chrdiller hello, would you be willing to share the code for this paper?

apondo yanick2 years ago

@chrdiller You have said it all proff

Ling-Hao (Evan) CHEN2 years ago

@chrdiller impressive work!

Related Videos

New study introduces a system to generate realistic, physically plausible human-object interactions from natural language instructions. It uses LLMs for planning, a motion generator for synchronized body and finger movements, and RL for physical accuracy.

New study introduces a system to generate realistic, physically plausible human-object interactions from natural language instructions. It uses LLMs for planning, a motion generator for synchronized body and finger movements, and RL for physical accuracy.

The Humanoid Hub

27,606 views • 1 year ago

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

Kostas Daniilidis

47,642 views • 4 months ago

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

Jiaman Li

15,886 views • 1 year ago

Experience the thrill of Adventure Mode with HappyOyster 1.0✨ Build immersive scenes from text or image prompts and explore them in real time! Now with expanded character actions and object interactions for even richer storytelling. Watch the video to check out the enhanced interactivity! HappyOyster

Experience the thrill of Adventure Mode with HappyOyster 1.0✨ Build immersive scenes from text or image prompts and explore them in real time! Now with expanded character actions and object interactions for even richer storytelling. Watch the video to check out the enhanced interactivity! HappyOyster

Alibaba Group

10,298,896 views • 1 month ago

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

AK

66,486 views • 2 years ago

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 views • 3 years ago

Mind blown by this new CVPR 2026 paper: tttLRM🤯 It introduces a Test-Time Training layer for long-context autoregressive 3D reconstruction with linear complexity. Multiple image observations are compressed into fast weights that keep updating during inference, forming an implicit scene representation that can be decoded into explicit formats such as 3D Gaussian Splats. Progressive 3D reconstruction from streaming inputs is getting wild🥳It’s interesting to see how AI inference and real-world capture are starting to converge.

Mind blown by this new CVPR 2026 paper: tttLRM🤯 It introduces a Test-Time Training layer for long-context autoregressive 3D reconstruction with linear complexity. Multiple image observations are compressed into fast weights that keep updating during inference, forming an implicit scene representation that can be decoded into explicit formats such as 3D Gaussian Splats. Progressive 3D reconstruction from streaming inputs is getting wild🥳It’s interesting to see how AI inference and real-world capture are starting to converge.

KIRI Engine - 3D Scanner App

16,949 views • 4 months ago

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Felix Heide

27,779 views • 11 months ago

📢 SHeaP: Self-Supervised Head Predictor Learned via 2D Gaussians 📢 Given a single input image, we predict accurate 3D head geometry, pose, and expression. Previous works (e.g. DECA, EMOCA) use differentiable mesh rasterization to learn a self-supervised head geometry predictor via a photometric reconstruction loss. We borrow these ideas, but our key insight is to replace the mesh rendering with 2D Gaussian Splatting. This leads to much higher accuracy of the underlying predicted geometry and thus more gradient signal during training. 🌍 🎥 Great work by Liam Schoneveld Davide Davoli Jiapeng Tang

📢 SHeaP: Self-Supervised Head Predictor Learned via 2D Gaussians 📢 Given a single input image, we predict accurate 3D head geometry, pose, and expression. Previous works (e.g. DECA, EMOCA) use differentiable mesh rasterization to learn a self-supervised head geometry predictor via a photometric reconstruction loss. We borrow these ideas, but our key insight is to replace the mesh rendering with 2D Gaussian Splatting. This leads to much higher accuracy of the underlying predicted geometry and thus more gradient signal during training. 🌍 🎥 Great work by Liam Schoneveld Davide Davoli Jiapeng Tang

Matthias Niessner

28,559 views • 1 year ago

We asked Cyan Banister about her favorite book The Diamond Age. "In this book, every object is 3D printed by nanotech. A little girl is given a book with AI guidance. Since everything is perfect, human artistry becomes valuable." "This shapes how I think about AI and abundance. I love Neal Stephenson. He predicted Meta, VR and Second Life - I base a lot of my investments on his work."

We asked Cyan Banister about her favorite book The Diamond Age. "In this book, every object is 3D printed by nanotech. A little girl is given a book with AI guidance. Since everything is perfect, human artistry becomes valuable." "This shapes how I think about AI and abundance. I love Neal Stephenson. He predicted Meta, VR and Second Life - I base a lot of my investments on his work."

TBPN

35,145 views • 1 year ago

Images, video, audio, actions — generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode. Language is the lone exception: still generated token by token, as a long discrete stream. Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder. Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline — thanks to compression and one-step generation. Co-led w/ Zhengyang Geng 🧵

Images, video, audio, actions — generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode. Language is the lone exception: still generated token by token, as a long discrete stream. Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder. Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline — thanks to compression and one-step generation. Co-led w/ Zhengyang Geng 🧵

Mihir Prabhudesai

77,745 views • 20 days ago

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

Xueyan Zou

38,299 views • 1 month ago

No wonder they say teleoperation is dead Peking University’s DAGroup released HumanNet, a massive 1 million hour dataset of human-centric videos that turns everyday internet footage into gold for embodied AI. It’s got first-person and third-person views, super detailed actions, object interactions, tool use, and long sequences of real behavior. Everything’s cleaned up and annotated with 3D poses, SLAM tracks, and robot-ready labels. The crazy part is,Training on just 1,000 hours of their first-person videos performs as well as (or even slightly better than) 100 hours of actual robot teleop data on downstream tasks. Basically, human videos can now stand in for expensive robot data. Scaling laws for robotics just got a whole lot more realistic. Human priors + smart curation = the future of world models.

No wonder they say teleoperation is dead Peking University’s DAGroup released HumanNet, a massive 1 million hour dataset of human-centric videos that turns everyday internet footage into gold for embodied AI. It’s got first-person and third-person views, super detailed actions, object interactions, tool use, and long sequences of real behavior. Everything’s cleaned up and annotated with 3D poses, SLAM tracks, and robot-ready labels. The crazy part is,Training on just 1,000 hours of their first-person videos performs as well as (or even slightly better than) 100 hours of actual robot teleop data on downstream tasks. Basically, human videos can now stand in for expensive robot data. Scaling laws for robotics just got a whole lot more realistic. Human priors + smart curation = the future of world models.

CyberRobo

27,048 views • 2 months ago

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,595 views • 2 years ago

Infinite Photorealistic Worlds using Procedural Generation paper page: introduce Infinigen, a procedural generator of photorealistic 3D scenes of the natural world. Infinigen is entirely procedural: every asset, from shape to texture, is generated from scratch via randomized mathematical rules, using no external source and allowing infinite variation and composition. Infinigen offers broad coverage of objects and scenes in the natural world including plants, animals, terrains, and natural phenomena such as fire, cloud, rain, and snow. Infinigen can be used to generate unlimited, diverse training data for a wide range of computer vision tasks including object detection, semantic segmentation, optical flow, and 3D reconstruction. We expect Infinigen to be a useful resource for computer vision research and beyond.

Infinite Photorealistic Worlds using Procedural Generation paper page: introduce Infinigen, a procedural generator of photorealistic 3D scenes of the natural world. Infinigen is entirely procedural: every asset, from shape to texture, is generated from scratch via randomized mathematical rules, using no external source and allowing infinite variation and composition. Infinigen offers broad coverage of objects and scenes in the natural world including plants, animals, terrains, and natural phenomena such as fire, cloud, rain, and snow. Infinigen can be used to generate unlimited, diverse training data for a wide range of computer vision tasks including object detection, semantic segmentation, optical flow, and 3D reconstruction. We expect Infinigen to be a useful resource for computer vision research and beyond.

AK

275,403 views • 3 years ago

On July 11, 1991, during a total solar eclipse, multiple civilians in Mexico City famously filmed a shiny, disk-shaped object hovering in the daytime sky. A TV executive captured the first video from a rooftop, which was broadcast on national television and prompted 40,000+ phone calls and over a dozen other videos from viewers across the city. Witnesses described the object as a bright, metallic “larger object” that “was giving off light” and clearly not a star or plane. The incident is often cited as one of the most documented mass UFO sightings in history, with similar glowing discs recorded simultaneously in cities miles apart.

On July 11, 1991, during a total solar eclipse, multiple civilians in Mexico City famously filmed a shiny, disk-shaped object hovering in the daytime sky. A TV executive captured the first video from a rooftop, which was broadcast on national television and prompted 40,000+ phone calls and over a dozen other videos from viewers across the city. Witnesses described the object as a bright, metallic “larger object” that “was giving off light” and clearly not a star or plane. The incident is often cited as one of the most documented mass UFO sightings in history, with similar glowing discs recorded simultaneously in cities miles apart.

Disclosure Party

12,207 views • 1 year ago

🚀Introducing Hunyuan3D-PolyGen, our newly upgraded and industry-first art-grade 3D generative model. It brings effortless intelligent retopology, making AI-generated models ready for professional art pipelines. ✅ Superior Mesh Topology: Our self-developed mesh autoregressive model ensures higher-quality mesh topology that meets stringent art standards. ✅ Complex Object Modeling: Leveraging our high-compression BPT representation, we can generate models with 10K+ faces, enabling more complex geometry, higher topology precision, and better detail. ✅ Flexible Output: Supports both tri and quad meshes, meeting diverse pipeline requirements. Hunyuan3D-PolyGen enables direct application of AI-generated 3D assets in game development and significantly boosts artist modeling efficiency. It's a robust foundation for the future of 3D content creation. 👉Try it now:

🚀Introducing Hunyuan3D-PolyGen, our newly upgraded and industry-first art-grade 3D generative model. It brings effortless intelligent retopology, making AI-generated models ready for professional art pipelines. ✅ Superior Mesh Topology: Our self-developed mesh autoregressive model ensures higher-quality mesh topology that meets stringent art standards. ✅ Complex Object Modeling: Leveraging our high-compression BPT representation, we can generate models with 10K+ faces, enabling more complex geometry, higher topology precision, and better detail. ✅ Flexible Output: Supports both tri and quad meshes, meeting diverse pipeline requirements. Hunyuan3D-PolyGen enables direct application of AI-generated 3D assets in game development and significantly boosts artist modeling efficiency. It's a robust foundation for the future of 3D content creation. 👉Try it now:

Tencent Hy

161,346 views • 1 year ago