Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing ClickDiffusion! We developed a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface.

Alec Helbling

10,618 subscribers

36,293 views • 2 years ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

6 Comments

Alec Helbling2 years ago

Check out the paper here:

Alec Helbling2 years ago

And the code here:

Ryan2 years ago

This is awesome! Could you do this across images? Take object A from image One and place it into image 2?

Naina Chaturvedi2 years ago

++ thanks for sharing. Will be covering it here -

marmik2 years ago

this is amazing!

Yufei Huang@ICML2 years ago

This is so cooool! Thanks for open-sourcing!

Related Videos

We just got backed by a16z a16z speedrun 🧊. We believe the next interface is natural language. We call it the Language User Interface (LUI) An interface that talks to your APIs, understands your product, and lets users get work done with natural language (thread below)

We just got backed by a16z a16z speedrun 🧊. We believe the next interface is natural language. We call it the Language User Interface (LUI) An interface that talks to your APIs, understands your product, and lets users get work done with natural language (thread below)

jai

135,731 views • 10 months ago

Accurate and controllable scene generation has been difficult with natural language alone. You instead need a language for scenes. Introducing the Scene Language — a visual representation for high-quality 3D/4D generation by integrating programs, words, and embeddings — 🧵(1/6)

Accurate and controllable scene generation has been difficult with natural language alone. You instead need a language for scenes. Introducing the Scene Language — a visual representation for high-quality 3D/4D generation by integrating programs, words, and embeddings — 🧵(1/6)

Yunzhi Zhang

63,813 views • 1 year ago

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from Stanford University designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from Stanford University designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)

Cheng Chi

438,672 views • 2 years ago

branching interface for writing a tighter creative feedback loop guided by a language model to weave through the possibility space of text

branching interface for writing a tighter creative feedback loop guided by a language model to weave through the possibility space of text

Azlen

51,705 views • 1 year ago

Talking to your AI glasses is silly, so we built Reality Proxy, a direct manipulation interface that lets you instantly select real-world objects and share context with AI. A step toward JARVIS.

Talking to your AI glasses is silly, so we built Reality Proxy, a direct manipulation interface that lets you instantly select real-world objects and share context with AI. A step toward JARVIS.

Xiaoan Liu

150,821 views • 10 months ago

OpenArm is a fully open-source robotic arm designed for human manipulation data collection - developed by the Reazon Human Interaction Lab in Japan. V0.2 beta features smoother teleop with gravity compensation and bilateral force feedback.

OpenArm is a fully open-source robotic arm designed for human manipulation data collection - developed by the Reazon Human Interaction Lab in Japan. V0.2 beta features smoother teleop with gravity compensation and bilateral force feedback.

The Humanoid Hub

38,018 views • 1 year ago

Can we learn whole-body mobile manipulation directly from human demonstrations? Introducing Whole-Body Mobile Manipulation Interface (HoMMI) Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception

Can we learn whole-body mobile manipulation directly from human demonstrations? Introducing Whole-Body Mobile Manipulation Interface (HoMMI) Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception

Xiaomeng Xu ✈️ CVPR

73,832 views • 2 months ago

Robostral can now follow natural language instructions. It responds to voice commands and pointing. It is also getting better at fine-grained manipulation where precise force control matters. It generalizes to new objects and tasks not present in the training data.

Robostral can now follow natural language instructions. It responds to voice commands and pointing. It is also getting better at fine-grained manipulation where precise force control matters. It generalizes to new objects and tasks not present in the training data.

Olivier Duchenne

41,656 views • 10 days ago

Introducing the new Atlas3 - Triton Update! 🌟 Triton Update is an entirely new redesign with a fresh design language and a better user experience. It combines all the feedback we've received over the past year. Enjoy a fresh design and easy access to all the features.

Introducing the new Atlas3 - Triton Update! 🌟 Triton Update is an entirely new redesign with a fresh design language and a better user experience. It combines all the feedback we've received over the past year. Enjoy a fresh design and easy access to all the features.

Blocksmith Labs

61,171 views • 1 year ago

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Physical Intelligence

116,806 views • 1 year ago

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI Breakfast

33,718 views • 3 years ago

State-of-the-art robot policies often need hundreds of hours of data. What if we needed none? Introducing TiPToP: a manipulation system that zero-shots open-world tasks from pixels and language using vision foundation models and GPU-parallelized Task and Motion Planning (TAMP).

State-of-the-art robot policies often need hundreds of hours of data. What if we needed none? Introducing TiPToP: a manipulation system that zero-shots open-world tasks from pixels and language using vision foundation models and GPU-parallelized Task and Motion Planning (TAMP).

Nishanth Kumar

77,200 views • 2 months ago

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,739 views • 1 year ago

We have seen a lot of legged robots doing navigation in the wild. But how about mobile manipulation in the wild? I have been pushing the direction of learning a unified, efficient, and dynamic 3D representation of scenes (for navigation) and objects (for manipulation) for the past two years. And now we have GeFF --- our large-scale, generalizable feature field, that combines the speed of a feed-forward neural network with the rich semantics from Foundation Models, to handle dynamically changing scenes, and enable open-ended, language-grounded scene and object understanding.

We have seen a lot of legged robots doing navigation in the wild. But how about mobile manipulation in the wild? I have been pushing the direction of learning a unified, efficient, and dynamic 3D representation of scenes (for navigation) and objects (for manipulation) for the past two years. And now we have GeFF --- our large-scale, generalizable feature field, that combines the speed of a feed-forward neural network with the rich semantics from Foundation Models, to handle dynamically changing scenes, and enable open-ended, language-grounded scene and object understanding.

Xiaolong Wang

42,767 views • 2 years ago

Today, we introduce HunyuanImage 3.0-Instruct, a native multimodal model focusing on image-editing by integrating visual understanding with precise image synthesis! 🚀 It understands input images and reasons before generating images. Built on an 80B-parameter MoE architecture (13B activated), it natively unifies deep multimodal comprehension and high-fidelity generation. 🧠 A "Thinking" Model with Native CoT & MixGRPO: The model doesn’t just execute commands, it processes them through a Native Chain-of-Thought (CoT) schema. Enhanced by our self-developed MixGRPO algorithm, it reasons through complex instructions to achieve flawless intent alignment and human-preference consistency. 🎨 Precise Editing & Multi-Image Fusion: The model enables accurate image editing by adding, removing, or modifying elements while keeping non-target areas perfectly intact. It also excels at seamless multi-image fusion, synthesizing complex scenes by extracting and blending elements from multiple sources into a unified, consistent output. 🏆 SOTA Performance: HunyuanImage 3.0-Instruct sets a new benchmark in visual quality and alignment, delivering performance that matches leading proprietary models. We aim to enable the community to explore new ideas with a state-of-the-art foundation model, fostering a dynamic and vibrant image generation ecosystem. 🛠️🎨 💻Try it at (PC only):

Today, we introduce HunyuanImage 3.0-Instruct, a native multimodal model focusing on image-editing by integrating visual understanding with precise image synthesis! 🚀 It understands input images and reasons before generating images. Built on an 80B-parameter MoE architecture (13B activated), it natively unifies deep multimodal comprehension and high-fidelity generation. 🧠 A "Thinking" Model with Native CoT & MixGRPO: The model doesn’t just execute commands, it processes them through a Native Chain-of-Thought (CoT) schema. Enhanced by our self-developed MixGRPO algorithm, it reasons through complex instructions to achieve flawless intent alignment and human-preference consistency. 🎨 Precise Editing & Multi-Image Fusion: The model enables accurate image editing by adding, removing, or modifying elements while keeping non-target areas perfectly intact. It also excels at seamless multi-image fusion, synthesizing complex scenes by extracting and blending elements from multiple sources into a unified, consistent output. 🏆 SOTA Performance: HunyuanImage 3.0-Instruct sets a new benchmark in visual quality and alignment, delivering performance that matches leading proprietary models. We aim to enable the community to explore new ideas with a state-of-the-art foundation model, fostering a dynamic and vibrant image generation ecosystem. 🛠️🎨 💻Try it at (PC only):

Tencent Hy

125,659 views • 4 months ago

AGIBOT founder Zhihui Peng unveiled the Lingxi X2, a bipedal humanoid robot integrating motion, interaction, and task intelligence. With 28 degrees of freedom, a Diffusion-based motion engine, and a computing system that enables natural human-robot interaction. It can walk, run, dance, and ride a bicycle, while also excelling in precise object manipulation.

AGIBOT founder Zhihui Peng unveiled the Lingxi X2, a bipedal humanoid robot integrating motion, interaction, and task intelligence. With 28 degrees of freedom, a Diffusion-based motion engine, and a computing system that enables natural human-robot interaction. It can walk, run, dance, and ride a bicycle, while also excelling in precise object manipulation.

The Humanoid Hub

124,533 views • 1 year ago

Changing a light bulb with a robot! 💡 Researchers at the Korea Institute of Machinery & Materials developed a gripper that can handle complex manipulation tasks like bulb and bolt installation. It has three fingers with nine degrees of freedom (9 DoF), allowing precise control. Each finger moves using a special linkage system that enables flexible motion. The gripper can grasp objects of different shapes and adjust their positions accurately. Tests showed that it can lift up to 12 kg and apply a fingertip force of 40 N with high precision. Complex manipulation can be handled by robots! 🦾

Changing a light bulb with a robot! 💡 Researchers at the Korea Institute of Machinery & Materials developed a gripper that can handle complex manipulation tasks like bulb and bolt installation. It has three fingers with nine degrees of freedom (9 DoF), allowing precise control. Each finger moves using a special linkage system that enables flexible motion. The gripper can grasp objects of different shapes and adjust their positions accurately. Tests showed that it can lift up to 12 kg and apply a fingertip force of 40 N with high precision. Complex manipulation can be handled by robots! 🦾

Lukas Ziegler

27,548 views • 1 year ago

I've always dreamed of a multi-modal canvas, and when i was at figma that meant code + direct manipulation, and now it means agents + code + direct manipulation I might cry

I've always dreamed of a multi-modal canvas, and when i was at figma that meant code + direct manipulation, and now it means agents + code + direct manipulation I might cry

Michael Feldstein

11,136 views • 2 months ago

Chinese humanoid robotics company LimX Dynamics has unveiled COSA (Cognitive Operating System of Agents). COSA is described as a unified "brain-body" architecture that allows the robot to think and act simultaneously in the real world. It integrates: - high-level cognition (reasoning, planning, adaptation) - and whole-body motion control (low-latency dynamic locomotion, manipulation) It enables the Oli robot to act on natural language instructions and perform tasks that involve both walking and manipulation while being adaptive to interruptions.

Chinese humanoid robotics company LimX Dynamics has unveiled COSA (Cognitive Operating System of Agents). COSA is described as a unified "brain-body" architecture that allows the robot to think and act simultaneously in the real world. It integrates: - high-level cognition (reasoning, planning, adaptation) - and whole-body motion control (low-latency dynamic locomotion, manipulation) It enables the Oli robot to act on natural language instructions and perform tasks that involve both walking and manipulation while being adaptive to interruptions.

The Humanoid Hub

44,724 views • 4 months ago