正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Can we leverage VLMs for robot manipulation in the open world? Checkout our new work MOKA, a simple and effective visual prompting method!

Fangchen Liu

1,414 subscribers

81,193 次观看 • 2 年前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Long-horizon visual goals remain surprisingly hard for robot manipulation. We introduce Act2Goal, a goal-conditioned policy that uses a visual world model to reason about progress toward a goal, and practice it autonomously in the real world.

Long-horizon visual goals remain surprisingly hard for robot manipulation. We introduce Act2Goal, a goal-conditioned policy that uses a visual world model to reason about progress toward a goal, and practice it autonomously in the real world.

Jianlan Luo

95,100 次观看 • 7 个月前

How can we more effectively leverage robot data from different embodiments for skill transfer? Excited to share that our new work, RoVi-Aug, has been accepted to Conference on Robot Learning as an oral paper！ WIth RoVi-Aug, you can augment an existing robot dataset into a different robot and different viewpoints. A policy trained on the augmented dataset can zero-shot deploy on the unseen target robot with significantly different camera angles! 🧵👇 🔗 Check out our paper:

How can we more effectively leverage robot data from different embodiments for skill transfer? Excited to share that our new work, RoVi-Aug, has been accepted to Conference on Robot Learning as an oral paper！ WIth RoVi-Aug, you can augment an existing robot dataset into a different robot and different viewpoints. A policy trained on the augmented dataset can zero-shot deploy on the unseen target robot with significantly different camera angles! 🧵👇 🔗 Check out our paper:

Chenfeng_X

28,308 次观看 • 1 年前

Evaluating policies on a real robot can be painful. Can we use a world model to get a rough estimate of how good a policy is? Checkout "Evaluating Robot Policies in a World Model". Paper: Demo: Code:

Evaluating policies on a real robot can be painful. Can we use a world model to get a rough estimate of how good a policy is? Checkout "Evaluating Robot Policies in a World Model". Paper: Demo: Code:

Sherry Yang

36,692 次观看 • 1 年前

🤖Why is robot manipulation still an open challenge? This video shows a kitting task -- packing multiple items into a single product. No robot today can do this autonomously. Big challenges = big opportunities for research and industry. #robotics #manipulation #automation

🤖Why is robot manipulation still an open challenge? This video shows a kitting task -- packing multiple items into a single product. No robot today can do this autonomously. Big challenges = big opportunities for research and industry. #robotics #manipulation #automation

Yu Xiang

21,236 次观看 • 1 年前

How can we leverage diverse human videos to improve robot manipulation? Excited to introduce EgoVLA — a Vision-Language-Action model trained on egocentric human videos by explicitly modeling wrist & hand motion. We build a shared action space between humans and robots, enabling seamless transfer. With some robot demos, EgoVLA becomes a powerful, generalizable robot policy.

How can we leverage diverse human videos to improve robot manipulation? Excited to introduce EgoVLA — a Vision-Language-Action model trained on egocentric human videos by explicitly modeling wrist & hand motion. We build a shared action space between humans and robots, enabling seamless transfer. With some robot demos, EgoVLA becomes a powerful, generalizable robot policy.

Ruihan Yang

58,800 次观看 • 1 年前

This is how a simple self-defence technique can help you escape danger — effective moves focus on leverage, balance, and timing rather than brute force.

This is how a simple self-defence technique can help you escape danger — effective moves focus on leverage, balance, and timing rather than brute force.

Fight Club

39,741 次观看 • 2 个月前

Turns out you can train humanoid hands without any robot data. The idea in HUG is quite simple: (a) collect human data with smart glasses, (b) train a human manipulation model, (c) retarget to multi-fingered robot hands.

Turns out you can train humanoid hands without any robot data. The idea in HUG is quite simple: (a) collect human data with smart glasses, (b) train a human manipulation model, (c) retarget to multi-fingered robot hands.

Lerrel Pinto

33,816 次观看 • 2 个月前

Learn about how to make humanoid robot teleoperation work for humanoid *mobile* manipulation in this podcast. Really cool work by Siyuan Huang and yixuan_li!

Learn about how to make humanoid robot teleoperation work for humanoid mobile manipulation in this podcast. Really cool work by Siyuan Huang and yixuan_li!

Chris Paxton

25,060 次观看 • 1 年前

🚀 Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. Our findings fly in the face of conventional wisdom! This is a big joint effort from PRIOR Ai2 (6 first authors!).

🚀 Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. Our findings fly in the face of conventional wisdom! This is a big joint effort from PRIOR Ai2 (6 first authors!).

Kiana Ehsani

61,032 次观看 • 2 年前

We started using super-b trailers for loading all our drills. Just trying a new method of filling in the fields.

We started using super-b trailers for loading all our drills. Just trying a new method of filling in the fields.

Tyler Burns

22,253 次观看 • 1 年前

Spent the past few weeks building a fully automated pipeline for egocentric manipulation data with Kyle Jiang. Any new manipulation task captured using our own wearable sensing platform can now be dropped into the same pipeline with minimal manual fix. Instead of raw videos and annotations, we transform human demonstrations into robot-ready training data. Next, we're bringing touch into the loop with tactile gloves.

Spent the past few weeks building a fully automated pipeline for egocentric manipulation data with Kyle Jiang. Any new manipulation task captured using our own wearable sensing platform can now be dropped into the same pipeline with minimal manual fix. Instead of raw videos and annotations, we transform human demonstrations into robot-ready training data. Next, we're bringing touch into the loop with tactile gloves.

Grace Zhang

23,110 次观看 • 8 个月前

There are many things going on in the world, many evil things, the most important is the manipulation of our the air we breathe and the manipulation of the weather. Stay focused.

There are many things going on in the world, many evil things, the most important is the manipulation of our the air we breathe and the manipulation of the weather. Stay focused.

The GeoFight

58,547 次观看 • 6 个月前

We always open the windows for the evening birdsong and sunset at our farm here in New Hampshire.

We always open the windows for the evening birdsong and sunset at our farm here in New Hampshire.

Shaun Terhune

19,498 次观看 • 2 个月前

We’ve seen humanoid robots walk around for a while, but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:

We’ve seen humanoid robots walk around for a while, but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:

Yanjie Ze

75,312 次观看 • 1 年前

NVIDIA announces the first open humanoid robot reference design built for robotics research. The NVIDIA Isaac GR00T Reference Humanoid Robot combines the Unitree H2 humanoid robot, Sharpa Wave five-fingered hands for dexterous manipulation, Jetson Thor onboard compute, and Isaac GR00T open software and models, giving researchers a full-stack platform from data capture to model deployment. Read the #NVIDIAGTC Taipei announcement:

NVIDIA announces the first open humanoid robot reference design built for robotics research. The NVIDIA Isaac GR00T Reference Humanoid Robot combines the Unitree H2 humanoid robot, Sharpa Wave five-fingered hands for dexterous manipulation, Jetson Thor onboard compute, and Isaac GR00T open software and models, giving researchers a full-stack platform from data capture to model deployment. Read the #NVIDIAGTC Taipei announcement:

NVIDIA Robotics

169,902 次观看 • 2 个月前

Learning bimanual, contact-rich robot manipulation policies that generalize over diverse objects has long been a challenge. Excited to share our work: Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation! 🧵1/n

Learning bimanual, contact-rich robot manipulation policies that generalize over diverse objects has long been a challenge. Excited to share our work: Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation! 🧵1/n

Xuanlin Li (Simon)

20,925 次观看 • 1 年前

World Models are the path for some AI Models in the future. But how can we efficiently train these models to not only see the world the way humans do but to see the world in a new and unique way. By visualizing, what is normally sequenced audio patterns, we can derive much more insights. Here we see Paganini in a visual form that can than be described and transcribed into a World Model. We can observe connections in a manner that may not have been clear prior to the digitalization of music and sound in this way. The company with the most valuable potential in building a World Model is Tesla. Not that this type of visualization is being used, but that the mechanisms are in place, and the technology is in place for the company to thrive in this new form of AI.

World Models are the path for some AI Models in the future. But how can we efficiently train these models to not only see the world the way humans do but to see the world in a new and unique way. By visualizing, what is normally sequenced audio patterns, we can derive much more insights. Here we see Paganini in a visual form that can than be described and transcribed into a World Model. We can observe connections in a manner that may not have been clear prior to the digitalization of music and sound in this way. The company with the most valuable potential in building a World Model is Tesla. Not that this type of visualization is being used, but that the mechanisms are in place, and the technology is in place for the company to thrive in this new form of AI.

Brian Roemmele

57,454 次观看 • 9 个月前

Introducing Moto: Latent Motion Token as the Bridging Language for Robot Manipulation. Motion prior learned from videos can be seamlessly transferred to robot manipulation. Code and model released! Yi CHEN Yuying Ge Yixiao Ge Mingyu Ding Ying Shan

Introducing Moto: Latent Motion Token as the Bridging Language for Robot Manipulation. Motion prior learned from videos can be seamlessly transferred to robot manipulation. Code and model released! Yi CHEN Yuying Ge Yixiao Ge Mingyu Ding Ying Shan

Xihui Liu

15,030 次观看 • 1 年前

World modeling and imitation learning have largely been considered two disparate worlds. In our recent work, Unified World Models, just accepted to #RSS2025, Chuning Zhu provides a dead-simple unifying solution: just train a joint diffusion model over actions and future states, but with *decoupled* diffusion time steps across these modalities. Manipulating these decoupled time steps then allows for marginalization or conditioning on actions or states; a single model can serve as a policy, forward dynamics model, video prediction model, or inverse dynamics model by simply setting diffusion timesteps carefully. The resulting model can leverage video datasets along with robot training data much more effectively, and shows improved robustness, generalization, and flexibility. This is exciting because it is frustratingly simple, scalable, and shows strong improvement on real-world robotics problems. Please refer to Chuning Zhu 's excellent thread for more details! More details/code can be found on our website and in the paper -

World modeling and imitation learning have largely been considered two disparate worlds. In our recent work, Unified World Models, just accepted to #RSS2025, Chuning Zhu provides a dead-simple unifying solution: just train a joint diffusion model over actions and future states, but with decoupled diffusion time steps across these modalities. Manipulating these decoupled time steps then allows for marginalization or conditioning on actions or states; a single model can serve as a policy, forward dynamics model, video prediction model, or inverse dynamics model by simply setting diffusion timesteps carefully. The resulting model can leverage video datasets along with robot training data much more effectively, and shows improved robustness, generalization, and flexibility. This is exciting because it is frustratingly simple, scalable, and shows strong improvement on real-world robotics problems. Please refer to Chuning Zhu 's excellent thread for more details! More details/code can be found on our website and in the paper -

Abhishek Gupta

11,430 次观看 • 1 年前

Can MLLMs actually track what's happening in a video? Introducing VSTAT 🎯, our new benchmark for visual state tracking. The tasks are simple: count cups, read typed words, count page flips. Humans solve them easily. MLLMs don't. 🧵 [1/11]

Can MLLMs actually track what's happening in a video? Introducing VSTAT 🎯, our new benchmark for visual state tracking. The tasks are simple: count cups, read typed words, count page flips. Humans solve them easily. MLLMs don't. 🧵 [1/11]

Sihyun Yu

167,352 次观看 • 2 个月前