Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Robots need strong visuo-motor representations to manipulate objects, but it’s hard to learn these using demo data alone. Our #RSS2024 project vastly improves robotic representations, using human affordances mined from Ego4D! w/ Mohan Kumar Srirama Shikhar Bahl Abhinav Gupta

Sudeep Dasari

1,481 subscribers

11,022 views • 1 year ago •via X (Twitter)

Science & Technology Education #RSS2024

Anya Rossi• Live Now

Private livecam show

2 Comments

Sudeep Dasari1 year ago

If you’re at @RoboticsSciSys in Delft, drop by our presentation and poster this Wednesday (session 10) at 1:30 PM! Website: Paper:

Yang1 year ago

Want to learn how practical AI skills and automations for your business and work? Check out our 50+ step-by-step video tutorials 100% FREE 20+ hours of Ai and Automation goodness absolutely free 🥳

Related Videos

We bust some common myths in visuo-motor pretraining! Our data-centric experiments show: 📷 ImageNet > Ego4D for *self-supervised* robotic pretraining 🏆 Diversity matters more than content: training on a broader image distribution beats SOTA by 30% 📊Sim evals can mislead! 1/n

We bust some common myths in visuo-motor pretraining! Our data-centric experiments show: 📷 ImageNet > Ego4D for self-supervised robotic pretraining 🏆 Diversity matters more than content: training on a broader image distribution beats SOTA by 30% 📊Sim evals can mislead! 1/n

Sudeep Dasari

54,329 views • 2 years ago

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

Chenhao Li

13,649 views • 11 days ago

Our Robotic Foundation Model, trained on data from real-world warehouse operations and simulated scenarios, enables our robots to identify objects, understand 3D spaces, and predict optimal grasping and placement.

Our Robotic Foundation Model, trained on data from real-world warehouse operations and simulated scenarios, enables our robots to identify objects, understand 3D spaces, and predict optimal grasping and placement.

Covariant

33,508 views • 1 year ago

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

Hila Chefer

67,523 views • 3 months ago

Classic childhood activities like tea parties and sword fights with sticks demonstrate the human ability to generate secondary representations, conditions we know aren’t “real” but that we nonetheless engage with. Whether nonhuman animals are capable of these types of representations has been difficult to test. In a new Science study, researchers studied a language-trained bonobo, Kanzi, to see whether he could understand and engage with pretend conditions. Across three different experiments, Kanzi was able to identify pretend objects, demonstrating that he could create a secondary representation and showing that humans are not alone in this ability. Learn more:

Classic childhood activities like tea parties and sword fights with sticks demonstrate the human ability to generate secondary representations, conditions we know aren’t “real” but that we nonetheless engage with. Whether nonhuman animals are capable of these types of representations has been difficult to test. In a new Science study, researchers studied a language-trained bonobo, Kanzi, to see whether he could understand and engage with pretend conditions. Across three different experiments, Kanzi was able to identify pretend objects, demonstrating that he could create a secondary representation and showing that humans are not alone in this ability. Learn more:

Science Magazine

25,565 views • 4 months ago

Using monkey brain data to operate robotic arms! 🙉 Project Jenkins explores brain-computer interfaces by decoding neural activity into robotic movement and generating synthetic brain data from motion patterns. Using neural recordings from a macaque monkey named Jenkins, researchers developed models to decode brain signals into robotic arm movements and encode movement data into simulated neural activity. The project utilizes robotic arms and an interactive web console for real-time brain data generation from joystick inputs. The research paves the way for adaptive brain-computer interfaces by modeling brain activity more flexibly. Congrats to authors Dima Yanovsky, and Andrii Zahorodnii! 🦾

Using monkey brain data to operate robotic arms! 🙉 Project Jenkins explores brain-computer interfaces by decoding neural activity into robotic movement and generating synthetic brain data from motion patterns. Using neural recordings from a macaque monkey named Jenkins, researchers developed models to decode brain signals into robotic arm movements and encode movement data into simulated neural activity. The project utilizes robotic arms and an interactive web console for real-time brain data generation from joystick inputs. The research paves the way for adaptive brain-computer interfaces by modeling brain activity more flexibly. Congrats to authors Dima Yanovsky, and Andrii Zahorodnii! 🦾

Lukas Ziegler

21,504 views • 1 year ago

Using computer vision techniques, the robot analyzed each frame of video and was able to identify the different objects and features. In the future, robots will be able to learn human behaviors just by watching videos.

Using computer vision techniques, the robot analyzed each frame of video and was able to identify the different objects and features. In the future, robots will be able to learn human behaviors just by watching videos.

Aakash Gupta

55,257 views • 3 years ago

Classic childhood activities like tea parties and sword fights with sticks demonstrate the human ability to generate secondary representations, conditions we know aren’t “real” but that we nonetheless engage with. Whether nonhuman animals are capable of these types of representations has been difficult to test. In a new Science study, researchers studied a language-trained bonobo, Kanzi, to see whether he could understand and engage with pretend conditions. Across three different experiments, Kanzi was able to identify pretend objects, demonstrating that he could create a secondary representation and showing that humans are not alone in this ability. Learn more:

Classic childhood activities like tea parties and sword fights with sticks demonstrate the human ability to generate secondary representations, conditions we know aren’t “real” but that we nonetheless engage with. Whether nonhuman animals are capable of these types of representations has been difficult to test. In a new Science study, researchers studied a language-trained bonobo, Kanzi, to see whether he could understand and engage with pretend conditions. Across three different experiments, Kanzi was able to identify pretend objects, demonstrating that he could create a secondary representation and showing that humans are not alone in this ability. Learn more:

Science Magazine

24,825 views • 4 months ago

Having humans annotate data to pre-train robots is expensive and time-consuming! Introducing SPRINT: A pre-training approach using LLMs and offline RL to equip robots w/ many language-annotated skills while minimizing human annotation effort! URL: 🧵👇

Having humans annotate data to pre-train robots is expensive and time-consuming! Introducing SPRINT: A pre-training approach using LLMs and offline RL to equip robots w/ many language-annotated skills while minimizing human annotation effort! URL: 🧵👇

Jesse Zhang

24,662 views • 3 years ago

Teaching robots real dexterity has always been a challenge. But what if they could handle tools like a human? DexterityGen (DexGen) is a new system that helps robots use their hands better. It improves how they grip, move, and handle objects… from holding a pen to using a screwdriver. DexGen learns in simulation and refines its skills in the real world, making robotic hands much more useful. What makes DexGen special? ✅ Smarter movements that refine rough actions into precise skills ✅ Trained on a massive collection of dexterous tasks for better learning ✅ Better teleoperation that makes robotic hand control easier and safer ✅ Handles real-world challenges like small objects, tricky angles, and gravity This moves robots closer to real dexterity. It makes tool use more natural, improves stability, and brings robotic hands one step closer to human-level skill. Seen at Zhao-Heng Yin 🫶 Github: Paper:

Teaching robots real dexterity has always been a challenge. But what if they could handle tools like a human? DexterityGen (DexGen) is a new system that helps robots use their hands better. It improves how they grip, move, and handle objects… from holding a pen to using a screwdriver. DexGen learns in simulation and refines its skills in the real world, making robotic hands much more useful. What makes DexGen special? ✅ Smarter movements that refine rough actions into precise skills ✅ Trained on a massive collection of dexterous tasks for better learning ✅ Better teleoperation that makes robotic hand control easier and safer ✅ Handles real-world challenges like small objects, tricky angles, and gravity This moves robots closer to real dexterity. It makes tool use more natural, improves stability, and brings robotic hands one step closer to human-level skill. Seen at Zhao-Heng Yin 🫶 Github: Paper:

Ilir Aliu

51,490 views • 1 year ago

How can robots learn generalizable manipulation skills for diverse objects? Going beyond pick-and-place, our recent work “HACMan” enables complex interactions for unseen objects, such as flipping, pushing, or tilting, using spatial action maps + RL with point clouds. (w/ @MetaAI)

How can robots learn generalizable manipulation skills for diverse objects? Going beyond pick-and-place, our recent work “HACMan” enables complex interactions for unseen objects, such as flipping, pushing, or tilting, using spatial action maps + RL with point clouds. (w/ @MetaAI)

Wenxuan Zhou

49,846 views • 3 years ago

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

Ilir Aliu - eu/acc

10,509 views • 1 year ago

Excited to share my final PhD project😀 We show how simple, yet elegant changes enable diffusion transformers to learn SOTA robotic policies on real robots. Our method improves performance by 20% across a wide range of highly dexterous tasks - like cutting sushi! 1/n

Excited to share my final PhD project😀 We show how simple, yet elegant changes enable diffusion transformers to learn SOTA robotic policies on real robots. Our method improves performance by 20% across a wide range of highly dexterous tasks - like cutting sushi! 1/n

Sudeep Dasari

20,536 views • 1 year ago

Introducing: AutoFAB (Autonomous Factory-in-a-Box) World's First Self-Replicating Robotic Factory. Using Robots to Build Robots Faster & Cheaper. AutoFAB

Introducing: AutoFAB (Autonomous Factory-in-a-Box) World's First Self-Replicating Robotic Factory. Using Robots to Build Robots Faster & Cheaper. AutoFAB

Ansel Dias

11,934 views • 2 months ago

This is how large language models turn objects to vector representations. In this video, we explore how large language models (LLMs) convert objects into internal representations, especially when translating between languages like English and Hindi. Using real-world examples, we highlight the challenges of gender inference, grammatical structure, and why direct word-to-word translations often fail. If you're curious about how LLMs deal with multilingual contexts and what it takes to improve translation quality across languages, this video is for you. #LLMs #Vectors #LCM

This is how large language models turn objects to vector representations. In this video, we explore how large language models (LLMs) convert objects into internal representations, especially when translating between languages like English and Hindi. Using real-world examples, we highlight the challenges of gender inference, grammatical structure, and why direct word-to-word translations often fail. If you're curious about how LLMs deal with multilingual contexts and what it takes to improve translation quality across languages, this video is for you. #LLMs #Vectors #LCM

Gaurav Sen

27,368 views • 1 year ago

While travelling across the State, I receive numerous representations from the people regarding their issues. All these representations are carefully studied by my office and concerned offices are directed to solve people's issues at the earliest. 📍 Lakhimpur

While travelling across the State, I receive numerous representations from the people regarding their issues. All these representations are carefully studied by my office and concerned offices are directed to solve people's issues at the earliest. 📍 Lakhimpur

Himanta Biswa Sarma

56,849 views • 1 year ago

Did you know that Antigravity can be used not just to generate data, but also manipulate that data in Google Sheets using the browser integration?

Did you know that Antigravity can be used not just to generate data, but also manipulate that data in Google Sheets using the browser integration?

Google Antigravity

131,294 views • 5 months ago

Drop the camera, and bang -- the robots do the job without camera calibration or additional data collection. We want robots to manipulate anywhere! Our new work, **Maniwhere**, enables robots to manipulate objects from any camera view with any background. Isn't this the generalization every roboticist is looking for? How do we achieve this with sim2real? You can check this thread from Zhecheng Yuan ! And the website is here:

Drop the camera, and bang -- the robots do the job without camera calibration or additional data collection. We want robots to manipulate anywhere! Our new work, Maniwhere, enables robots to manipulate objects from any camera view with any background. Isn't this the generalization every roboticist is looking for? How do we achieve this with sim2real? You can check this thread from Zhecheng Yuan ! And the website is here:

Huazhe Harry Xu

26,214 views • 1 year ago

Google Researchers worked closely with sciences in the Human Pangenome project to improve the pangenome using Google Research's DeepVariant and DeepConsensus methods, which use deep learning to improve the quality of genomics data. Learn more →

Google Researchers worked closely with sciences in the Human Pangenome project to improve the pangenome using Google Research's DeepVariant and DeepConsensus methods, which use deep learning to improve the quality of genomics data. Learn more →

Google AI

118,540 views • 2 years ago