正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Meta just open-sourced its dexterity stack! 🪬 Most physics simulators were built for things that move through space. Walking robots, drones, cars. Contact is the part they approximate worst, and obviously dexterous manipulation is nothing but contact. Project SuperDex, from Meta Reality Labs Research, is built the other way... around, a contact-first physics engine with the whole platform stacked on top of it. The cool part is that it's on GitHub. The engine runs one solver across rigid bodies, soft bodies, rods and tendons, shells and cloth, in the same model. → Non-convex collision with accurate contact force distributions, so a multi-finger grasp gets simulated rather than approximated → Tactile sensors and soft contact as first-class primitives → Numerical stability without the tight time-step limits explicit solvers force on you → Constraint-aware inverse kinematics running on the same optimization core as the forward dynamics Then the data layer. Put on a Quest 3, teleoperate the simulated hand with haptic feedback, and generate demonstration datasets without touching real hardware. They show a shape-sorting policy trained entirely in simulation and deployed zero-shot on a real robotic hand. Robot hands are getting good. Data for them isn't that fast. Meta is betting the cheapest way to collect contact-rich demonstrations is a headset people already own, pointed at a simulator instead of a game. 🔗 Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →show more

Lukas Ziegler

64,928 subscribers

60,922 次观看 • 2 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

9 条评论

Gill 的头像

Gill2 个月前

Physics engines usually ignore soft contacts so this could change robotics sim a lot.

Vad | Exylos Robotics 的头像

Vad | Exylos Robotics2 个月前

Went through the repo today and the Quest teleop part isn't in there yet. Waiting for that piece to land to test it properly, mostly curious how much of the desktop engine actually runs natively on the headset.

Michael 的头像

Michael2 个月前

@pangram

Dima Nurm 的头像

Dima Nurm2 个月前

fine motor skills for Robo Touch

Muhammad Ahmed 的头像

Muhammad Ahmed1 个月前

Contact is where small simulation errors become obvious. Does SuperDex explicitly model fingertip compliance and friction variation, or depend mainly on domain randomization for sim-to-real transfer?

Adrian Xue 的头像

Adrian Xue2 个月前

So are the data collection gloves still valuable?

Telixia 的头像

Telixia2 个月前

A contact-first engine solves the biggest bottleneck in robotic manipulation. Leveraging Quest 3 headsets to collect synthetic demonstration data at scale is a brilliant move that will fast-track zero-shot deployment for complex physical AI. @lukas_m_ziegler.Thanks for sharing

Diego Boss 的头像

Diego Boss1 个月前

Super interesting

Sebastian Buzdugan 的头像

Sebastian Buzdugan2 个月前

better contact simulation still misses tendon friction, which often breaks transfer to real hands

相关视频

Robot hands still break most simulators. Meta Reality Labs just open-sourced Project SuperDex — a contact-first physics engine built for the hard part: multi-finger grasps, in-hand reorientation, soft contact, tactile sensing. VR teleop for cheap demo data. Gym-style RL on top. projectsuperdex. com

Robot hands still break most simulators. Meta Reality Labs just open-sourced Project SuperDex — a contact-first physics engine built for the hard part: multi-finger grasps, in-hand reorientation, soft contact, tactile sensing. VR teleop for cheap demo data. Gym-style RL on top. projectsuperdex. com

Carlos E. Perez

27,741 次观看 • 2 个月前

Open-source dexterous hands with fingertip sensors! 🪬 ORCA Dexterity just released three dexterous hand models. Their mission: is to democratize robotic hand dexterity. They're sharing progress on exceptional, low-cost hardware and a software layer from low-level control to robotic hand learning. This is the open-source approach to dexterous manipulation. It has 83 taxels per finger with 0.1 N force detection which pretty is impressive for an open-source design. Tactile sensing is critical for dexterous manipulation, knowing contact forces enables gentle grasping, slip detection, and force-controlled assembly. Also 700g weight for the lite version makes it practical for mounting on robot arms without exceeding payload limits. Lower weight means faster movements and lower torque requirements. Open hardware accelerates robotics by letting researchers and builders modify designs for their specific needs without starting from scratch! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Open-source dexterous hands with fingertip sensors! 🪬 ORCA Dexterity just released three dexterous hand models. Their mission: is to democratize robotic hand dexterity. They're sharing progress on exceptional, low-cost hardware and a software layer from low-level control to robotic hand learning. This is the open-source approach to dexterous manipulation. It has 83 taxels per finger with 0.1 N force detection which pretty is impressive for an open-source design. Tactile sensing is critical for dexterous manipulation, knowing contact forces enables gentle grasping, slip detection, and force-controlled assembly. Also 700g weight for the lite version makes it practical for mounting on robot arms without exceeding payload limits. Lower weight means faster movements and lower torque requirements. Open hardware accelerates robotics by letting researchers and builders modify designs for their specific needs without starting from scratch! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

38,323 次观看 • 7 个月前

A lot of robot learning pipelines treat embodiment mismatch as a software problem. Collect data with one device, retarget the motion to another robot, and try to deal with the differences later. TwinDEX flips that idea around. This system is from a Chinese robotics company called X Square. Instead of starting with a generic data-collection device, the team co-designs a wearable three-finger end-effector and a robotic hand as two sides of the same learning system. They align the things that actually matter during manipulation: joint structure, link proportions, contact geometry, surface properties, visual appearance, sensing, and timing. That matters because dexterous manipulation is much more than matching hand poses. A few millimeters of contact error or a different force relationship can completely change how a task is executed. In the collection evaluation, TwinDEX delivered up to 5.3 times the effective throughput of on-robot teleoperation. It’s a useful reminder that scaling robot learning may not only be about bigger models and more data. Sometimes the hardware collecting the data is part of the data problem itself.

A lot of robot learning pipelines treat embodiment mismatch as a software problem. Collect data with one device, retarget the motion to another robot, and try to deal with the differences later. TwinDEX flips that idea around. This system is from a Chinese robotics company called X Square. Instead of starting with a generic data-collection device, the team co-designs a wearable three-finger end-effector and a robotic hand as two sides of the same learning system. They align the things that actually matter during manipulation: joint structure, link proportions, contact geometry, surface properties, visual appearance, sensing, and timing. That matters because dexterous manipulation is much more than matching hand poses. A few millimeters of contact error or a different force relationship can completely change how a task is executed. In the collection evaluation, TwinDEX delivered up to 5.3 times the effective throughput of on-robot teleoperation. It’s a useful reminder that scaling robot learning may not only be about bigger models and more data. Sometimes the hardware collecting the data is part of the data problem itself.

Alamin

48,112 次观看 • 1 个月前

That's genuinely insane! 🤯 An AI agent built a physics twin of a robot task from a phone video in under an hour. This project RoboSimStudio was built on Newton by researchers from UC Berkeley and NVIDIA Robotics. Open toolkit, every physics parameter exposed as something a human can slide or an agent can edit. The demo is a bimanual setup, one arm holding a bar, the other wiping a marker stripe off it with a foam sponge. Input was a smartphone video. What's more important no CAD, and no calibrated cameras. Claude and GPT-6 Astra each built a simulation of it independently. One finished in 56 minutes. Then they ran it on the real robot and the bar support moved. In simulation it had been modelled as fixed, so the sponge kept contact. In reality it shifted and contact was lost. The agent went back and replaced the fixed constraint with a free rigid body. The revised simulation then reproduced the tipping the original could not. That is the loop actually closing. The real world found the modelling error and the fix went back into the twin. 🔗 Check the project here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

That's genuinely insane! 🤯 An AI agent built a physics twin of a robot task from a phone video in under an hour. This project RoboSimStudio was built on Newton by researchers from UC Berkeley and NVIDIA Robotics. Open toolkit, every physics parameter exposed as something a human can slide or an agent can edit. The demo is a bimanual setup, one arm holding a bar, the other wiping a marker stripe off it with a foam sponge. Input was a smartphone video. What's more important no CAD, and no calibrated cameras. Claude and GPT-6 Astra each built a simulation of it independently. One finished in 56 minutes. Then they ran it on the real robot and the bar support moved. In simulation it had been modelled as fixed, so the sponge kept contact. In reality it shifted and contact was lost. The agent went back and replaced the fixed constraint with a free rigid body. The revised simulation then reproduced the tipping the original could not. That is the loop actually closing. The real world found the modelling error and the fix went back into the twin. 🔗 Check the project here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

38,647 次观看 • 3 天前

A policy that teaches robot hands to touch things the way humans do... not just grab and move, but feel and adjust in real time. Robot manipulation research often stops at picking up objects and placing them. CGP goes further: it handles tasks like opening jars, flipping objects in-hand, wiping dishes, and grasping fragile eggs, the kind of dexterous, contact-rich skills that require constant micro-adjustments based on what the fingers are actually feeling. The robot doesn't just see what it's doing; it predicts what contact should feel like at each step, then checks whether reality matches the prediction. If a finger is slipping, the policy knows before the object drops. Works on real robot hands (both 4-finger and 5-finger designs) with tactile sensors embedded in the fingertips Robust to visual distractions! The robot keeps flipping a box correctly even when the camera view is disrupted, because it's grounding decisions in touch, not just vision. Baseline policies without contact grounding fail in predictable ways: slipping mid-task, incomplete motions, loss of grasp, CGP avoids these This is a meaningful step toward robots that can handle the physical world with the kind of reliable, adaptive grip that humans take for granted. Relevant for manufacturing, logistics, assistive robotics, and anywhere fragile or irregular objects need to be handled carefully. Published at RSS 2026, developed with Meta Reality Labs Research. Thanks for sharing, Zhengtong Xu / Zhengtong Xu ——- Weekly robotics and AI insights. Subscribe free:

A policy that teaches robot hands to touch things the way humans do... not just grab and move, but feel and adjust in real time. Robot manipulation research often stops at picking up objects and placing them. CGP goes further: it handles tasks like opening jars, flipping objects in-hand, wiping dishes, and grasping fragile eggs, the kind of dexterous, contact-rich skills that require constant micro-adjustments based on what the fingers are actually feeling. The robot doesn't just see what it's doing; it predicts what contact should feel like at each step, then checks whether reality matches the prediction. If a finger is slipping, the policy knows before the object drops. Works on real robot hands (both 4-finger and 5-finger designs) with tactile sensors embedded in the fingertips Robust to visual distractions! The robot keeps flipping a box correctly even when the camera view is disrupted, because it's grounding decisions in touch, not just vision. Baseline policies without contact grounding fail in predictable ways: slipping mid-task, incomplete motions, loss of grasp, CGP avoids these This is a meaningful step toward robots that can handle the physical world with the kind of reliable, adaptive grip that humans take for granted. Relevant for manufacturing, logistics, assistive robotics, and anywhere fragile or irregular objects need to be handled carefully. Published at RSS 2026, developed with Meta Reality Labs Research. Thanks for sharing, Zhengtong Xu / Zhengtong Xu ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

12,860 次观看 • 4 个月前

That's sick! 🤯 Genesis AI simulates robots playing yo-yo! 🪀 Genesis AI just open-sourced Genesis World 1.0, and it might be one of the most important infrastructure releases in robotics this year. Robotics is still bottlenecked by the 1× speed of the physical world. Every model needs to be tested on real hardware, slowly, expensively, with limited coverage. Genesis World 1.0 from Genesis AI flips that equation: One hour in reality becomes 100 days in simulation. That turns a wall-clock bottleneck into a compute problem. And compute problems are solvable. The technical stack they rebuilt from scratch is serious: → GPU-accelerated cross-platform compiler via Quadrants, 10x faster launch time and up to 4.6x runtime vs the initial Genesis release → Penetration-free multi-physics contact solvers, the thing that makes simulation actually trustworthy → Unified rigid AND deformable physics in a single engine → Nyx, a high-performance path-traced rendering engine purpose-built for physical AI The sim-to-real gap has historically been the graveyard of robotics research. Policies that work beautifully in simulation fall apart on real hardware. Genesis World 1.0 is a direct attack on that problem. And it's fully open-source. The companies that master simulation infrastructure will train better robots faster than anyone else. Find it here: Genesis World 1.0: Quadrants: Nyx: Theophile Gervet, Zhou Xian congrats! 👏🏼 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

That's sick! 🤯 Genesis AI simulates robots playing yo-yo! 🪀 Genesis AI just open-sourced Genesis World 1.0, and it might be one of the most important infrastructure releases in robotics this year. Robotics is still bottlenecked by the 1× speed of the physical world. Every model needs to be tested on real hardware, slowly, expensively, with limited coverage. Genesis World 1.0 from Genesis AI flips that equation: One hour in reality becomes 100 days in simulation. That turns a wall-clock bottleneck into a compute problem. And compute problems are solvable. The technical stack they rebuilt from scratch is serious: → GPU-accelerated cross-platform compiler via Quadrants, 10x faster launch time and up to 4.6x runtime vs the initial Genesis release → Penetration-free multi-physics contact solvers, the thing that makes simulation actually trustworthy → Unified rigid AND deformable physics in a single engine → Nyx, a high-performance path-traced rendering engine purpose-built for physical AI The sim-to-real gap has historically been the graveyard of robotics research. Policies that work beautifully in simulation fall apart on real hardware. Genesis World 1.0 is a direct attack on that problem. And it's fully open-source. The companies that master simulation infrastructure will train better robots faster than anyone else. Find it here: Genesis World 1.0: Quadrants: Nyx: Theophile Gervet, Zhou Xian congrats! 👏🏼 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

57,061 次观看 • 5 个月前

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN

Berkeley AI Research

112,249 次观看 • 28 天前

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

60,070 次观看 • 10 个月前

Pretty excited about the new SuperDex robotics simulator by Meta. 1. It's contact-first simulator in C++ designed for smooth, non-convex rigid–soft physics (vs mujoco designed for convex shapes) 2. Rather than standalone simulator, it includes a full suite of tools (RL/gym environments ("Superdex Lab"), GUI authoring tool ("Superdex Studio"), VR teleop). 3. It runs well on Mac! Trying it out below:

Pretty excited about the new SuperDex robotics simulator by Meta. 1. It's contact-first simulator in C++ designed for smooth, non-convex rigid–soft physics (vs mujoco designed for convex shapes) 2. Rather than standalone simulator, it includes a full suite of tools (RL/gym environments ("Superdex Lab"), GUI authoring tool ("Superdex Studio"), VR teleop). 3. It runs well on Mac! Trying it out below:

pfung

15,352 次观看 • 2 个月前

We are back again :) After three weeks of quiet building. Introducing Genesis World 1.0, our latest simulation platform, the second release in our full-stack suite. Open-sourced. Robotics is still bottlenecked by the 1× speed of the physical world. Every model, checkpoint, and data recipe eventually needs to be tested on physical hardware, slowly, expensively, and with limited coverage. One hour in reality can become 100 days in simulation. That is how robotics model iteration moves from a wall-clock bottleneck to a compute problem. To make this work, simulation has to be both fast and trustworthy. Over the past year, we rebuilt the entire stack: a GPU-accelerated cross-platform compiler, penetration-free multi-physics contact solvers, unified rigid and deformable physics, and a photo-realistic renderer purpose-built for physical AI applications. We built Nyx, a high-performance path-traced rendering engine for robotics application. Genesis World 1.0 achieves near realtime performance with our latest development for penetration-free IPC solver, supporting various types of deformables beyond rigid bodies. It supports contact-rich, dexterous manipulation simulation across different embodiments: unitree, sharpa, wuji, genesis hand and various types of grippers. Under the hood is Quadrants, our effort in pushing forward cross-platform GPU-accelerated computation. Quadrants started as a fork of Taichi, and we rebuilt most of the critical parts for optimizing simulation workloads, giving 10x faster launch time and up to 4.6x runtime performance compared to the initial Genesis release. Together, they bring us to an unprecedentedly low sim-to-real gap, enabling zero-shot real-to-sim model evaluation and much faster iteration of GENE. All available today. Genesis World 1.0: Quadrants: Nyx:

We are back again :) After three weeks of quiet building. Introducing Genesis World 1.0, our latest simulation platform, the second release in our full-stack suite. Open-sourced. Robotics is still bottlenecked by the 1× speed of the physical world. Every model, checkpoint, and data recipe eventually needs to be tested on physical hardware, slowly, expensively, and with limited coverage. One hour in reality can become 100 days in simulation. That is how robotics model iteration moves from a wall-clock bottleneck to a compute problem. To make this work, simulation has to be both fast and trustworthy. Over the past year, we rebuilt the entire stack: a GPU-accelerated cross-platform compiler, penetration-free multi-physics contact solvers, unified rigid and deformable physics, and a photo-realistic renderer purpose-built for physical AI applications. We built Nyx, a high-performance path-traced rendering engine for robotics application. Genesis World 1.0 achieves near realtime performance with our latest development for penetration-free IPC solver, supporting various types of deformables beyond rigid bodies. It supports contact-rich, dexterous manipulation simulation across different embodiments: unitree, sharpa, wuji, genesis hand and various types of grippers. Under the hood is Quadrants, our effort in pushing forward cross-platform GPU-accelerated computation. Quadrants started as a fork of Taichi, and we rebuilt most of the critical parts for optimizing simulation workloads, giving 10x faster launch time and up to 4.6x runtime performance compared to the initial Genesis release. Together, they bring us to an unprecedentedly low sim-to-real gap, enabling zero-shot real-to-sim model evaluation and much faster iteration of GENE. All available today. Genesis World 1.0: Quadrants: Nyx:

Genesis AI

326,131 次观看 • 5 个月前

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,130 次观看 • 10 个月前

Feeling what a robot feels is becoming real! 🫰 Fluid Reality just ran its first full end-to-end touch teleoperation system, and the demo shows why high-resolution haptics might be the missing piece in robot control. A 22 mm fingertip display with 32 independently actuated “bubbles” was mapped directly to tactile sensors on a robot hand. 🫧 Three fingers streamed real-time touch data robot → human, giving the operator actual contact feedback instead of waving their hands blind, like most teleop systems today. It’s still early, but this is a real step toward dexterous ops and high-quality data for training Physical AI systems. Companies involved: Samsung Electronics, Alt-Bionics, Inc. (the robot hand), MANUS™ (hand tracking), and Sensible Robotics (fingertip sensors) Touch isn’t a nice-to-have in robotics. It’s the difference between manipulation that works in the real world P.S. Guess how many touch receptors we humans, have in our hand! :) ~~ ♻ Join the weekly robotics newsletter, and never miss any news →

Feeling what a robot feels is becoming real! 🫰 Fluid Reality just ran its first full end-to-end touch teleoperation system, and the demo shows why high-resolution haptics might be the missing piece in robot control. A 22 mm fingertip display with 32 independently actuated “bubbles” was mapped directly to tactile sensors on a robot hand. 🫧 Three fingers streamed real-time touch data robot → human, giving the operator actual contact feedback instead of waving their hands blind, like most teleop systems today. It’s still early, but this is a real step toward dexterous ops and high-quality data for training Physical AI systems. Companies involved: Samsung Electronics, Alt-Bionics, Inc. (the robot hand), MANUS™ (hand tracking), and Sensible Robotics (fingertip sensors) Touch isn’t a nice-to-have in robotics. It’s the difference between manipulation that works in the real world P.S. Guess how many touch receptors we humans, have in our hand! :) ~~ ♻ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

46,927 次观看 • 10 个月前

What if a dexterous robot could learn a 20+ step chemistry experiment without a single on-robot training demo? Meet TwinDEX: a pair of co-designed, three-finger, nine-DoF dexterous manipulation interface: one wearable for data collection, one for robot deployment. The twinned design shares identical kinematics, contact surfaces, visual appearance, and sensors across collection and deployment — keeping observations and actions aligned end to end. Trained from scratch on only a few hundred wearable demonstrations - with zero on-robot training or intervention data - TwinDEX completed a standardized chemistry experiment involving tool switches, fine force control, and bimanual coordination. Robot-free data showed comparable learning efficiency on the multi-task evaluation, TwinDEX delivered 5.3 times effective throughput than on-robot teleoperation. TwinDEX demonstrates that high-quality robot-free data can fully substitute for on-robot teleoperation data on challenging dexterous tasks — removing the dependency on real-robot hardware that has been the central bottleneck to scaling dexterous manipulation data. This was the proof-it phase. Now comes scale: what emerges at tens of thousands, or millions, of episodes? Watch the demo and read the technical blog: #TwinDEX #Robotics #EmbodiedAI #DexterousManipulation

What if a dexterous robot could learn a 20+ step chemistry experiment without a single on-robot training demo? Meet TwinDEX: a pair of co-designed, three-finger, nine-DoF dexterous manipulation interface: one wearable for data collection, one for robot deployment. The twinned design shares identical kinematics, contact surfaces, visual appearance, and sensors across collection and deployment — keeping observations and actions aligned end to end. Trained from scratch on only a few hundred wearable demonstrations - with zero on-robot training or intervention data - TwinDEX completed a standardized chemistry experiment involving tool switches, fine force control, and bimanual coordination. Robot-free data showed comparable learning efficiency on the multi-task evaluation, TwinDEX delivered 5.3 times effective throughput than on-robot teleoperation. TwinDEX demonstrates that high-quality robot-free data can fully substitute for on-robot teleoperation data on challenging dexterous tasks — removing the dependency on real-robot hardware that has been the central bottleneck to scaling dexterous manipulation data. This was the proof-it phase. Now comes scale: what emerges at tens of thousands, or millions, of episodes? Watch the demo and read the technical blog: #TwinDEX #Robotics #EmbodiedAI #DexterousManipulation

X Square Robot

1,346,470 次观看 • 1 个月前

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

AI at Meta

453,860 次观看 • 2 年前

THIS GUY BUILT REAL FLYING SWORD DRONES AND CONTROLS THEM WITH HAND GESTURES fan shisan is a physics student in chengdu who coded and built flying drones shaped like actual swords he controls them entirely with hand gestures. no controller or a phone. just his hands wave your hand and the sword follows. point and it flies where you direct it. multiple swords in the air at the same time responding to different gestures he built the gesture recognition system HIMSELF and coded the flight control algorithms. designed the drone hardware in the shape of swords. the gesture tracking uses sensors that read the position and movement of his hands in real time and translate them into flight commands for each drone independently he even built one large enough to ride on a physics student with coding skills just brought something straight out of a kung fu movie into the real world the line between science fiction and side projects is getting thinner every month

THIS GUY BUILT REAL FLYING SWORD DRONES AND CONTROLS THEM WITH HAND GESTURES fan shisan is a physics student in chengdu who coded and built flying drones shaped like actual swords he controls them entirely with hand gestures. no controller or a phone. just his hands wave your hand and the sword follows. point and it flies where you direct it. multiple swords in the air at the same time responding to different gestures he built the gesture recognition system HIMSELF and coded the flight control algorithms. designed the drone hardware in the shape of swords. the gesture tracking uses sensors that read the position and movement of his hands in real time and translate them into flight commands for each drone independently he even built one large enough to ride on a physics student with coding skills just brought something straight out of a kung fu movie into the real world the line between science fiction and side projects is getting thinner every month

Om Patel

308,000 次观看 • 5 个月前

Training robot hands at scale! 🪬 Lightwheel released EgoSuite-Open100K, a 100,000-hour open dataset of egocentric video with hand and body pose annotations. Everything is available on Hugging Face. Multiple variants for different needs. → Head-view only for hand-centric tasks. → Wrist-view for close-range contact and grasping detail. → Full-body pose variants for whole-body context. → Raw video variants without annotations for lightweight inspection. 15,000+ distinct tasks across 15,000+ different scenes. Egocentric data is gold for imitation learning. Show a robot how humans actually manipulate things from the perspective humans have, from their eyes, from the wrist. Hand pose annotations transfer directly to robot hands. Wrist-camera view shows contact dynamics most robots need to learn. Multiple data formats, LeRobot v3 and MCAP, so researchers can use whatever their pipeline expects. Semantic annotations at the subtask level, not just raw video. Congrats Steve Xie, and team! 🦾 Find it on HF: ~~ ♻ Join the weekly robotics newsletter, and never miss any news →

Training robot hands at scale! 🪬 Lightwheel released EgoSuite-Open100K, a 100,000-hour open dataset of egocentric video with hand and body pose annotations. Everything is available on Hugging Face. Multiple variants for different needs. → Head-view only for hand-centric tasks. → Wrist-view for close-range contact and grasping detail. → Full-body pose variants for whole-body context. → Raw video variants without annotations for lightweight inspection. 15,000+ distinct tasks across 15,000+ different scenes. Egocentric data is gold for imitation learning. Show a robot how humans actually manipulate things from the perspective humans have, from their eyes, from the wrist. Hand pose annotations transfer directly to robot hands. Wrist-camera view shows contact dynamics most robots need to learn. Multiple data formats, LeRobot v3 and MCAP, so researchers can use whatever their pipeline expects. Semantic annotations at the subtask level, not just raw video. Congrats Steve Xie, and team! 🦾 Find it on HF: ~~ ♻ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

13,421 次观看 • 2 个月前

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually slow them down so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

Jim Fan

567,809 次观看 • 1 年前

Seeing touch without pressing down! 🫆 Most tactile sensors rely on deformation. Press down, deform the sensor, measure the deformation. But light contact, touching water, cream, soft films, doesn't deform anything. The sensor sees nothing. LightTact flips the approach. It uses optical detection instead. An ambient-blocking configuration suppresses external light and internal illumination everywhere except at actual contact points. Only scattered light from true contact gets through. The results? High-contrast raw images. Non-contact pixels stay near-black. Contact pixels show the natural appearance of whatever's touching. This works across material properties, contact forces, surface colors, and lighting conditions. Water, facial cream, thin films, liquids, all become visible the moment they touch. The sensor outputs spatially aligned visual-tactile images. Vision-language models can read them directly. Here's the paper: Changyi Lin and team - hats off ;) ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Seeing touch without pressing down! 🫆 Most tactile sensors rely on deformation. Press down, deform the sensor, measure the deformation. But light contact, touching water, cream, soft films, doesn't deform anything. The sensor sees nothing. LightTact flips the approach. It uses optical detection instead. An ambient-blocking configuration suppresses external light and internal illumination everywhere except at actual contact points. Only scattered light from true contact gets through. The results? High-contrast raw images. Non-contact pixels stay near-black. Contact pixels show the natural appearance of whatever's touching. This works across material properties, contact forces, surface colors, and lighting conditions. Water, facial cream, thin films, liquids, all become visible the moment they touch. The sensor outputs spatially aligned visual-tactile images. Vision-language models can read them directly. Here's the paper: Changyi Lin and team - hats off ;) ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

207,951 次观看 • 3 个月前

Force-sensing fingers! 🧤 Stanford researchers just released UMI-FT, a handheld data collection platform that puts compact six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose data. Many manipulation tasks require careful force modulation: too little force and the task fails, too much and you cause damage. But commercial force/torque sensors are expensive, bulky, and fragile, which has limited large-scale force-aware policy learning. UMI-FT changes the economics. The platform uses an iPhone for RGB vision, ultrawide RGB, depth, and pose via ARKit, with each finger sensorized using a CoinFT sensor to capture per-finger wrench information during manipulation. This multimodal data trains an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. The learned policy runs slowest and generates reference targets, while model-based compliance and force controllers provide delicate 6D compliance control and real-time force modulation. They tested on three contact-rich, force-sensitive tasks: whiteboard wiping (locate eraser, grasp, wipe until clean), skewering zucchini (grasp slice firmly, push onto stick until punctured), and lightbulb insertion (grasp bulb, align bayonet pin with socket slit, insert while overcoming spring force, rotate to light up). The results are clear. Policies without compliance struggle to modulate contact force and trigger safety faults from excessive force. Policies without force sensing fail to grasp unseen objects or resist reaction forces, causing slippage. Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Force-sensing fingers! 🧤 Stanford researchers just released UMI-FT, a handheld data collection platform that puts compact six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose data. Many manipulation tasks require careful force modulation: too little force and the task fails, too much and you cause damage. But commercial force/torque sensors are expensive, bulky, and fragile, which has limited large-scale force-aware policy learning. UMI-FT changes the economics. The platform uses an iPhone for RGB vision, ultrawide RGB, depth, and pose via ARKit, with each finger sensorized using a CoinFT sensor to capture per-finger wrench information during manipulation. This multimodal data trains an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. The learned policy runs slowest and generates reference targets, while model-based compliance and force controllers provide delicate 6D compliance control and real-time force modulation. They tested on three contact-rich, force-sensitive tasks: whiteboard wiping (locate eraser, grasp, wipe until clean), skewering zucchini (grasp slice firmly, push onto stick until punctured), and lightbulb insertion (grasp bulb, align bayonet pin with socket slit, insert while overcoming spring force, rotate to light up). The results are clear. Policies without compliance struggle to modulate contact force and trigger safety faults from excessive force. Policies without force sensing fail to grasp unseen objects or resist reaction forces, causing slippage. Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

12,868 次观看 • 9 个月前

A valve spring inside a running engine does more than compress and release. The spring also rotates slowly along its axis with each cycle, and that rotation is intentional. The valve face strikes the same metal seat thousands of times a minute, and without rotation any single contact point would wear down unevenly. By spreading that contact around the full circumference, the rotation evens out heat and mechanical wear so the valve seals cleanly for much longer. White marks painted on the coils make the rotation visible on camera. Without them, the movement would be impossible to see.

A valve spring inside a running engine does more than compress and release. The spring also rotates slowly along its axis with each cycle, and that rotation is intentional. The valve face strikes the same metal seat thousands of times a minute, and without rotation any single contact point would wear down unevenly. By spreading that contact around the full circumference, the rotation evens out heat and mechanical wear so the valve seals cleanly for much longer. White marks painted on the coils make the rotation visible on camera. Without them, the movement would be impossible to see.

Mechanical Knowledge

38,421 次观看 • 3 个月前