正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check... show more

Jiafei Duan

7,057 subscribers

74,089 次观看 • 7 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

11 条评论

Jiafei Duan 的头像

Jiafei Duan7 个月前

1/🧵 Think of how we control LLMs. We don’t retrain GPT-4 every time we want a specific tone; we use prompting or guided decoding (such as in Diffusion-LM). VLS brings this same "Inference-Time Control" to physical hardware.

Jiafei Duan 的头像

Jiafei Duan7 个月前

2/🧵 We bridge the Semantic-Action Gap by introducing a new intermediate interface: Programmatic Rewards. Instead of mapping pixels directly to actions—which is brittle—we use a VLM to translate high-level intent into a differentiable "reward landscape." This reward function acts as the "steering wheel" for the robot's existing motor primitives.

Jiafei Duan 的头像

Jiafei Duan7 个月前

3/🧵 To handle complex real-world environments, we combine grounding, diversity, guidance, and selection in a single inference-time loop: 1. OOD Grounding Ground observation–language inputs into a compact geometric scaffold (task-relevant 3D keypoints). This exposes what matters spatially before any action refinement. 2. RBF Repulsion Force Maintains particle diversity during early sampling, preventing premature collapse to uniform (and brittle) action proposals. 3. VLM-Guided Gradient Refinement Stage-aware, differentiable rewards synthesized by a VLM provide gradients that continuously steer action particles toward task-consistent behaviors. 4. Feynman–Kac Resampling A gradient-free population evolution step that replicates high-reward particles and prunes poor ones—escaping local optima without retraining. 5. Closed-Loop Phase Switching A Schmitt trigger monitors execution feedback to automatically decide when to transition to the next stage—no hand-crafted schedules.

Jiafei Duan 的头像

Jiafei Duan7 个月前

4/🧵 How good is VLS at controlling or “steering” pretrained robot policies? We benchmark VLS against prior inference-time steering methods on CALVIN: -DynaGuide: steering via latent DINO feature spaces -ITPS: human-in-the-loop feedback Results: VLS delivers a +31% absolute improvement in success rate: Movable Objects: 94% success (7.4× over the base policy) Articulated Parts: 87% success (9.6× over the base policy) VLS enables precise, spatially grounded control—pushing test-time adaptation well beyond prior steering methods.

Jiafei Duan 的头像

Jiafei Duan7 个月前

5/🧵 In extreme OOD settings, steering is the real pivot. On LIBERO-PRO, which stress-tests robots with severe task & pose shifts, VLS boosts a frozen Pi-0.5 by up to +13% success—outperforming OpenVLA, Pi-0, and Pi-0.5 without changing a single weight. Example: “Put the cream cheese into the box.” Base Pi-0.5 replays memorized trajectories. With VLS, the same policy is steered into a new action distribution—and succeeds.

Jiafei Duan 的头像

Jiafei Duan7 个月前

6/🧵 In the real world, VLS lets a Franka robot survive radical distribution shifts: • Appearance: red plate → unseen yellow • Position: targets swapped mid-execution • Object: banana → novel mug The base policy collapses on novel objects. VLS stays stable by grounding new task geometry in real time. Example OOD swap: ask the robot to put the orange into a new green plate. Left: Pi-0.5 memorizes old trajectories. Right: VLS + Pi-0.5 adapts—and succeeds.

Jiafei Duan 的头像

Jiafei Duan7 个月前

7/🧵 We broke VLS down to see what matters most: Gradient Guidance: Essential. Without it, success rates collapse. Feynman–Kac (FK) Resampling: Crucial for efficiency. It prunes "bad" action trajectories early. RBF Diversity: Prevents the policy from getting stuck in a single (possibly wrong) mode.

Jiafei Duan 的头像

Jiafei Duan7 个月前

8/🧵 The "OOD Dilemma" is the biggest hurdle to home robotics. VLS proves that we don't need more data for every edge case, we just need better ways to "control" these models that we already have.

Jiafei Duan 的头像

Jiafei Duan7 个月前

9/🧵 Read the full paper here: Check out the project page for videos of VLS in action: Code will be release here soon:

Jiafei Duan 的头像

Jiafei Duan7 个月前

This is a great project led by @liu_shuo42927 and supported by @ishneet0710, @YiqingXu6 and @RanjayKrishna ! * @liu_shuo42927 is graduating with a Master’s and brings strong robotics + research experience. He’s currently on the market for Research Engineer (RE) or Research Assistant (RA) roles, reach out if you’re hiring!

Lukas Die Kunst 的头像

Lukas Die Kunst7 个月前

VLS's gradient-based refinement solves out-of-distribution challenges brilliantly. Exactly the kind of training-free adaptation robotics needs!

相关视频

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Lerrel Pinto

56,591 次观看 • 2 年前

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

Kostas Daniilidis

54,578 次观看 • 6 个月前

Today, we publicly released RoboCasa365, a large-scale simulation benchmark for training and systematically evaluating generalist robot models. Built upon our original RoboCasa framework, it offers: • 2,500 realistic kitchen environments; • 365 everyday tasks (basic skills + long-horizon mobile manipulation); • Over 3,200 objects with many articulated fixtures/appliances. All are designed for fully controlled, reproducible benchmarking of robotic policies. Progress in robotic foundation models is real. But it’s still hard to answer basic questions like: How close are we to general-purpose autonomy? What factors drive generalization? What are the model/data scaling curves like? Real-world eval is slow and noisy, and existing sims (like LIBERO, which we built 3 years ago) often lack sufficient task and scene diversity. This benchmark comes with 2,200+ hours of demonstrations and 500K+ trajectories to support studies of multi-task training, pretraining, and continual learning at scale. Check it out at

Today, we publicly released RoboCasa365, a large-scale simulation benchmark for training and systematically evaluating generalist robot models. Built upon our original RoboCasa framework, it offers: • 2,500 realistic kitchen environments; • 365 everyday tasks (basic skills + long-horizon mobile manipulation); • Over 3,200 objects with many articulated fixtures/appliances. All are designed for fully controlled, reproducible benchmarking of robotic policies. Progress in robotic foundation models is real. But it’s still hard to answer basic questions like: How close are we to general-purpose autonomy? What factors drive generalization? What are the model/data scaling curves like? Real-world eval is slow and noisy, and existing sims (like LIBERO, which we built 3 years ago) often lack sufficient task and scene diversity. This benchmark comes with 2,200+ hours of demonstrations and 500K+ trajectories to support studies of multi-task training, pretraining, and continual learning at scale. Check it out at

Yuke Zhu

24,390 次观看 • 6 个月前

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 次观看 • 1 年前

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

20,434 次观看 • 9 个月前

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 次观看 • 1 年前

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 次观看 • 1 年前

Can vision-language-action (VLA) models generalize to diverse OOD tasks and align with customized objectives? 🤔 🚀 We introduce GRAPE, a plug-and-play algorithm to generalize robot policies via preference alignment. GRAPE unfolds three benefits to boost the generalizability of VLAs: 👉1. GRAPE aligns VLAs on a trajectory level and endows the model with the ability for global decision-making, instead of merely cloning behavior; 👉2. GRAPE implicitly models reward from both successful and failed trials to boost generalizability to diverse tasks; 👉3. GRAPE adopts a scalable preference synthesis algorithm to rank trajectories with preferences that align with arbitrary objectives. Our experiments on a diverse array of real-world and simulated robotic tasks reveal: 1⃣GRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%; 2⃣GRAPE is versatile to be aligned with diverse objectives and reduce collision rates by 44.31% or rollout length by 11.15% when aligning towards safer or more efficient manipulation policy, respectively. Check out our full project for more details: 🔥 Paper: 🔥 Project: 🔥 Code:

Can vision-language-action (VLA) models generalize to diverse OOD tasks and align with customized objectives? 🤔 🚀 We introduce GRAPE, a plug-and-play algorithm to generalize robot policies via preference alignment. GRAPE unfolds three benefits to boost the generalizability of VLAs: 👉1. GRAPE aligns VLAs on a trajectory level and endows the model with the ability for global decision-making, instead of merely cloning behavior; 👉2. GRAPE implicitly models reward from both successful and failed trials to boost generalizability to diverse tasks; 👉3. GRAPE adopts a scalable preference synthesis algorithm to rank trajectories with preferences that align with arbitrary objectives. Our experiments on a diverse array of real-world and simulated robotic tasks reveal: 1⃣GRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%; 2⃣GRAPE is versatile to be aligned with diverse objectives and reduce collision rates by 44.31% or rollout length by 11.15% when aligning towards safer or more efficient manipulation policy, respectively. Check out our full project for more details: 🔥 Paper: 🔥 Project: 🔥 Code:

Huaxiu Yao

20,071 次观看 • 1 年前

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Abhishek Gupta

35,058 次观看 • 2 个月前

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 次观看 • 7 个月前

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.

Praveen Kumar Verma

136,567 次观看 • 22 天前

Watching Generalist AI demonstrate real-time intuition is incredible! 🤖✨ Check out this "Physical Commonsense" in action: the robot flips a washer, picks it up and then double-nudges it perfectly into a tight foam slot. This isn't a scripted motion; it’s an emergent behavior from a model trained on 270,000+ hours of real-world interaction. We are seeing the era where robots are starting to intuit and react to physics.

Watching Generalist AI demonstrate real-time intuition is incredible! 🤖✨ Check out this "Physical Commonsense" in action: the robot flips a washer, picks it up and then double-nudges it perfectly into a tight foam slot. This isn't a scripted motion; it’s an emergent behavior from a model trained on 270,000+ hours of real-world interaction. We are seeing the era where robots are starting to intuit and react to physics.

Humanoids daily

14,762 次观看 • 7 个月前

A humanoid robot just unloaded a dishwasher for 67 hours straight. By end of 2026, the same robot will perform surgery. That's not a prediction from a sci-fi writer. That's Brett Adcock. CEO of Figure AI. Said it on camera this month. Here's what makes this insane. Two years ago Figure's robot put a Keurig cup in a coffee maker. That was the breakthrough moment. The whole team celebrated. A coffee cup. Today that same robot runs 67 consecutive hours in a BMW factory. Grabbing packages. Finding barcodes. Placing them. One operation every second. Just a neural network that learned by watching. And Brett just said the hands that do that the same fingers, the same sensors, the same neural net are already capable of surgical precision from a hardware standpoint. "From a hardware perspective in 2026 we'll be able to do what surgeons can do." The AI just has to catch up to the hands. Two years ago: a coffee cup. This year: a factory. Next: the operating table. The robot didn't change, the learning did and it's not stopping.

A humanoid robot just unloaded a dishwasher for 67 hours straight. By end of 2026, the same robot will perform surgery. That's not a prediction from a sci-fi writer. That's Brett Adcock. CEO of Figure AI. Said it on camera this month. Here's what makes this insane. Two years ago Figure's robot put a Keurig cup in a coffee maker. That was the breakthrough moment. The whole team celebrated. A coffee cup. Today that same robot runs 67 consecutive hours in a BMW factory. Grabbing packages. Finding barcodes. Placing them. One operation every second. Just a neural network that learned by watching. And Brett just said the hands that do that the same fingers, the same sensors, the same neural net are already capable of surgical precision from a hardware standpoint. "From a hardware perspective in 2026 we'll be able to do what surgeons can do." The AI just has to catch up to the hands. Two years ago: a coffee cup. This year: a factory. Next: the operating table. The robot didn't change, the learning did and it's not stopping.

Shruti

47,597 次观看 • 4 个月前

🤖 Ever dreamed of controlling a humanoid robot to perform complex, long-horizon tasks — using just a single Vision Pro? 🎉 Meet CLONE: a holistic, closed-loop, whole-body teleoperation system for long-horizon humanoid control! 🏃‍♂️🧍 CLONE enables rich and coordinated interactive tasks: 🥊 boxing 🏓 table tennis 🤲 object pickup 📦 room arrangement 🤝 handover … and more! 🌀 Our closed-loop error correction powered by LiDAR odometry ensures precision, while motion-captured demonstrations supercharge policy learning — unlocking the full potential of the G1 robot. 🎥 It’s hard to squeeze the magic into 1 minute — check out the full video demo and project page here: 🔗

🤖 Ever dreamed of controlling a humanoid robot to perform complex, long-horizon tasks — using just a single Vision Pro? 🎉 Meet CLONE: a holistic, closed-loop, whole-body teleoperation system for long-horizon humanoid control! 🏃‍♂️🧍 CLONE enables rich and coordinated interactive tasks: 🥊 boxing 🏓 table tennis 🤲 object pickup 📦 room arrangement 🤝 handover … and more! 🌀 Our closed-loop error correction powered by LiDAR odometry ensures precision, while motion-captured demonstrations supercharge policy learning — unlocking the full potential of the G1 robot. 🎥 It’s hard to squeeze the magic into 1 minute — check out the full video demo and project page here: 🔗

Siyuan Huang

66,986 次观看 • 1 年前

Tiff In Tech (600K+ on YouTube) just dropped a new video: Robots Are ‘Doomed to Succeed’ and That’s a Problem. Excited to see OpenMind featured in the conversation, highlighting the missing gaps in physical AI we solve: - Robot understanding of human intent - Building real social intelligence in robots - Operating systems that work across all robot manufacturers and form-factors Check out the full video below👇

Tiff In Tech (600K+ on YouTube) just dropped a new video: Robots Are ‘Doomed to Succeed’ and That’s a Problem. Excited to see OpenMind featured in the conversation, highlighting the missing gaps in physical AI we solve: - Robot understanding of human intent - Building real social intelligence in robots - Operating systems that work across all robot manufacturers and form-factors Check out the full video below👇

OpenMind

11,197 次观看 • 5 个月前

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Scaling vision-language-action (VLA) models to high-DoF dexterous hands has long been a "holy grail" challenge due to the high-dimensional action space and data scarcity. As a wrap up of the year 2025, we are releasing GR-Dexter, a holistic hardware-model-data framework for generalist manipulation on a bimanual dexterous-hand robot. This is the first VLA system to achieve: ✅ High-DoF Control: Managing a 56-DoF bimanual system (21-DoF per hand). ✅ Long-Horizon Tasks with tool use: Vacuuming, bread serving with tongs, and table decluttering. ✅ Open-World Generalization: Robust performance with unseen objects and abstract instructions. Project page: ArXiv:

Xiao Ma

94,108 次观看 • 8 个月前

$We’re excited to launch Instance – a success detector for robot rollouts! Describe your task and drop in a dataset, and it judges success/fail on each rollout with detailed subtask captions– it’s more accurate than Claude Opus 4.8, at a fraction of the latency. This is just the first step toward our larger goal: an autonomous evaluation rig for robot learning. Today, evaluating a robot policy means humans have to watch the robot roll out, mark success, and reset the scene– we're automating that, starting with the success judge.$

We’re excited to launch Instance – a success detector for robot rollouts! Describe your task and drop in a dataset, and it judges success/fail on each rollout with detailed subtask captions– it’s more accurate than Claude Opus 4.8, at a fraction of the latency. This is just the first step toward our larger goal: an autonomous evaluation rig for robot learning. Today, evaluating a robot policy means humans have to watch the robot roll out, mark success, and reset the scene– we're automating that, starting with the success judge.

Lucy Cai

73,929 次观看 • 2 个月前

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Ilir Aliu

18,210 次观看 • 7 个月前

“Proof of reserves” today is like a bar sign that says free beer tomorrow - the promise is always just out of reach. In this clip, Jiritsu founder Jacob Guedalia breaks down the real problem with tokenization: A token isn’t an asset if it’s not bound to custody.

“Proof of reserves” today is like a bar sign that says free beer tomorrow - the promise is always just out of reach. In this clip, Jiritsu founder Jacob Guedalia breaks down the real problem with tokenization: A token isn’t an asset if it’s not bound to custody.

Jiritsu Network

11,425 次观看 • 1 年前

Can GPT-4 teach a robot hand to do pen spinning tricks better than you do? I'm excited to announce Eureka, an open-ended agent that designs reward functions for robot dexterity at super-human level. It’s like Voyager in the space of a physics simulator API! Eureka bridges the gap between high-level reasoning (coding) and low-level motor control. It is a “hybrid-gradient architecture”: a black box, inference-only LLM instructs a white box, learnable neural network. The outer loop runs GPT-4 to refine the reward function (gradient-free), while the inner loop runs reinforcement learning to train a robot controller (gradient-based). We are able to scale up Eureka thanks to IsaacGym, a GPU-accelerated physics simulator that speeds up reality by 1000x. On a benchmark suite of 29 tasks across 10 robots, Eureka rewards outperform expert human-written ones on 83% of the tasks by 52% improvement margin on average. We are surprised that Eureka is able to learn pen spinning tricks, which are very difficult even for CGI artists to animate frame by frame! Eureka also enables a new form of in-context RLHF, which is able to incorporate a human operator’s feedback in natural language to steer and align the reward functions. It can serve as a powerful co-pilot for robot engineers to design sophisticated motor behaviors. As usual, we open-source everything! Welcome you all to check out our video gallery and try the codebase today: Paper: Code: Deep dive with me: 🧵

Can GPT-4 teach a robot hand to do pen spinning tricks better than you do? I'm excited to announce Eureka, an open-ended agent that designs reward functions for robot dexterity at super-human level. It’s like Voyager in the space of a physics simulator API! Eureka bridges the gap between high-level reasoning (coding) and low-level motor control. It is a “hybrid-gradient architecture”: a black box, inference-only LLM instructs a white box, learnable neural network. The outer loop runs GPT-4 to refine the reward function (gradient-free), while the inner loop runs reinforcement learning to train a robot controller (gradient-based). We are able to scale up Eureka thanks to IsaacGym, a GPU-accelerated physics simulator that speeds up reality by 1000x. On a benchmark suite of 29 tasks across 10 robots, Eureka rewards outperform expert human-written ones on 83% of the tasks by 52% improvement margin on average. We are surprised that Eureka is able to learn pen spinning tricks, which are very difficult even for CGI artists to animate frame by frame! Eureka also enables a new form of in-context RLHF, which is able to incorporate a human operator’s feedback in natural language to steer and align the reward functions. It can serve as a powerful co-pilot for robot engineers to design sophisticated motor behaviors. As usual, we open-source everything! Welcome you all to check out our video gallery and try the codebase today: Paper: Code: Deep dive with me: 🧵

Jim Fan

2,675,676 次观看 • 2 年前