Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🚧 Progress update: reBot Arm is taking shape.👉 🎯#OpenSource 6+1 DoF force-controlled robot arm ; 🎯Native support for high-end torque #motors - #DAMIAO 4310, RobStride Dynamics 06 QDD; 🎯Deliver smooth joint-level force/position control, open low-level protocols, and #ROS2/#Python compatibility; 🎯Built for bridging #simulation and real-world AI manipulation.

seeedstudio's profile picture

Seeed Studio

78,371 subscribers

14,991 Aufrufe • vor 7 Monaten •via X (Twitter)

#OpenSource#motors#DAMIAO
LIVE

Anya Rossi• Live Now

Private livecam show

8 Kommentare

Profilbild von Seeed Studio
Seeed Studiovor 7 Monaten

DAMIAO 4310 actuator: RobStride 06 QDD: Actuator getting started wiki:

Profilbild von Felix Manufaktur
Felix Manufakturvor 7 Monaten

@RobStride_com The reBot Arm's force control could transform additive manufacturing workflows. Particularly impressed by the DAMIAO 4310 integration - this level of precision opens new possibilities!

Profilbild von LightShift Studio
LightShift Studiovor 7 Monaten

@RobStride_com Hey there!

Profilbild von Shadow Defense
Shadow Defensevor 7 Monaten

@RobStride_com how much is each motor?

Profilbild von Asier Arranz
Asier Arranzvor 7 Monaten

@RobStride_com This looks amazing! Congrats!!!

Profilbild von Damian @ The Lighthouse 🤖💡🏠
Damian @ The Lighthouse 🤖💡🏠vor 7 Monaten

@RobStride_com @i2cjak

Profilbild von Xx_Chuuya_xX
Xx_Chuuya_xXvor 7 Monaten

@RobStride_com how is the tested torque on that thing? robstrides specifically i mean. Does it live up to its numbers?

Profilbild von Damian @ The Lighthouse 🤖💡🏠
Damian @ The Lighthouse 🤖💡🏠vor 7 Monaten

@RobStride_com Update?

Ähnliche Videos

🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous control—combining active vision, high-res touch, and high-DoF robot hands SharpaWave — to complete complex, real-world tasks. Code is fully released; check out our: Homepage: Paper link: Github:
3:55

🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous control—combining active vision, high-res touch, and high-DoF robot hands SharpaWave — to complete complex, real-world tasks. Code is fully released; check out our: Homepage: Paper link: Github:

Haoran Geng

96,272 Aufrufe • vor 1 Jahr

🔥 #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora — the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. 🦾 Dual Arms 🖐️ Dual Hands 🎯 36 DoF Control 🌍 Open Source Trained on: • 100K simulated trajectories • 10K real-world demonstrations Dexora achieves: ✓ 90%+ success on basic manipulation ✓ Strong dexterous manipulation performance ✓ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. 🎥 Demos: 📄 Paper:
2:58

🔥 #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora — the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. 🦾 Dual Arms 🖐️ Dual Hands 🎯 36 DoF Control 🌍 Open Source Trained on: • 100K simulated trajectories • 10K real-world demonstrations Dexora achieves: ✓ 90%+ success on basic manipulation ✓ Strong dexterous manipulation performance ✓ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. 🎥 Demos: 📄 Paper:

Hao Zhao

17,474 Aufrufe • vor 4 Monaten

Open-source dexterous hands with fingertip sensors! 🪬 ORCA Dexterity just released three dexterous hand models. Their mission: is to democratize robotic hand dexterity. They're sharing progress on exceptional, low-cost hardware and a software layer from low-level control to robotic hand learning. This is the open-source approach to dexterous manipulation. It has 83 taxels per finger with 0.1 N force detection which pretty is impressive for an open-source design. Tactile sensing is critical for dexterous manipulation, knowing contact forces enables gentle grasping, slip detection, and force-controlled assembly. Also 700g weight for the lite version makes it practical for mounting on robot arms without exceeding payload limits. Lower weight means faster movements and lower torque requirements. Open hardware accelerates robotics by letting researchers and builders modify designs for their specific needs without starting from scratch! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:20

Open-source dexterous hands with fingertip sensors! 🪬 ORCA Dexterity just released three dexterous hand models. Their mission: is to democratize robotic hand dexterity. They're sharing progress on exceptional, low-cost hardware and a software layer from low-level control to robotic hand learning. This is the open-source approach to dexterous manipulation. It has 83 taxels per finger with 0.1 N force detection which pretty is impressive for an open-source design. Tactile sensing is critical for dexterous manipulation, knowing contact forces enables gentle grasping, slip detection, and force-controlled assembly. Also 700g weight for the lite version makes it practical for mounting on robot arms without exceeding payload limits. Lower weight means faster movements and lower torque requirements. Open hardware accelerates robotics by letting researchers and builders modify designs for their specific needs without starting from scratch! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

38,323 Aufrufe • vor 7 Monaten

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:26

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

40,583 Aufrufe • vor 6 Monaten

Humanoid robots are cool. But most of them aren’t ready for real-world work. X-Humanoid just launched Embodied Tien Kung 3.0 - a full-size humanoid robot built to actually operate in industrial and commercial environments. What makes it different? • Strong, high-torque joints for heavy tasks • Can move dynamically and stay stable • Handles complex motions (even over obstacles) • Millimeter-level precision for detailed work It runs on the Wise KaiWu AI platform, which allows the robot to see, think, and act in a continuous loop with less human control and support for multi-robot collaboration. It also supports ROS2, MQTT, and TCP/IP, and offers more open hardware + software interfaces to make development easier for teams. The goal is simple: make humanoid robots practical, easier to build on, and ready for real-world use. Visit: Check out the official launch video below to see what it can do! 👇 #Robotics #HumanoidRobot #AI
2:18

Humanoid robots are cool. But most of them aren’t ready for real-world work. X-Humanoid just launched Embodied Tien Kung 3.0 - a full-size humanoid robot built to actually operate in industrial and commercial environments. What makes it different? • Strong, high-torque joints for heavy tasks • Can move dynamically and stay stable • Handles complex motions (even over obstacles) • Millimeter-level precision for detailed work It runs on the Wise KaiWu AI platform, which allows the robot to see, think, and act in a continuous loop with less human control and support for multi-robot collaboration. It also supports ROS2, MQTT, and TCP/IP, and offers more open hardware + software interfaces to make development easier for teams. The goal is simple: make humanoid robots practical, easier to build on, and ready for real-world use. Visit: Check out the official launch video below to see what it can do! 👇 #Robotics #HumanoidRobot #AI

Chidanand Tripathi

69,604 Aufrufe • vor 8 Monaten

X-Humanoid just officially dropped Embodied Tien Kung 3.0, A universal platform designed to be way more open and developer-friendly. 🤖 Built on their Wise Kaiwu AI platform, this next-gen humanoid is all about slashing development costs. It’s a fully interoperable ecosystem that supports everything from tactile interaction to high-dynamic motion control at a full humanoid scale. ➤ Radical Openness: X-Humanoid is open-sourcing the full stack—robot body, motion control, VLM/VLA models, and the RoboMIND dataset. It fully supports ROS2, MQTT, and TCP/IP, so developers can customize use cases without re-engineering the basics. ➤ High-Performance Hardware: With high-torque integrated joints, Tien Kung 3.0 can clear 1-meter (3.3ft) obstacles and handle dexterous moves like kneeling and bending. It hits millimeter-level precision, making it a solid fit for industrial-grade tasks. ➤ True Autonomy: The bot runs a continuous perception-decision-execution loop. It uses world models to break down complex language commands and VLA models for real-time obstacle avoidance and navigation. ➤ Scalable Collaboration: The platform moves beyond single-unit tasks to support multi-robot collaboration with autonomous scheduling. It’s built to move embodied AI from the lab straight into real-world commercial and industrial environments. Source: X-Humanoid #Humanoid #OpenSource #Robotics #EmbodiedAI #PhysicalAI #Automation #XHumanoid #TienKung #WiseKaiwu
2:18

X-Humanoid just officially dropped Embodied Tien Kung 3.0, A universal platform designed to be way more open and developer-friendly. 🤖 Built on their Wise Kaiwu AI platform, this next-gen humanoid is all about slashing development costs. It’s a fully interoperable ecosystem that supports everything from tactile interaction to high-dynamic motion control at a full humanoid scale. ➤ Radical Openness: X-Humanoid is open-sourcing the full stack—robot body, motion control, VLM/VLA models, and the RoboMIND dataset. It fully supports ROS2, MQTT, and TCP/IP, so developers can customize use cases without re-engineering the basics. ➤ High-Performance Hardware: With high-torque integrated joints, Tien Kung 3.0 can clear 1-meter (3.3ft) obstacles and handle dexterous moves like kneeling and bending. It hits millimeter-level precision, making it a solid fit for industrial-grade tasks. ➤ True Autonomy: The bot runs a continuous perception-decision-execution loop. It uses world models to break down complex language commands and VLA models for real-time obstacle avoidance and navigation. ➤ Scalable Collaboration: The platform moves beyond single-unit tasks to support multi-robot collaboration with autonomous scheduling. It’s built to move embodied AI from the lab straight into real-world commercial and industrial environments. Source: X-Humanoid #Humanoid #OpenSource #Robotics #EmbodiedAI #PhysicalAI #Automation #XHumanoid #TienKung #WiseKaiwu

RoboHub🤖

49,440 Aufrufe • vor 8 Monaten

A 25-DoF dual-arm wheeled humanoid for under $3K. 🤖 VLAI Robotics just launched K1 in China, continuing its low-cost strategy with a starting price of just ¥19,800 (~$2.9K). The hardware: • Two 6 kg-payload arms • Movable waist and head • 4-wheel omnidirectional base • ±0.02 mm repeatability • <10 ms control latency Grippers and dexterous hands are optional. The developer stack is unusually open for the price: LeRobot, ROS 2, MoveIt 2, Isaac Sim/Lab and MuJoCo, with support for teleop, VLA and world-model workflows. VLAI says cumulative shipments are already approaching 1,000 units. At this price point, mobile manipulation hardware is getting cheap enough that the real bottleneck may shift from buying the robot to making it actually useful.
2:33

A 25-DoF dual-arm wheeled humanoid for under $3K. 🤖 VLAI Robotics just launched K1 in China, continuing its low-cost strategy with a starting price of just ¥19,800 (~$2.9K). The hardware: • Two 6 kg-payload arms • Movable waist and head • 4-wheel omnidirectional base • ±0.02 mm repeatability • <10 ms control latency Grippers and dexterous hands are optional. The developer stack is unusually open for the price: LeRobot, ROS 2, MoveIt 2, Isaac Sim/Lab and MuJoCo, with support for teleop, VLA and world-model workflows. VLAI says cumulative shipments are already approaching 1,000 units. At this price point, mobile manipulation hardware is getting cheap enough that the real bottleneck may shift from buying the robot to making it actually useful.

RoboHub🤖

24,848 Aufrufe • vor 2 Monaten

Today I assembled the first batch of reBot DevArm in the U.S! First one on the planet! 🇺🇸⚙️ But this isn’t about a robot arm. This is about what happens when open source hits robotics. For years: •Hardware was locked •Robotics was expensive •Embodied AI was out of reach Now? Anyone can get their hands on: •6 DOF manipulation •Real precision (<0.2 mm repeatability) •Full stack access (ROS, Isaac Sim, LeRobot) This is how movements start. Not in labs. Not behind paywalls. But in the hands of builders. Massive respect to Seeed Studio for pushing true open-source robotics forward! We’re about to see: → Thousands of robot arms trained like athletes → Real-world datasets explode → Physical AI built in garages, not just research labs At SoloTech , this is the unlock we’ve been waiting for, training and deploying Physical AI skill models on real hardware, anywhere. No sims. No vaporware. No excuses. Production Grade Physical AI. Want to get access? Check out the Physical AI Gym in San Francisco or DM me.
0:20

Today I assembled the first batch of reBot DevArm in the U.S! First one on the planet! 🇺🇸⚙️ But this isn’t about a robot arm. This is about what happens when open source hits robotics. For years: •Hardware was locked •Robotics was expensive •Embodied AI was out of reach Now? Anyone can get their hands on: •6 DOF manipulation •Real precision (<0.2 mm repeatability) •Full stack access (ROS, Isaac Sim, LeRobot) This is how movements start. Not in labs. Not behind paywalls. But in the hands of builders. Massive respect to Seeed Studio for pushing true open-source robotics forward! We’re about to see: → Thousands of robot arms trained like athletes → Real-world datasets explode → Physical AI built in garages, not just research labs At SoloTech , this is the unlock we’ve been waiting for, training and deploying Physical AI skill models on real hardware, anywhere. No sims. No vaporware. No excuses. Production Grade Physical AI. Want to get access? Check out the Physical AI Gym in San Francisco or DM me.

Dhruv Diddi

32,335 Aufrufe • vor 6 Monaten

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla
1:06:07

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 Aufrufe • vor 9 Monaten

Italy's IIT really has so many humanoid gems. Came across their 2018 Centauro--a centaur-like disaster response robot that combines a quadruped lower body with a humanoid torso and arms. Built for harsh settings like rubble fields or disaster zones, it needs both stable movement over uneven ground and the strength to use human tools for forceful tasks such as breaching doors or clearing obstacles. The four legs deliver terrain-following locomotion and stair climbing, while the upper body handles dexterous, high-force manipulation. The robot blends full-body teleoperation--mapping an operator’s movements directly to the machine, with local autonomous modules for motion and impedance control, supporting supervised or higher-level autonomy when useful. At 1.5m tall and 93kg, it carries roughly 11kg payload per arm with impact-resilient actuators. Battery life is about 2.5 hours. Its hybrid wheeled-legged legs (6 DoF each, with wheels at the ankles acting like rolling hooves) provide better stability than pure bipeds while keeping tool use intuitive. This focus on matching robot morphology to real task demands, rather than defaulting to a standard humanoid form, remains a useful reference. Curious what others think:could hybrid centaur-style platforms play a larger role in future rescue or specialized operations?
2:06

Italy's IIT really has so many humanoid gems. Came across their 2018 Centauro--a centaur-like disaster response robot that combines a quadruped lower body with a humanoid torso and arms. Built for harsh settings like rubble fields or disaster zones, it needs both stable movement over uneven ground and the strength to use human tools for forceful tasks such as breaching doors or clearing obstacles. The four legs deliver terrain-following locomotion and stair climbing, while the upper body handles dexterous, high-force manipulation. The robot blends full-body teleoperation--mapping an operator’s movements directly to the machine, with local autonomous modules for motion and impedance control, supporting supervised or higher-level autonomy when useful. At 1.5m tall and 93kg, it carries roughly 11kg payload per arm with impact-resilient actuators. Battery life is about 2.5 hours. Its hybrid wheeled-legged legs (6 DoF each, with wheels at the ankles acting like rolling hooves) provide better stability than pure bipeds while keeping tool use intuitive. This focus on matching robot morphology to real task demands, rather than defaulting to a standard humanoid form, remains a useful reference. Curious what others think:could hybrid centaur-style platforms play a larger role in future rescue or specialized operations?

CyberRobo

41,894 Aufrufe • vor 3 Monaten

NVIDIA just unleashed SANA-WM and it’s an absolute MONSTER for the future of open source AI! A blazing-fast 2.6B-parameter open-source world model that doesn’t just generate video… it creates controllable, physics-rich, high-fidelity worlds on demand. Why this is insanely powerful: • One image + text prompt + 6-DoF camera trajectory → generates 720p videos up to 60 seconds long with buttery-smooth, precisely controlled camera movement. You’re not just watching, you’re piloting the simulation. • Runs locally on a single consumer GPU (RTX 5090 level) thanks to heavy distillation + NVFP4 quantization. Full 60-second clip denoised in ~34 seconds. No massive clusters required. • 36× higher throughput than previous open models while rivaling (or beating) closed industrial giants in visual quality and consistency. • Trained lightning-fast: ~213K public videos in just 15 days on 64 H100s. • Built with next-level tech: Hybrid Linear Attention, dual-branch camera control, two-stage pipeline, and rock-solid metric-scale pose understanding. This is a true open world model, the foundation for embodied AI, robotics, autonomous systems, and hyper-realistic simulations that can run anywhere. Project: At our Zero-Human Company, we’re already running SANA-WM live in our core pipelines. It’s supercharging autonomous agent training, generating unlimited synthetic training data, and powering full end-to-end simulation loops, zero humans in the loop. The speed and control let us test thousands of edge-case scenarios overnight, iterate at lightspeed, and push our fully autonomous operations further than ever before. This is the kind of breakthrough that turns science fiction into daily reality. World models just leveled up — hard. The age of personal, local, controllable universes is here.
1:13

NVIDIA just unleashed SANA-WM and it’s an absolute MONSTER for the future of open source AI! A blazing-fast 2.6B-parameter open-source world model that doesn’t just generate video… it creates controllable, physics-rich, high-fidelity worlds on demand. Why this is insanely powerful: • One image + text prompt + 6-DoF camera trajectory → generates 720p videos up to 60 seconds long with buttery-smooth, precisely controlled camera movement. You’re not just watching, you’re piloting the simulation. • Runs locally on a single consumer GPU (RTX 5090 level) thanks to heavy distillation + NVFP4 quantization. Full 60-second clip denoised in ~34 seconds. No massive clusters required. • 36× higher throughput than previous open models while rivaling (or beating) closed industrial giants in visual quality and consistency. • Trained lightning-fast: ~213K public videos in just 15 days on 64 H100s. • Built with next-level tech: Hybrid Linear Attention, dual-branch camera control, two-stage pipeline, and rock-solid metric-scale pose understanding. This is a true open world model, the foundation for embodied AI, robotics, autonomous systems, and hyper-realistic simulations that can run anywhere. Project: At our Zero-Human Company, we’re already running SANA-WM live in our core pipelines. It’s supercharging autonomous agent training, generating unlimited synthetic training data, and powering full end-to-end simulation loops, zero humans in the loop. The speed and control let us test thousands of edge-case scenarios overnight, iterate at lightspeed, and push our fully autonomous operations further than ever before. This is the kind of breakthrough that turns science fiction into daily reality. World models just leveled up — hard. The age of personal, local, controllable universes is here.

Brian Roemmele

619,679 Aufrufe • vor 5 Monaten

AgiBot’s new generation of industrial-grade interactive embodied robot, AgiBot G2, has officially launched! The G2 has already secured orders worth hundreds of millions of RMB, including two separate contracts each exceeding 100 million RMB, and has begun its first commercial deliveries. The AgiBot G2 is built to industrial standards, featuring high-performance joints, precision torque sensors, and an advanced spatial perception system. It supports rapid learning and deployment, offers strong multimodal voice interaction, and is designed for general use in industrial, logistics, and guidance scenarios. Inheriting the successful "Collect-Train-Deploy" model of its predecessor, the G1, the G2 brings significant upgrades, including a high-performance AI computing platform and actuators that enable omnidirectional obstacle avoidance and high-precision force-control tasks. Its 3-DOF waist allows for human-like bending and lateral body movement. A key feature is the G2's globally first-of-its-kind cross-shaped wrist force-control arm, which uses precision joint torque sensors and joint impedance control to delicately perceive external forces and respond smoothly. For continuous operation, the G2 supports autonomous charging and features a dual-battery hot-swapping system, meeting the 24-hour cycle demands of factory production lines. During the launch event, AgiBot demonstrated the G2’s ultra-low latency remote operation (teleoperation) capabilities. Operators successfully demonstrated precision shots (like hitting a floating balloon in Shanghai while operating from Beijing), showcasing the robot's high accuracy and low latency in both line-of-sight and beyond-line-of-sight scenarios. The G2 is already being deployed across four key real-world scenarios: In automotive parts production, it assists humans with tasks like safety belt lock core pressing and material handling. In precision operations, it used reinforcement learning to master delicate tasks like inserting memory sticks in just one hour. In logistics, the G2, enhanced by AgiBot's OmniHand dexterous hand, efficiently handles various package types for sorting and loading. Its strong mobility allows it to adapt to over 95% of factory floors. AgiBot is also commencing the first batch of commercial deliveries under an over 100 million RMB procurement contract with Joyson Electronic, formally landing the G2 in the automotive parts manufacturing sector.
2:28

AgiBot’s new generation of industrial-grade interactive embodied robot, AgiBot G2, has officially launched! The G2 has already secured orders worth hundreds of millions of RMB, including two separate contracts each exceeding 100 million RMB, and has begun its first commercial deliveries. The AgiBot G2 is built to industrial standards, featuring high-performance joints, precision torque sensors, and an advanced spatial perception system. It supports rapid learning and deployment, offers strong multimodal voice interaction, and is designed for general use in industrial, logistics, and guidance scenarios. Inheriting the successful "Collect-Train-Deploy" model of its predecessor, the G1, the G2 brings significant upgrades, including a high-performance AI computing platform and actuators that enable omnidirectional obstacle avoidance and high-precision force-control tasks. Its 3-DOF waist allows for human-like bending and lateral body movement. A key feature is the G2's globally first-of-its-kind cross-shaped wrist force-control arm, which uses precision joint torque sensors and joint impedance control to delicately perceive external forces and respond smoothly. For continuous operation, the G2 supports autonomous charging and features a dual-battery hot-swapping system, meeting the 24-hour cycle demands of factory production lines. During the launch event, AgiBot demonstrated the G2’s ultra-low latency remote operation (teleoperation) capabilities. Operators successfully demonstrated precision shots (like hitting a floating balloon in Shanghai while operating from Beijing), showcasing the robot's high accuracy and low latency in both line-of-sight and beyond-line-of-sight scenarios. The G2 is already being deployed across four key real-world scenarios: In automotive parts production, it assists humans with tasks like safety belt lock core pressing and material handling. In precision operations, it used reinforcement learning to master delicate tasks like inserting memory sticks in just one hour. In logistics, the G2, enhanced by AgiBot's OmniHand dexterous hand, efficiently handles various package types for sorting and loading. Its strong mobility allows it to adapt to over 95% of factory floors. AgiBot is also commencing the first batch of commercial deliveries under an over 100 million RMB procurement contract with Joyson Electronic, formally landing the G2 in the automotive parts manufacturing sector.

RoboHub🤖

33,831 Aufrufe • vor 1 Jahr

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵
2:21

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,710 Aufrufe • vor 1 Jahr

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10–15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paper’s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and π0.5 across the tested tasks. 🧵 1.
1:17

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10–15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paper’s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and π0.5 across the tested tasks. 🧵 1.

Rohan Paul

11,253 Aufrufe • vor 3 Monaten

🚀 Introducing PantheonOS ( A Fully Open-Source Agent OS for Science PantheonOS began as a research project in my Stanford lab and has since evolved into a vision to redefine data science in the era of AI—starting with computational biology, especially single-cell and spatial genomics. PantheonOS is a general agent platform built from the ground up. It is arguably the first distributed agent framework designed for scientific data analysis. 🔑 Key Features 1. Multi-Agent Collaboration – Built-in paradigms for distributed, cross-machine cooperation among agents and toolsets. 2. Native Toolset Support – Python, R, Julia, LaTeX, and more—designed for real scientific workflows. 3. Modular & Extensible – Developer-friendly design with shallow wrappers, plus LLM-driven toolset generation. 4. Evolvable Agents – Capable of evolving large-scale code projects to achieve superhuman performance (e.g., evolving upon the original Harmony [I Korsunsky, 2019, Nature Biotechnology] and Scanorama [BL Hie, 2019, Nature Biotechnology] implementations), and even evolving the system itself to adapt to new fields. 🎉 Stepwise Release Strategy We’re releasing PantheonOS in stages: Pantheon-CLI (today!), followed by Pantheon-Lab, Pantheon-Notebook, Pantheon-Slack, and more. 🌟 Pantheon-CLI Highlights - We're not just building another CLI tool. We're defining how scientists will interact with data in the AI era. - Open, Powerful, Python-First – The first fully open-source, endlessly extendable scientific “vibe analysis” framework. - Mixed Programming Magic – Combine Python, natural language, R, or Julia—seamlessly in the same environment. - PhD-Level Assistant – A command-line agent for complex real-world genomics and beyond, handling workflows at the PhD level. - Privacy by Design – Run entirely offline with local LLMs—your data never leaves your computer. ✅ Proven Applications (10 Demonstrations) Computational biology: 1. ATAC-seq: From raw reads to peak matrix 2. RNA-seq: From raw reads to expression matrix 3. Complex single-cell workflows (PhD-level) 4. Hybrid natural language + R for Seurat annotation 5. Learning from web tutorials + invoking single-cell foundation models 6. Cell segmentation on 10x Genomics HD Visium data And beyond: 7. Mixed Python & R programming examples 8. Molecular docking & structural analysis 9. Exploratory factor analysis for behavioral survey data 10. Customer segmentation & finance analytics 🌐 Learn More & Get Started Website: Pantheon-CLI Documentation: GitHub Repo: 💬 Join our community: PantheonOS Slack: PantheonOS Discord:
1:16

🚀 Introducing PantheonOS ( A Fully Open-Source Agent OS for Science PantheonOS began as a research project in my Stanford lab and has since evolved into a vision to redefine data science in the era of AI—starting with computational biology, especially single-cell and spatial genomics. PantheonOS is a general agent platform built from the ground up. It is arguably the first distributed agent framework designed for scientific data analysis. 🔑 Key Features 1. Multi-Agent Collaboration – Built-in paradigms for distributed, cross-machine cooperation among agents and toolsets. 2. Native Toolset Support – Python, R, Julia, LaTeX, and more—designed for real scientific workflows. 3. Modular & Extensible – Developer-friendly design with shallow wrappers, plus LLM-driven toolset generation. 4. Evolvable Agents – Capable of evolving large-scale code projects to achieve superhuman performance (e.g., evolving upon the original Harmony [I Korsunsky, 2019, Nature Biotechnology] and Scanorama [BL Hie, 2019, Nature Biotechnology] implementations), and even evolving the system itself to adapt to new fields. 🎉 Stepwise Release Strategy We’re releasing PantheonOS in stages: Pantheon-CLI (today!), followed by Pantheon-Lab, Pantheon-Notebook, Pantheon-Slack, and more. 🌟 Pantheon-CLI Highlights - We're not just building another CLI tool. We're defining how scientists will interact with data in the AI era. - Open, Powerful, Python-First – The first fully open-source, endlessly extendable scientific “vibe analysis” framework. - Mixed Programming Magic – Combine Python, natural language, R, or Julia—seamlessly in the same environment. - PhD-Level Assistant – A command-line agent for complex real-world genomics and beyond, handling workflows at the PhD level. - Privacy by Design – Run entirely offline with local LLMs—your data never leaves your computer. ✅ Proven Applications (10 Demonstrations) Computational biology: 1. ATAC-seq: From raw reads to peak matrix 2. RNA-seq: From raw reads to expression matrix 3. Complex single-cell workflows (PhD-level) 4. Hybrid natural language + R for Seurat annotation 5. Learning from web tutorials + invoking single-cell foundation models 6. Cell segmentation on 10x Genomics HD Visium data And beyond: 7. Mixed Python & R programming examples 8. Molecular docking & structural analysis 9. Exploratory factor analysis for behavioral survey data 10. Customer segmentation & finance analytics 🌐 Learn More & Get Started Website: Pantheon-CLI Documentation: GitHub Repo: 💬 Join our community: PantheonOS Slack: PantheonOS Discord:

evo-devo

17,476 Aufrufe • vor 1 Jahr

My conversation with Sergey Levine (Sergey Levine). Sergey is the co-founder of Physical Intelligence -- a company building foundation models that can control any robot to do any task in any environment. The company's thesis is that generality is more scalable than specialization, meaning that a model trained across many different robots and tasks will ultimately outperform any system built to do one thing well (eg, just wash dishes). Sergey is a researcher by background, but I think you will appreciate how practical and commercially grounded this conversation is. We discuss: - Why changing a diaper will be the last task a robot masters - The simulation v. real-world data debate - How multimodal LLMs give robots common sense - Moravec's Paradox + Robot Olympics - Why robots can do long-horizon tasks now - A realistic timeline for robots in our homes I should note that I am an investor in Physical Intelligence -- I made the investment because I believe it is one of the most important companies tackling the problem of robotics. Enjoy! Timestamps: 0:00 Intro 2:39 Defining Physical Intelligence 5:19 The Challenge of Building General Models 6:34 The Stakes and Future of General Purpose Robotics 8:15 Pros and Cons of Humanoid Robots 10:12 Historical Milestones in Robotics Research 15:31 Combining Generative AI and Deep RL 21:24 Moravec's Paradox 25:33 Kitchen Robots 29:30 Simulation vs. Real-World Data 30:48 The Robot Olympics 36:31 The Physiological Reality of Embodiment 38:56 Controversies in the Robotics Community 44:18 What Makes a Great Researcher 48:27 How Businesses Should Prepare for Robotics 54:09 Tracking Progress Through Research Papers 57:02 The Next Step: Mid-Level Reasoning 1:02:00 The Kindest Thing
1:15:15

My conversation with Sergey Levine (Sergey Levine). Sergey is the co-founder of Physical Intelligence -- a company building foundation models that can control any robot to do any task in any environment. The company's thesis is that generality is more scalable than specialization, meaning that a model trained across many different robots and tasks will ultimately outperform any system built to do one thing well (eg, just wash dishes). Sergey is a researcher by background, but I think you will appreciate how practical and commercially grounded this conversation is. We discuss: - Why changing a diaper will be the last task a robot masters - The simulation v. real-world data debate - How multimodal LLMs give robots common sense - Moravec's Paradox + Robot Olympics - Why robots can do long-horizon tasks now - A realistic timeline for robots in our homes I should note that I am an investor in Physical Intelligence -- I made the investment because I believe it is one of the most important companies tackling the problem of robotics. Enjoy! Timestamps: 0:00 Intro 2:39 Defining Physical Intelligence 5:19 The Challenge of Building General Models 6:34 The Stakes and Future of General Purpose Robotics 8:15 Pros and Cons of Humanoid Robots 10:12 Historical Milestones in Robotics Research 15:31 Combining Generative AI and Deep RL 21:24 Moravec's Paradox 25:33 Kitchen Robots 29:30 Simulation vs. Real-World Data 30:48 The Robot Olympics 36:31 The Physiological Reality of Embodiment 38:56 Controversies in the Robotics Community 44:18 What Makes a Great Researcher 48:27 How Businesses Should Prepare for Robotics 54:09 Tracking Progress Through Research Papers 57:02 The Next Step: Mid-Level Reasoning 1:02:00 The Kindest Thing

Patrick OShaughnessy

134,397 Aufrufe • vor 6 Monaten

One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything *just right* or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!
0:43

One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything *just right* or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!

Harrison Kinsley

54,223 Aufrufe • vor 1 Monat