Загрузка видео...

Не удалось загрузить видео

На главную

The next evolution: VLA+ models Just yesterday Microsoft Research released Rho-alpha (ρα) – their first robotics model, built on the Phi family. While most Vision-Language-Action (VLA) models stop at vision and language, Rho-alpha adds: ▪️ Tactile sensing to feel objects during manipulation ▪️ Online learning that lets it improve...

TheTuringPost's profile picture

Turing Post

87,258 subscribers

62,494 просмотров • 7 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:34

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

60,985 просмотров • 7 месяцев назад

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:47

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,090 просмотров • 8 месяцев назад

We believe we’re the first robotics company to demonstrate a robot peeling an apple with dual dexterous human-like hands. This breakthrough closes a key gap in robotics, achieving bimanual, contact-rich manipulation and moving far beyond the limits of simple grippers. 🧵↓ Today’s AI models (VLMs) are excellent at perception but struggle with action. Controlling high-degree-of-freedom hands for tasks like this is incredibly complex, and precise finger-level teleoperation is nearly impossible for humans. Our first step was a shared-autonomy system: rather than controlling every finger, the operator triggers pre-learned skills like a “rotate apple or tennis ball” primitive via a keyboard press or pedal. This makes scalable data collection and RL training possible. How does the AI manage this? We created "MoDE-VLA" (Mixture of Dexterous Experts). It fuses vision, language, force, and touch data by using a team of specialist "experts," making control in high-dimensional spaces stable and effective. The combination of these two innovations allows for seamless, contact-rich manipulation. The human provides high-level guidance, and the robot executes the complex in-hand coordination required. This work paves the way for robots that can safely handle delicate tasks in human environments. Want the full technical details? 📄 Read the full research paper: Visit us at NVIDIA GTC Booth #1838, Hall 3 to learn more! #Robotics #AI #DexterousManipulation #VLA #NVIDIAGTC Nancy Villicaña NVIDIA GTC
2:51

We believe we’re the first robotics company to demonstrate a robot peeling an apple with dual dexterous human-like hands. This breakthrough closes a key gap in robotics, achieving bimanual, contact-rich manipulation and moving far beyond the limits of simple grippers. 🧵↓ Today’s AI models (VLMs) are excellent at perception but struggle with action. Controlling high-degree-of-freedom hands for tasks like this is incredibly complex, and precise finger-level teleoperation is nearly impossible for humans. Our first step was a shared-autonomy system: rather than controlling every finger, the operator triggers pre-learned skills like a “rotate apple or tennis ball” primitive via a keyboard press or pedal. This makes scalable data collection and RL training possible. How does the AI manage this? We created "MoDE-VLA" (Mixture of Dexterous Experts). It fuses vision, language, force, and touch data by using a team of specialist "experts," making control in high-dimensional spaces stable and effective. The combination of these two innovations allows for seamless, contact-rich manipulation. The human provides high-level guidance, and the robot executes the complex in-hand coordination required. This work paves the way for robots that can safely handle delicate tasks in human environments. Want the full technical details? 📄 Read the full research paper: Visit us at NVIDIA GTC Booth #1838, Hall 3 to learn more! #Robotics #AI #DexterousManipulation #VLA #NVIDIAGTC Nancy Villicaña NVIDIA GTC

Sharpa

20,429 просмотров • 6 месяцев назад

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.
1:01

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

The Humanoid Hub

35,204 просмотров • 7 месяцев назад

I ran a quick test of Gemini Robotics ER 1.5 controlling my LeKiwi using the MCP I developed before. No extra code required - just specified a new model. Unfortunately, it doesn't work well, even though I had expected it to be the best, as it is the first major VLM model trained specifically for robotics applications. The video below is a relatively good attempt. It does understand what is going on and generates a reasonable action plan, but it is not doing well with the low-level control. I guess it was not trained for this and was used together with the VLA, so it is trained to generate more high-level commands like "grab this object and put it there" rather than doing low-level control specifying every robot movement step by step. Did anybody try to use it in a similar setup? What is your experience?
1:11

I ran a quick test of Gemini Robotics ER 1.5 controlling my LeKiwi using the MCP I developed before. No extra code required - just specified a new model. Unfortunately, it doesn't work well, even though I had expected it to be the best, as it is the first major VLM model trained specifically for robotics applications. The video below is a relatively good attempt. It does understand what is going on and generates a reasonable action plan, but it is not doing well with the low-level control. I guess it was not trained for this and was used together with the VLA, so it is trained to generate more high-level commands like "grab this object and put it there" rather than doing low-level control specifying every robot movement step by step. Did anybody try to use it in a similar setup? What is your experience?

Ilia

15,643 просмотров • 11 месяцев назад

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:
0:38

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Ilir Aliu

18,210 просмотров • 7 месяцев назад

X-Humanoid just officially dropped Embodied Tien Kung 3.0, A universal platform designed to be way more open and developer-friendly. 🤖 Built on their Wise Kaiwu AI platform, this next-gen humanoid is all about slashing development costs. It’s a fully interoperable ecosystem that supports everything from tactile interaction to high-dynamic motion control at a full humanoid scale. ➤ Radical Openness: X-Humanoid is open-sourcing the full stack—robot body, motion control, VLM/VLA models, and the RoboMIND dataset. It fully supports ROS2, MQTT, and TCP/IP, so developers can customize use cases without re-engineering the basics. ➤ High-Performance Hardware: With high-torque integrated joints, Tien Kung 3.0 can clear 1-meter (3.3ft) obstacles and handle dexterous moves like kneeling and bending. It hits millimeter-level precision, making it a solid fit for industrial-grade tasks. ➤ True Autonomy: The bot runs a continuous perception-decision-execution loop. It uses world models to break down complex language commands and VLA models for real-time obstacle avoidance and navigation. ➤ Scalable Collaboration: The platform moves beyond single-unit tasks to support multi-robot collaboration with autonomous scheduling. It’s built to move embodied AI from the lab straight into real-world commercial and industrial environments. Source: X-Humanoid #Humanoid #OpenSource #Robotics #EmbodiedAI #PhysicalAI #Automation #XHumanoid #TienKung #WiseKaiwu
2:18

X-Humanoid just officially dropped Embodied Tien Kung 3.0, A universal platform designed to be way more open and developer-friendly. 🤖 Built on their Wise Kaiwu AI platform, this next-gen humanoid is all about slashing development costs. It’s a fully interoperable ecosystem that supports everything from tactile interaction to high-dynamic motion control at a full humanoid scale. ➤ Radical Openness: X-Humanoid is open-sourcing the full stack—robot body, motion control, VLM/VLA models, and the RoboMIND dataset. It fully supports ROS2, MQTT, and TCP/IP, so developers can customize use cases without re-engineering the basics. ➤ High-Performance Hardware: With high-torque integrated joints, Tien Kung 3.0 can clear 1-meter (3.3ft) obstacles and handle dexterous moves like kneeling and bending. It hits millimeter-level precision, making it a solid fit for industrial-grade tasks. ➤ True Autonomy: The bot runs a continuous perception-decision-execution loop. It uses world models to break down complex language commands and VLA models for real-time obstacle avoidance and navigation. ➤ Scalable Collaboration: The platform moves beyond single-unit tasks to support multi-robot collaboration with autonomous scheduling. It’s built to move embodied AI from the lab straight into real-world commercial and industrial environments. Source: X-Humanoid #Humanoid #OpenSource #Robotics #EmbodiedAI #PhysicalAI #Automation #XHumanoid #TienKung #WiseKaiwu

RoboHub🤖

49,440 просмотров • 7 месяцев назад

Most robotics AI models suffer from the "stop-and-think" problem. They take a static picture, pause to reason, execute an action, and repeat. In the real world, that latency causes spills, collisions, and failed tasks. Google DeepMind just launched Gemini Robotics ER 2: an embodied reasoning model that thinks and acts at the speed of the physical world. Here's why this is a step-change for physical AI engineering: Traditional robotics models rely on static snapshots. But knowing *when* a task is done, such as when to stop pouring coffee into a cup or when a trash bag is securely tied, requires continuous temporal awareness. Gemini Robotics ER 2 integrates directly with the bidirectional streaming Gemini Live API to reason about what comes next while simultaneously executing motor actions. What makes Gemini Robotics ER 2 different: 🎯 91.3% accuracy on live video moment-finding (0.96s mean absolute distance) at 4x the execution speed of frontier models 📈 Continuous progress tracking across 5 completion stages (57.4% accuracy) to self-correct mid-task without restarting 🛠️ Native agentic tool orchestration that commands lower-level VLA models, navigation APIs, and Google Search 🤝 Multi-robot collaboration allowing physically diverse machines (like Apptronik's Apollo 2 humanoid and Franka's FR3 Duo arm) to hand off tasks in shared spaces 🛡️ Built-in physical safety that autonomously halts robots when humans enter a workspace and resumes once clear
2:59

Most robotics AI models suffer from the "stop-and-think" problem. They take a static picture, pause to reason, execute an action, and repeat. In the real world, that latency causes spills, collisions, and failed tasks. Google DeepMind just launched Gemini Robotics ER 2: an embodied reasoning model that thinks and acts at the speed of the physical world. Here's why this is a step-change for physical AI engineering: Traditional robotics models rely on static snapshots. But knowing *when* a task is done, such as when to stop pouring coffee into a cup or when a trash bag is securely tied, requires continuous temporal awareness. Gemini Robotics ER 2 integrates directly with the bidirectional streaming Gemini Live API to reason about what comes next while simultaneously executing motor actions. What makes Gemini Robotics ER 2 different: 🎯 91.3% accuracy on live video moment-finding (0.96s mean absolute distance) at 4x the execution speed of frontier models 📈 Continuous progress tracking across 5 completion stages (57.4% accuracy) to self-correct mid-task without restarting 🛠️ Native agentic tool orchestration that commands lower-level VLA models, navigation APIs, and Google Search 🤝 Multi-robot collaboration allowing physically diverse machines (like Apptronik's Apollo 2 humanoid and Franka's FR3 Duo arm) to hand off tasks in shared spaces 🛡️ Built-in physical safety that autonomously halts robots when humans enter a workspace and resumes once clear

Karl Weinmeister

28,313 просмотров • 1 месяц назад

Robotics is getting its Raspberry Pi moment. This solves one of the biggest frictions in Physical AI: Everything has been fragmented. Different tools for control. Sensors. Calibration. Learning. Now it’s one unified stack: • motor control to VLA models in one system • built for real hardware, not just simulation -> norma_core_dev just entered PUBLIC ALPHA. The real unlock is ElRobot: • 7+1 DOF arm • fully 3D printable • ~220$ cost • ships with CAD, URDF, full build guide Not a demo… Infrastructure. And the part most people will underestimate: auto-calibration for any robot No manual tuning. No trial and error. Just: code → hardware → working system That removes one of the biggest hidden bottlenecks in robotics. Which unlocks: • faster iteration • cheaper experiments • real-world data loops Exactly what Physical AI needs. We’re moving from: “Can we make it work?” to “How fast can we improve it?” Thank you so much for sharing, Erick! 📍Repro: Credit:
0:50

Robotics is getting its Raspberry Pi moment. This solves one of the biggest frictions in Physical AI: Everything has been fragmented. Different tools for control. Sensors. Calibration. Learning. Now it’s one unified stack: • motor control to VLA models in one system • built for real hardware, not just simulation -> norma_core_dev just entered PUBLIC ALPHA. The real unlock is ElRobot: • 7+1 DOF arm • fully 3D printable • ~220$ cost • ships with CAD, URDF, full build guide Not a demo… Infrastructure. And the part most people will underestimate: auto-calibration for any robot No manual tuning. No trial and error. Just: code → hardware → working system That removes one of the biggest hidden bottlenecks in robotics. Which unlocks: • faster iteration • cheaper experiments • real-world data loops Exactly what Physical AI needs. We’re moving from: “Can we make it work?” to “How fast can we improve it?” Thank you so much for sharing, Erick! 📍Repro: Credit:

Ilir Aliu

66,329 просмотров • 5 месяцев назад

NEWS: NVIDIA just announced Alpamayo, what CEO Jensen Huang calls the world’s first thinking, reasoning autonomous vehicle AI, launching on U.S. roads later this year, starting with the Mercedes CLA. Jensen: "It's trained end-to-end. Literally from camera in to actuation out; It reasons what action it is about to take, the reason by which is came about that action, and the trajectory." Alpamayo introduces Vision-Language-Action (VLA) models, which enable self-driving systems to interpret what they see, reason about complex driving scenarios, and generate driving actions. The platform includes large reasoning models, simulation tools for testing rare and edge-case scenarios, and open datasets for training and validation. NVIDIA says the approach improves transparency, safety, and robustness in autonomous systems, particularly in complex real-world environments, and supports progress toward higher levels of vehicle autonomy: "With a 10-billion-parameter architecture, Alpamayo 1 uses video input to generate trajectories alongside reasoning traces, showing the logic behind each decision. Developers can adapt Alpamayo 1 into smaller runtime models for vehicle development, or use it as a foundation for AV development tools such as reasoning-based evaluators and auto-labeling systems. Alpamayo 1 provides open model weights and open-source inferencing scripts. Future models in the family will feature larger parameter counts, more detailed reasoning capabilities, more input and output flexibility, and options for commercial usage."
2:41

NEWS: NVIDIA just announced Alpamayo, what CEO Jensen Huang calls the world’s first thinking, reasoning autonomous vehicle AI, launching on U.S. roads later this year, starting with the Mercedes CLA. Jensen: "It's trained end-to-end. Literally from camera in to actuation out; It reasons what action it is about to take, the reason by which is came about that action, and the trajectory." Alpamayo introduces Vision-Language-Action (VLA) models, which enable self-driving systems to interpret what they see, reason about complex driving scenarios, and generate driving actions. The platform includes large reasoning models, simulation tools for testing rare and edge-case scenarios, and open datasets for training and validation. NVIDIA says the approach improves transparency, safety, and robustness in autonomous systems, particularly in complex real-world environments, and supports progress toward higher levels of vehicle autonomy: "With a 10-billion-parameter architecture, Alpamayo 1 uses video input to generate trajectories alongside reasoning traces, showing the logic behind each decision. Developers can adapt Alpamayo 1 into smaller runtime models for vehicle development, or use it as a foundation for AV development tools such as reasoning-based evaluators and auto-labeling systems. Alpamayo 1 provides open model weights and open-source inferencing scripts. Future models in the family will feature larger parameter counts, more detailed reasoning capabilities, more input and output flexibility, and options for commercial usage."

Sawyer Merritt

1,604,755 просмотров • 8 месяцев назад

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla
1:06:07

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 просмотров • 8 месяцев назад

There’s not enough hype about Flexion Robotics. They are literally one of the sexiest European companies these days! 🇪🇺 I had a chance to meet the founder Nikita last year and learn his story. He turned down offers from top physical AI companies to build his own. The reason was his specific vision of how robot intelligence should be built to actually scale. So he decided to build one himself. Flexion is building an ultimate brain for humanoid robots. Yesterday they released Reflect v1.0. It’s an intelligence platform for long-horizon autonomous missions. The system has three layers. At the top, a custom vision-language model acts as mission control, observing the robot's camera feed and replanning continuously. Below that, a motion layer combines a VLA trained on real-world data with reinforcement learning-based skills. It translates perception into navigation, scene interaction, and manipulation commands. At the bottom, a whole-body controller runs in real-time, maintaining balance while the robot navigates and performs tasks. I highly recommend them visiting Zurich! 🇨🇭 Nikita Rudin cheapeu bas on the launch! 🎩 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → http://
0:23

There’s not enough hype about Flexion Robotics. They are literally one of the sexiest European companies these days! 🇪🇺 I had a chance to meet the founder Nikita last year and learn his story. He turned down offers from top physical AI companies to build his own. The reason was his specific vision of how robot intelligence should be built to actually scale. So he decided to build one himself. Flexion is building an ultimate brain for humanoid robots. Yesterday they released Reflect v1.0. It’s an intelligence platform for long-horizon autonomous missions. The system has three layers. At the top, a custom vision-language model acts as mission control, observing the robot's camera feed and replanning continuously. Below that, a motion layer combines a VLA trained on real-world data with reinforcement learning-based skills. It translates perception into navigation, scene interaction, and manipulation commands. At the bottom, a whole-body controller runs in real-time, maintaining balance while the robot navigates and performs tasks. I highly recommend them visiting Zurich! 🇨🇭 Nikita Rudin cheapeu bas on the launch! 🎩 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → http://

Lukas Ziegler

23,200 просмотров • 2 месяцев назад

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):
0:39

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Léo

32,331 просмотров • 1 месяц назад

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:
0:58

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:

Léo

15,662 просмотров • 1 месяц назад

Japan Just Built a HouseBot You Control Without Speaking and It Changes Everything! Donut Robotics has officially unveiled its first bipedal humanoid, Cinnamon 1, and instead of focusing on louder voices or bigger motors, the company went in the opposite direction. Silence. Cinnamon 1 introduces what Donut Robotics calls Silent Gesture Control, a system that allows the humanoid to be guided using simple hand and finger movements rather than spoken commands. This approach feels especially well suited for real world environments where traditional voice control falls apart. Busy factory floors. Construction sites filled with constant noise. Even quiet indoor settings where voice commands feel awkward or intrusive. It also opens the door for far more accessible human robot interaction, particularly for users with impairments. While the current Cinnamon 1 hardware is built on an OEM platform, the intelligence driving it is where Donut Robotics is placing its long term bet. The team is actively developing custom Vision Language Action AI that allows the robot to interpret what it sees, understand intent, and respond with physical action. The goal is not just smarter robots, but robots that feel more natural. Even more ambitious is the company’s plan for full domestic production. Donut Robotics has stated its intention to localize both manufacturing and AI development in Japan, reinforcing the country’s reputation for precision engineering and thoughtful robotics design. If timelines hold, Cinnamon 1 units are expected to begin deployment in factories and construction environments by the end of 2026. That puts this humanoid squarely in the category of near term reality rather than distant concept. The takeaway is simple but important. As humanoid robots move out of labs and into daily work environments, the winners may not be the loudest or flashiest machines. They may be the ones that understand us without a word being spoken.
0:31

Japan Just Built a HouseBot You Control Without Speaking and It Changes Everything! Donut Robotics has officially unveiled its first bipedal humanoid, Cinnamon 1, and instead of focusing on louder voices or bigger motors, the company went in the opposite direction. Silence. Cinnamon 1 introduces what Donut Robotics calls Silent Gesture Control, a system that allows the humanoid to be guided using simple hand and finger movements rather than spoken commands. This approach feels especially well suited for real world environments where traditional voice control falls apart. Busy factory floors. Construction sites filled with constant noise. Even quiet indoor settings where voice commands feel awkward or intrusive. It also opens the door for far more accessible human robot interaction, particularly for users with impairments. While the current Cinnamon 1 hardware is built on an OEM platform, the intelligence driving it is where Donut Robotics is placing its long term bet. The team is actively developing custom Vision Language Action AI that allows the robot to interpret what it sees, understand intent, and respond with physical action. The goal is not just smarter robots, but robots that feel more natural. Even more ambitious is the company’s plan for full domestic production. Donut Robotics has stated its intention to localize both manufacturing and AI development in Japan, reinforcing the country’s reputation for precision engineering and thoughtful robotics design. If timelines hold, Cinnamon 1 units are expected to begin deployment in factories and construction environments by the end of 2026. That puts this humanoid squarely in the category of near term reality rather than distant concept. The takeaway is simple but important. As humanoid robots move out of labs and into daily work environments, the winners may not be the loudest or flashiest machines. They may be the ones that understand us without a word being spoken.

The AI Robot Guy on X

257,928 просмотров • 7 месяцев назад

Without World Models, There Is No AGI. Google Just Proved It. If AGI ever happens, it will not come from bigger chatbots alone. From the very start of this interview, one thing is crystal clear: without world models, we will never reach AGI. And right now, Google is leading with its world simulator Genie 3. Here is the core of what Demis Hassabis explains in this conversation: • World models are the missing core of AGI Hassabis says his deepest long term focus has always been world models and simulations. Not just language. Not just prediction. Actual internal simulations of reality. • LLMs are impressive, but incomplete Language models understand more about the world than expected because human language encodes a lot of reality. Still, language is only a shadow of the real thing. • What text can never fully teach Reality includes things text struggles to express: •3D space and spatial dynamics •Physical causality and mechanics •Sensorimotor experience like movement, force, smell, or balance • Experience beats description To close the gap, AI must learn from interaction and experience, not just static text. That is how you build an internal world simulator. • Why Genie 3 matters With Google DeepMind pushing systems like Genie 3, AI starts to model reality itself, not just talk about it. • Robots and real world assistants depend on this True robotics, smart glasses, and universal assistants require AI that understands the physical world you live in, not just your screen. Bottom line: AGI will not emerge from better text prediction. It will emerge from systems that can simulate, predict, and understand reality itself. Right now, Google is clearly ahead on that path. Curious what you think. Are world models the real AGI unlock, or just another stepping stone?
1:26

Without World Models, There Is No AGI. Google Just Proved It. If AGI ever happens, it will not come from bigger chatbots alone. From the very start of this interview, one thing is crystal clear: without world models, we will never reach AGI. And right now, Google is leading with its world simulator Genie 3. Here is the core of what Demis Hassabis explains in this conversation: • World models are the missing core of AGI Hassabis says his deepest long term focus has always been world models and simulations. Not just language. Not just prediction. Actual internal simulations of reality. • LLMs are impressive, but incomplete Language models understand more about the world than expected because human language encodes a lot of reality. Still, language is only a shadow of the real thing. • What text can never fully teach Reality includes things text struggles to express: •3D space and spatial dynamics •Physical causality and mechanics •Sensorimotor experience like movement, force, smell, or balance • Experience beats description To close the gap, AI must learn from interaction and experience, not just static text. That is how you build an internal world simulator. • Why Genie 3 matters With Google DeepMind pushing systems like Genie 3, AI starts to model reality itself, not just talk about it. • Robots and real world assistants depend on this True robotics, smart glasses, and universal assistants require AI that understands the physical world you live in, not just your screen. Bottom line: AGI will not emerge from better text prediction. It will emerge from systems that can simulate, predict, and understand reality itself. Right now, Google is clearly ahead on that path. Curious what you think. Are world models the real AGI unlock, or just another stepping stone?

VraserX e/acc

23,784 просмотров • 8 месяцев назад

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵
2:21

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,069 просмотров • 1 год назад

One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything *just right* or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!
0:43

One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything *just right* or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!

Harrison Kinsley

52,303 просмотров • 8 дней назад

Terence Tao, Professor of Mathematics at UCLA and Fields Medalist, on why nobody can fully explain why LLMs work: Tao starts with the mechanics, which are no mystery at all. You gather an enormous amount of text and you fit a curve to it. "The magic of LLMs is that if you train these LLMs on enough data — so trillions and trillions of data points — and you really try to fit as good a curve as possible, and this takes like millions and millions of dollars of computing power and months and months of time, then suddenly, even when you iterate, it stays coherent. It begins to sound not like monkeys but it actually sounds like a human speaking." That is the entire recipe: data, compute, time, curve-fitting. None of it obviously adds up to fluent English. Then Tao says the part that most people building these systems move past quickly: "And somehow we don't fully understand why that's the case." The admission comes from one of the most capable living mathematicians, and the gap he describes sits at the centre of the field. His best account of what's happening puts the mystery in the language rather than the machine: "What seems to be true is that language, like English or other natural languages, contains a lot of hidden patterns that we're not consciously aware of. I mean, we know some of the laws of English, there's laws of grammar and things, but there are sort of unspoken, unwritten rules of language that humans pick up." It relocates the question: the structure was always latent in the text, and the model found it. Why enough curve-fitting surfaces that structure is still unanswered. Tao reaches for a child to explain it: "A human child, even though they're not taught what a noun is, what a verb is or whatever, they can pick up what order English words go in just by continual exposure to the language." Which is honest about the limits of the explanation, because we can't fully account for how children do it either. From there, the unexplained behaviour compounds. Exposure to language turns out to be enough to produce something that looks like reasoning: "It seems like you can teach these models to also pick up patterns in language to the point where you can give them math questions. The answer to 2 plus 3 is — and they will say five." And once a model handles language at all, you can push it into resembling self-correction: "Once you have a little bit of ability to speak English, you can kind of go in loops and sort of check your work and make fewer mistakes, and you can prompt these models to proceed step by step and not say something unless it's been double checked and so forth. And so they become a little bit smarter, quote unquote, to the point where they can solve many, many complicated tasks." The scare quotes around "smarter" carry the whole argument. Tao does not concede that the unexplained fluency implies anything underneath it: "But they're still just guessing the next word to say. It's not really grounded in any deep understanding of the real world. It's just that they have seen the patterns in the English language or other language that they've absorbed so well."
1:50

Terence Tao, Professor of Mathematics at UCLA and Fields Medalist, on why nobody can fully explain why LLMs work: Tao starts with the mechanics, which are no mystery at all. You gather an enormous amount of text and you fit a curve to it. "The magic of LLMs is that if you train these LLMs on enough data — so trillions and trillions of data points — and you really try to fit as good a curve as possible, and this takes like millions and millions of dollars of computing power and months and months of time, then suddenly, even when you iterate, it stays coherent. It begins to sound not like monkeys but it actually sounds like a human speaking." That is the entire recipe: data, compute, time, curve-fitting. None of it obviously adds up to fluent English. Then Tao says the part that most people building these systems move past quickly: "And somehow we don't fully understand why that's the case." The admission comes from one of the most capable living mathematicians, and the gap he describes sits at the centre of the field. His best account of what's happening puts the mystery in the language rather than the machine: "What seems to be true is that language, like English or other natural languages, contains a lot of hidden patterns that we're not consciously aware of. I mean, we know some of the laws of English, there's laws of grammar and things, but there are sort of unspoken, unwritten rules of language that humans pick up." It relocates the question: the structure was always latent in the text, and the model found it. Why enough curve-fitting surfaces that structure is still unanswered. Tao reaches for a child to explain it: "A human child, even though they're not taught what a noun is, what a verb is or whatever, they can pick up what order English words go in just by continual exposure to the language." Which is honest about the limits of the explanation, because we can't fully account for how children do it either. From there, the unexplained behaviour compounds. Exposure to language turns out to be enough to produce something that looks like reasoning: "It seems like you can teach these models to also pick up patterns in language to the point where you can give them math questions. The answer to 2 plus 3 is — and they will say five." And once a model handles language at all, you can push it into resembling self-correction: "Once you have a little bit of ability to speak English, you can kind of go in loops and sort of check your work and make fewer mistakes, and you can prompt these models to proceed step by step and not say something unless it's been double checked and so forth. And so they become a little bit smarter, quote unquote, to the point where they can solve many, many complicated tasks." The scare quotes around "smarter" carry the whole argument. Tao does not concede that the unexplained fluency implies anything underneath it: "But they're still just guessing the next word to say. It's not really grounded in any deep understanding of the real world. It's just that they have seen the patterns in the English language or other language that they've absorbed so well."

Big Brain AI

158,259 просмотров • 1 день назад