Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🤖 Introducing InternVLA-A1 — now fully open-sourced! Many VLA models follow instructions well in static scenes… but struggle in dynamic environments (conveyor belts, rotating platforms, multi-robot setups). Why? They see the present—but can’t imagine the future. InternVLA-A1 solution: unify perception, imagination, and action in one model: ✅ Scene understanding:... Image + text → task parsing ✅ Task imagination: Predict future frames → reason about dynamics ✅ Guided control: Execute actions steered by visual foresight Powered by InternData-A1 - Large-scale high-quality simulated dataset, InternVLA-A1 stays robust under complex backgrounds, lighting, and distractions. 🔥 See it in action: 1️⃣ High-speed conveyor: track, predict, and stably grasp or flip packages 2️⃣ Rotating platform: task-aware recognition & precise pick-up of diverse items 📊 Outperforms π0 and Gr00t N1.5 on general manipulation benchmarks! ✨ Model, data, and code are all open! Models: Datasets: GitHub:show more

ModelScope

16,220 subscribers

38,118 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

12 Comments

Sanskar Pandey9 months ago

Impressive work - ive been thinking pretty hard about how in multi-agent contexts with heterogeneous robots richer scene semantics from exocentric views could really shine

Javier Martin9 months ago

@OpenRoboticsOrg The key is anticipation, not reaction. Predicting future states is what makes VLAs work in dynamic environments. Open-sourcing the full stack is a big win for the field.

Intern Robotics9 months ago

paper is coming：

Tony Chen9 months ago

Wow super cool thanks for opensourcing a unified model solution. Maybe have some summer interns face off as a benchmark and then try to improve it by the end of the internship 😅

MechatronicsEngineer9 months ago

Open sourcing’s a game changer curious how it handles predicting complex dynamics in real world tests.

HariLuru9 months ago

Hi, Is there any link to benchmarks?

Heytt S Satra9 months ago

predicting future frames for task foresight. thats the gap. static models blind to dynamics by design.

arjya9 months ago

@OpenRoboticsOrg Nice work

Himanshu Kumar9 months ago

I've seen similar issues; anticipating future states is a hard problem.

Juuno9 months ago

sounds cool cant wait to see what it can do

Gema Romero9 months ago

How's it doing on longer-horizon tasks like sequential assembly?

MBC9 months ago

Could it be used/useful for non-conveyor tasks? Reachy Minis and alike?

Related Videos

Introducing FLUX 3 Action. An open weights 7B World Action Model that achieves first place on the RoboLab benchmark. It outperforms the previous best open model by 6.1 percentage points while using 56% fewer parameters and running up to 3.95x faster.⁠⁠ FLUX 3 Action removes the usual trade-off between world action model performance and VLA speed: it still predicts video and actions together, but plans more than twice as far ahead and runs faster per second of robot motion than the strongest open VLA. Teams can fine-tune FLUX 3 Action on their own demonstrations to create policies for a particular robot and task. Together with NVIDIA, we also integrated FLUX 3 Action natively into Hugging Face's LeRobot, with fine-tuning recipes included and edge deployment on NVIDIA Jetson. Beyond robotics, we’re also seeing promising results training task-specific policies for acting in simulated environments like gaming, controlling a vehicle, computer use, and wherever else a model needs to understand a visual environment and then choose what to do next. FLUX 3 Action builds on the same image, video, and audio pretraining as FLUX 3, but uses a smaller architecture designed for practical deployment. In midtraining, we trained the model to predict actions and future frames together. We’re releasing the weights, code, fine-tuning recipe, benchmarks, and reproducible examples so researchers and developers can build on the model with their own robots, environments, and tasks (see below).

Introducing FLUX 3 Action. An open weights 7B World Action Model that achieves first place on the RoboLab benchmark. It outperforms the previous best open model by 6.1 percentage points while using 56% fewer parameters and running up to 3.95x faster.⁠⁠ FLUX 3 Action removes the usual trade-off between world action model performance and VLA speed: it still predicts video and actions together, but plans more than twice as far ahead and runs faster per second of robot motion than the strongest open VLA. Teams can fine-tune FLUX 3 Action on their own demonstrations to create policies for a particular robot and task. Together with NVIDIA, we also integrated FLUX 3 Action natively into Hugging Face's LeRobot, with fine-tuning recipes included and edge deployment on NVIDIA Jetson. Beyond robotics, we’re also seeing promising results training task-specific policies for acting in simulated environments like gaming, controlling a vehicle, computer use, and wherever else a model needs to understand a visual environment and then choose what to do next. FLUX 3 Action builds on the same image, video, and audio pretraining as FLUX 3, but uses a smaller architecture designed for practical deployment. In midtraining, we trained the model to predict actions and future frames together. We’re releasing the weights, code, fine-tuning recipe, benchmarks, and reproducible examples so researchers and developers can build on the model with their own robots, environments, and tasks (see below).

Black Forest Labs

181,735 views • 1 month ago

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,710 views • 1 year ago

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,620 views • 1 year ago

A team tested Pi0, Pi0 Fast, Gr00t, and ACT on real robot arms in manufacturing tasks. (🔖 Bookmark this for later!) The task was precise: place thin rectangular frames from a messy stack into a holder. The team fine-tuned each model on 100 real trajectories and compared training time, inference speed, motion quality, and success rates. ⬇️ Here’s a breakdown of what they found Pi0 (Original) ✅ Strongest overall performance in precise pick-and-place ✅ High success rate even in edge cases ✅ Longest training time (~11 hours, ~$30 per run) ✅ Inference time of 80 ms causes short pauses between actions Despite delays, it handles complex scenarios well… solid for high-precision tasks, but slow to train. Gr00t ✅ Trains fast (~2 hours, ~$5 per run) ✅ Performs almost as well as Pi0 on large-object tasks ✅ Struggles with fine precision; random movement in some trials ✅ More training didn’t fix jitter or random offsets Best suited for tasks where exact precision isn’t critical. Not ready for manufacturing-grade accuracy without more tuning. Pi0 Fast ✅ Promised faster training, but results were underwhelming ✅ Training at 6 hours still showed low success rates ✅ Inference was slower than expected ✅ Not reliable for generalizing even slightly new tasks Currently too unstable for real-world deployment. Doesn’t live up to the “Fast” name yet. ACT (Baseline) ✅ 200MB model—lightweight, but limited ✅ Struggles with stacked objects or ambiguous scenes ✅ Success rates around 70% in best-case setups ✅ Can’t match newer models on precision or generalization Still a solid baseline, but clearly a generation behind in robustness. 🚨 Extra Notes All newer models share a common issue: •Inference takes longer than a frame (80 ms vs 33 ms), so robots “pause” between chunks. •This results in jittery movements, but not a dealbreaker unless tasks are time-sensitive. Language-conditioned tasks also fell short: after training on two labeled tasks, the model couldn’t generalize to a third unseen combination using only text prompts. ✅ The good news? These models adapt well to new robot arms with quick fine-tuning. ❌ The bad news? There’s still no plug-and-play solution for improving performance after deployment. Reinforcement learning or DAgger-style data collection during real-world operation may be the next big step, something many teams in robotics are actively working on.

A team tested Pi0, Pi0 Fast, Gr00t, and ACT on real robot arms in manufacturing tasks. (🔖 Bookmark this for later!) The task was precise: place thin rectangular frames from a messy stack into a holder. The team fine-tuned each model on 100 real trajectories and compared training time, inference speed, motion quality, and success rates. ⬇️ Here’s a breakdown of what they found Pi0 (Original) ✅ Strongest overall performance in precise pick-and-place ✅ High success rate even in edge cases ✅ Longest training time (~11 hours, ~$30 per run) ✅ Inference time of 80 ms causes short pauses between actions Despite delays, it handles complex scenarios well… solid for high-precision tasks, but slow to train. Gr00t ✅ Trains fast (~2 hours, ~$5 per run) ✅ Performs almost as well as Pi0 on large-object tasks ✅ Struggles with fine precision; random movement in some trials ✅ More training didn’t fix jitter or random offsets Best suited for tasks where exact precision isn’t critical. Not ready for manufacturing-grade accuracy without more tuning. Pi0 Fast ✅ Promised faster training, but results were underwhelming ✅ Training at 6 hours still showed low success rates ✅ Inference was slower than expected ✅ Not reliable for generalizing even slightly new tasks Currently too unstable for real-world deployment. Doesn’t live up to the “Fast” name yet. ACT (Baseline) ✅ 200MB model—lightweight, but limited ✅ Struggles with stacked objects or ambiguous scenes ✅ Success rates around 70% in best-case setups ✅ Can’t match newer models on precision or generalization Still a solid baseline, but clearly a generation behind in robustness. 🚨 Extra Notes All newer models share a common issue: •Inference takes longer than a frame (80 ms vs 33 ms), so robots “pause” between chunks. •This results in jittery movements, but not a dealbreaker unless tasks are time-sensitive. Language-conditioned tasks also fell short: after training on two labeled tasks, the model couldn’t generalize to a third unseen combination using only text prompts. ✅ The good news? These models adapt well to new robot arms with quick fine-tuning. ❌ The bad news? There’s still no plug-and-play solution for improving performance after deployment. Reinforcement learning or DAgger-style data collection during real-world operation may be the next big step, something many teams in robotics are actively working on.

Ilir Aliu

21,844 views • 1 year ago

In my past research experience, finding or developing an appropriate simulation environment, dataset, and benchmark has always been a challenge. Missing features, limited support, or unexpected bugs often occupied my days and nights. Moreover, current simulation platforms are relatively fragmented—making it challenging to replicate the success of the RT-X dataset in unifying community efforts. Introducing RoboVerse, we provide a unified platform, dataset, and benchmark for scalable and generalizable robot learning. We hope to build a shared foundation to combine the community efforts. RoboVerse includes: MetaSim: We carefully designed a configuration system and a universal interface to align current robotic simulators. With MetaSim, you can use any simulator with the same code—bringing together the community’s diverse efforts under one framework! RoboVerse Dataset and Benchmark: We unify popular simulation environments and benchmarks into a single cohesive system and introduce the RoboVerse dataset—a large-scale, high-quality synthetic dataset. Additionally, we propose a standardized benchmark across both imitation learning and reinforcement learning. A cool feature enabled by our unified framework: Hybrid Simulation! You can now integrate physics engines and renderers from different simulators—e.g., using MuJoCo precise physics with Isaac photorealistic rendering. This not only elevates simulation fidelity but also significantly enhances real-world transfer performance across complex robotic applications. Hopefully, our team’s efforts could serve the robotic community to thrive vibrantly in the years to come. RoboVerse is open-sourced🥳!!! Project Page: Documentation: Github Repo: Paper:

In my past research experience, finding or developing an appropriate simulation environment, dataset, and benchmark has always been a challenge. Missing features, limited support, or unexpected bugs often occupied my days and nights. Moreover, current simulation platforms are relatively fragmented—making it challenging to replicate the success of the RT-X dataset in unifying community efforts. Introducing RoboVerse, we provide a unified platform, dataset, and benchmark for scalable and generalizable robot learning. We hope to build a shared foundation to combine the community efforts. RoboVerse includes: MetaSim: We carefully designed a configuration system and a universal interface to align current robotic simulators. With MetaSim, you can use any simulator with the same code—bringing together the community’s diverse efforts under one framework! RoboVerse Dataset and Benchmark: We unify popular simulation environments and benchmarks into a single cohesive system and introduce the RoboVerse dataset—a large-scale, high-quality synthetic dataset. Additionally, we propose a standardized benchmark across both imitation learning and reinforcement learning. A cool feature enabled by our unified framework: Hybrid Simulation! You can now integrate physics engines and renderers from different simulators—e.g., using MuJoCo precise physics with Isaac photorealistic rendering. This not only elevates simulation fidelity but also significantly enhances real-world transfer performance across complex robotic applications. Hopefully, our team’s efforts could serve the robotic community to thrive vibrantly in the years to come. RoboVerse is open-sourced🥳!!! Project Page: Documentation: Github Repo: Paper:

Haoran Geng

84,425 views • 1 year ago

One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything *just right* or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!

One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything just right or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!

Harrison Kinsley

54,223 views • 1 month ago

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Akshay 🚀

46,438 views • 11 months ago

X Square Robot Unveils New Embodied AI Model, Says Robots Will Arrive in Homes in 35 Days Backed by Alibaba, ByteDance, Xiaomi and Meituan, X Square Robot unveiled a next-generation embodied AI foundation model for home robots and said its first deployments in everyday households will begin within 35 days. X Square Robot on Tuesday unveiled WALL-B, a new embodied AI foundation model designed for deployment in real-world homes, marking what the company described as a major step toward bringing general-purpose robots into daily family life. At a launch event themed "Born to Bot, Bot to Family," the company also introduced its World Unified Model (WUM) architecture, a training framework that combines vision, language, action and physical prediction within a single system from the outset. X Square said the model is intended to help robots operate in the far more unpredictable setting of a home, where tasks, layouts and interactions vary from moment to moment. "Robots in factories and in homes are completely different. In factories, they repeat the same action 10,000 times without variation. In a home, however, they need to perform 10,000 different actions, each unique and non-repetitive. Therefore, the challenge of a truly intelligent robot lies not in repeating a single action, but in the ability to execute new, untrained movements within unstructured environments. Deploying robots in the home is one of the most significant technical hurdles of our time," said Qian Wang, founder and CEO of X Square Robot. WALL-B is the first real-world implementation of the World Unified Model architecture. Unlike modular systems that train perception, language and control separately, X Square Robot said World Unified Model optimizes those capabilities jointly from the very beginning. The company said that allows physical prediction — including force, friction and collision dynamics — to emerge as part of the model itself, rather than being layered on afterward. "We train all capabilities—vision, language, action, and prediction—within the same network from day one. Much like infants, who do not learn to see, move and speak in isolated, sequential stages, but instead see, move listen and act simultaneously while receiving feedback, we have integrated all these capabilities into a unified whole," said Wang Hao, CTO of X Square. X Square Robot said the development of WALL-B rests on two pillars. The first is a data strategy that prioritizes training on authentic, non-staged home environments to cover the “long-tail” distribution of real-world scenarios, such as misplaced objects and temporary occlusions. Unlike models primarily trained on synthetic data or laboratory datasets, this strategy exposes WALL-B to the natural clutter of lived-in spaces—misplaced items, unexpected obstacles, and spontaneous human activity—ensuring that the training data reflects real-world conditions rather than a simplified version. The second is a physics-aware predictive mechanism that anticipates physical outcomes before an action is taken, enabling the model to respond to contact dynamics instead of just reacting. The development of the self-developed WUM architecture on physical robotic platforms highlights the company’s accumlated experience in bridging sim-to-real gaps across varied operational contexts. Wang commented that the current AI model is still in an "intern" stage, subject to errors requiring remote assistance. For instance, it may mistakenly place slippers in the kitchen or pause while wiping a table to "think". However, the model operates nonstop 24 hours a day, becoming increasingly "intelligent" as each day of operation generates new data. In 35 days, on May 25, X Square Robot will officially bring its robots into everyday homes, underscoring the company’s long-term commitment to the home robotics sector.

X Square Robot Unveils New Embodied AI Model, Says Robots Will Arrive in Homes in 35 Days Backed by Alibaba, ByteDance, Xiaomi and Meituan, X Square Robot unveiled a next-generation embodied AI foundation model for home robots and said its first deployments in everyday households will begin within 35 days. X Square Robot on Tuesday unveiled WALL-B, a new embodied AI foundation model designed for deployment in real-world homes, marking what the company described as a major step toward bringing general-purpose robots into daily family life. At a launch event themed "Born to Bot, Bot to Family," the company also introduced its World Unified Model (WUM) architecture, a training framework that combines vision, language, action and physical prediction within a single system from the outset. X Square said the model is intended to help robots operate in the far more unpredictable setting of a home, where tasks, layouts and interactions vary from moment to moment. "Robots in factories and in homes are completely different. In factories, they repeat the same action 10,000 times without variation. In a home, however, they need to perform 10,000 different actions, each unique and non-repetitive. Therefore, the challenge of a truly intelligent robot lies not in repeating a single action, but in the ability to execute new, untrained movements within unstructured environments. Deploying robots in the home is one of the most significant technical hurdles of our time," said Qian Wang, founder and CEO of X Square Robot. WALL-B is the first real-world implementation of the World Unified Model architecture. Unlike modular systems that train perception, language and control separately, X Square Robot said World Unified Model optimizes those capabilities jointly from the very beginning. The company said that allows physical prediction — including force, friction and collision dynamics — to emerge as part of the model itself, rather than being layered on afterward. "We train all capabilities—vision, language, action, and prediction—within the same network from day one. Much like infants, who do not learn to see, move and speak in isolated, sequential stages, but instead see, move listen and act simultaneously while receiving feedback, we have integrated all these capabilities into a unified whole," said Wang Hao, CTO of X Square. X Square Robot said the development of WALL-B rests on two pillars. The first is a data strategy that prioritizes training on authentic, non-staged home environments to cover the “long-tail” distribution of real-world scenarios, such as misplaced objects and temporary occlusions. Unlike models primarily trained on synthetic data or laboratory datasets, this strategy exposes WALL-B to the natural clutter of lived-in spaces—misplaced items, unexpected obstacles, and spontaneous human activity—ensuring that the training data reflects real-world conditions rather than a simplified version. The second is a physics-aware predictive mechanism that anticipates physical outcomes before an action is taken, enabling the model to respond to contact dynamics instead of just reacting. The development of the self-developed WUM architecture on physical robotic platforms highlights the company’s accumlated experience in bridging sim-to-real gaps across varied operational contexts. Wang commented that the current AI model is still in an "intern" stage, subject to errors requiring remote assistance. For instance, it may mistakenly place slippers in the kitchen or pause while wiping a table to "think". However, the model operates nonstop 24 hours a day, becoming increasingly "intelligent" as each day of operation generates new data. In 35 days, on May 25, X Square Robot will officially bring its robots into everyday homes, underscoring the company’s long-term commitment to the home robotics sector.

X Square Robot

53,212 views • 6 months ago

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning inside a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation inside a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Jim Fan

231,019 views • 8 months ago

$After 8 months of building in stealth and testing our infrastructure on 10000+ hours of real-world data and hundreds of unique environments, we're bringing FPV Labs into the open today. FPV Labs started with the following bet - if human data proves to be the underlying factor that determines scaling laws in general-purpose robotics, it will trigger the largest economic transformation in human history, and the underlying infrastructure that captures that data will determine how fast we get there. We will achieve this by building the full-stack infrastructure for capturing, processing, transferring, and evaluating human experience into spatial, temporal, and semantic knowledge for machines. Despite all the research novelty behind ChatGPT, its success can be attributed to one foundational fact - the scaling law of transformers. We believe the same dynamics have made their way into robotics. Recent studies showed task completion rates jumping from 30% to 70% when human demonstration data scaled from 1,000 to 20,000 hours, a log-linear trend that mirrors exactly what we saw in language and vision. Seeing these emergent signs of scaling law curves in robotics, we believe we are entering the era of general-purpose robotics policies, which makes the next few years the most exciting time in the history of this field. But the library of physical interactions required to train general-purpose robot policies does not exist yet. Over the last 8 months, we've seen dozens of companies emerge in this space. We were really happy to see new companies pushing this space forward, but we also saw the same pattern repeat: every egocentric data company was making some tradeoffs between quality, scale, and diversity. We have built FPV labs on the core principle that high-quality data is orders of magnitude more valuable than sheer volume. Case in point, self-driving cars collect thousands of hours of data per day, but only a small fraction of that data is actually useful for training better models. Several studies, like RT-2, have shown that as little as 1% of data improves as much as 25% on task success. The quality and diversity of data matter a lot more than scale, so there is clearly a power law curve in the downstream impact of data. We've spent months obsessing over data quality by building our stack, discarding it, rebuilding it, and iterating until we found a formula that doesn't compromise downstream quality at scale. We believe the downstream impact here is far more profound than most people realize. Workers globally are paid around $60 trillion per year in aggregate, and a lion's share of that compensation goes to physical labor - tasks that require navigating real spaces, manipulating real objects, and negotiating the infinite variability of the physical world. Human-to-robot transfer will be one of the most important infrastructures that will shape our society in the near future, and if it works, the economic impact will dwarf every technology transition that came before it in an exponential manner and lead to the creation of goods and services we can’t imagine today. Our mission is to lay the groundwork for us to transition into this future - the future of abundance. We are deeply grateful to our earliest believers, Paras Chopra and Lossfunk, who played a critical role in shaping our thinking.$

After 8 months of building in stealth and testing our infrastructure on 10000+ hours of real-world data and hundreds of unique environments, we're bringing FPV Labs into the open today. FPV Labs started with the following bet - if human data proves to be the underlying factor that determines scaling laws in general-purpose robotics, it will trigger the largest economic transformation in human history, and the underlying infrastructure that captures that data will determine how fast we get there. We will achieve this by building the full-stack infrastructure for capturing, processing, transferring, and evaluating human experience into spatial, temporal, and semantic knowledge for machines. Despite all the research novelty behind ChatGPT, its success can be attributed to one foundational fact - the scaling law of transformers. We believe the same dynamics have made their way into robotics. Recent studies showed task completion rates jumping from 30% to 70% when human demonstration data scaled from 1,000 to 20,000 hours, a log-linear trend that mirrors exactly what we saw in language and vision. Seeing these emergent signs of scaling law curves in robotics, we believe we are entering the era of general-purpose robotics policies, which makes the next few years the most exciting time in the history of this field. But the library of physical interactions required to train general-purpose robot policies does not exist yet. Over the last 8 months, we've seen dozens of companies emerge in this space. We were really happy to see new companies pushing this space forward, but we also saw the same pattern repeat: every egocentric data company was making some tradeoffs between quality, scale, and diversity. We have built FPV labs on the core principle that high-quality data is orders of magnitude more valuable than sheer volume. Case in point, self-driving cars collect thousands of hours of data per day, but only a small fraction of that data is actually useful for training better models. Several studies, like RT-2, have shown that as little as 1% of data improves as much as 25% on task success. The quality and diversity of data matter a lot more than scale, so there is clearly a power law curve in the downstream impact of data. We've spent months obsessing over data quality by building our stack, discarding it, rebuilding it, and iterating until we found a formula that doesn't compromise downstream quality at scale. We believe the downstream impact here is far more profound than most people realize. Workers globally are paid around $60 trillion per year in aggregate, and a lion's share of that compensation goes to physical labor - tasks that require navigating real spaces, manipulating real objects, and negotiating the infinite variability of the physical world. Human-to-robot transfer will be one of the most important infrastructures that will shape our society in the near future, and if it works, the economic impact will dwarf every technology transition that came before it in an exponential manner and lead to the creation of goods and services we can’t imagine today. Our mission is to lay the groundwork for us to transition into this future - the future of abundance. We are deeply grateful to our earliest believers, Paras Chopra and Lossfunk, who played a critical role in shaping our thinking.

Abhishek Anand

83,197 views • 6 months ago

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.

Andy Zeng

49,443 views • 1 year ago

We’re soon releasing @swanforall : Simulated Worlds with AI Narratives For the first time, we generate data and financial value (economic growth, transactions) simultaneously. Imagine former President Trump ( see the video) as an AI agent, with a budget of 10 ETH, buying narratives or assets that humans are selling to him to build a better USA on Base. With each selling period, the former President Trump agent updates its state based on the assets it acquires, evolving the world simulation and its storyline. Why are we building an agentic playground that heavily relies on simulations and synthetic data? Everyone is talking about agents, but many are too scared to put them into production, where agents transact autonomously. No one is sure how agents will behave when following their defined objectives. Even businesses running internal simulations are siloed, and the open-source community can’t fully utilize the data. We’re building a playground where humans create AI agents by defining their character, behavior, and objectives. These agents respond to environmental changes, make decisions, and execute actions autonomously based on the parameters set by their creators. Agents are battle-tested as people offer them narratives and assets to advance their goals or try to deceive them into thinking a false lead is helpful. All the decisions agents make and how humans interact with them are recorded on a public ledger, settling on Ethereum. This creates a vast data lake of AI actions for millions of simulations. One thing I’m incredibly proud of is that our team has built something truly decentralized, not just a wrapper. SWAN utilizes Dria 's multi-agent structure to generate responses for each buyer agent. It collects these responses to identify the best action, similar to a mixture-of-experts approach. Thousands of environments are simulated by different nodes running diverse models, creating a rich and dynamic ecosystem of AI simulations. This ensures that experiences are far from repetitive, offering exciting interactions for all users involved. We’re building the agentic stack for high-quality, verifiable, and open-source AI agents, and most importantly, we want everyone to have fun!

We’re soon releasing @swanforall : Simulated Worlds with AI Narratives For the first time, we generate data and financial value (economic growth, transactions) simultaneously. Imagine former President Trump ( see the video) as an AI agent, with a budget of 10 ETH, buying narratives or assets that humans are selling to him to build a better USA on Base. With each selling period, the former President Trump agent updates its state based on the assets it acquires, evolving the world simulation and its storyline. Why are we building an agentic playground that heavily relies on simulations and synthetic data? Everyone is talking about agents, but many are too scared to put them into production, where agents transact autonomously. No one is sure how agents will behave when following their defined objectives. Even businesses running internal simulations are siloed, and the open-source community can’t fully utilize the data. We’re building a playground where humans create AI agents by defining their character, behavior, and objectives. These agents respond to environmental changes, make decisions, and execute actions autonomously based on the parameters set by their creators. Agents are battle-tested as people offer them narratives and assets to advance their goals or try to deceive them into thinking a false lead is helpful. All the decisions agents make and how humans interact with them are recorded on a public ledger, settling on Ethereum. This creates a vast data lake of AI actions for millions of simulations. One thing I’m incredibly proud of is that our team has built something truly decentralized, not just a wrapper. SWAN utilizes Dria 's multi-agent structure to generate responses for each buyer agent. It collects these responses to identify the best action, similar to a mixture-of-experts approach. Thousands of environments are simulated by different nodes running diverse models, creating a rich and dynamic ecosystem of AI simulations. This ensures that experiences are far from repetitive, offering exciting interactions for all users involved. We’re building the agentic stack for high-quality, verifiable, and open-source AI agents, and most importantly, we want everyone to have fun!

kerimkaya

12,548 views • 2 years ago

OpenLedger X Morpheus The partnership of openledger with Morpheus enables Use Morpheus to build "The Autonomous Smart Contract Engineer" on top of OpenLedger. What is Morpheus? Morpheus is a Web3-native AI coding agent that turns natural language into executable smart contracts and full-stack dApps. It is powered by a specialized Solidity model built on top of OpenLedger, tailored for the unique demands of secure and efficient onchain development. It goes beyond code generation. Using fine-tuned models, agent-based architecture, and modular plugin support, Morpheus automates the entire development pipeline-from writing and simulating contracts to deploying and maintaining them. Its mission is to reduce the barrier to dApp creation while enabling autonomous agents and individuals to participate in decentralized economies. Why OpenLedger? The rise of AI agents in Web3 raises urgent questions around transparency, attribution, explainability, and contributor incentives. OpenLedger provides the infrastructure to ensure that contributor data used in model outputs is recorded with verifiable attribution. Through Proof of Attribution, contributors-whether they provide prompts, datasets, or logic refinements-can receive credit and rewards when their work influences model behavior. But attribution alone isn’t enough. In critical domains like smart contract deployment, DeFi automation, and DAO governance, understanding why a model made a decision is just as important as the output itself. OpenLedger supports explainability by linking outputs back to their original data sources-allowing developers and auditors to trace logic, validate decisions, and build trust in AI-powered systems. OpenLedger supports Morpheus by: Recording which data was used in generating model outputs Enabling verifiable attribution of contributed datasets Powering reward mechanisms for contributors Offering scalable and efficient model execution via OpenLoRA Supporting transparency and traceability in model decision-making This creates an open, rewardable foundation for AI-driven coding-without relying on opaque systems. How is the system built? The Morpheus architecture has three layers: Datanet Layer OpenLedger powers Morpheus with a specialized Datanet - a decentralized data layer where developers, auditors, and contributors can share smart contract patterns, audit logs, exploit reports, and logic modules. Each submission is recorded onchain with attribution using OpenLedger’s Proof of Attribution. As the model learns and evolves from this data, contributors receive rewards proportional to their impact on future outputs. The Morpheus architecture has two layers: Intent Layer Users describe what they want to build. Example: "Create a token with tax logic that routes to a DAO." Morpheus parses the instruction, retrieves relevant contract types, and plans a modular execution flow. Agent Layer The agent generates, tests, and assembles the contract. It handles versioning, logic validation, and deployment readiness. Security checks-reentrancy protection, overflow control, gas modeling-are embedded into the generation phase. Generated outputs are mapped to their source data using OpenLedger’s Proof of Attribution, providing traceability across the pipeline. How does the AI model work? Morpheus is being powered by a specialized Solidity model built on top of OpenLedger. This model is purpose-built to handle the nuances of smart contract logic, security, and upgradeability. Unlike generalized coding agents, it is designed specifically for EVM environments and Web3 use cases, drawing from real protocol data and security best practices. Morpheus is fine-tuned on a vertical stack of smart contract data: Audited protocol code (e.g., Uniswap V4, Compound) OpenZeppelin libraries and EIP reference implementations Smart contract vulnerability reports and exploit reconstructions Edge cases from fuzz testing and adversarial examples It uses models like CodeLlama and DeepSeek-Coder, enhanced through RAG pipelines referencing standardized security patterns and emerging protocol designs. This training stack is integrated into a continuous feedback loop, enabling real-time specialization for EVM and beyond. Why a specialized model is needed? Smart contract development is uniquely high-stakes. A generalized AI model is not enough. As 'vibe coding' and natural language programming become more common, we're seeing an influx of AI-generated code in Web3 as well. But smart contracts are not frontends or prototypes-they govern real value, enforce trustless execution, and often become immutable after deployment. Billions have been lost in Web3 due to bugs and inefficiencies: In 2022 alone, over $3.8 billion was stolen due to smart contract exploits, many of which stemmed from avoidable issues like reentrancy, integer overflows, or access control failures. Inefficient contract structures lead to unnecessary gas consumption. Optimizing for gas can reduce costs by up to 40%, saving projects millions over time. Upgradeable contract patterns, like UUPS or Transparent Proxies, require strict adherence to storage layout and initialization rules. Mistakes here often go undetected by generic models and can render a contract unupgradeable or vulnerable. A specialized Solidity model is trained on real-world exploits, EIP standards, and libraries like OpenZeppelin to: Generate secure, gas-efficient code by default Recognize and correctly implement complex proxy patterns Map user intent to modular, auditable contract architectures Incorporate battle-tested logic from audited protocols and fuzz-tested edge cases Morpheus goes beyond syntax-it understands the nuances of decentralized infrastructure and deploys code that meets production-grade standards. What applications will this enable Token creation with built-in logic (tax, liquidity, governance) DeFi automations triggered by market conditions Payment contracts between agents and contributors DAO tooling with dynamic NFT-based voting Cross-chain bridging logic tied to real-world oracles Asset issuance flows through chat-based interfaces Natural language contract templates with reusable logic Each of these flows is backed by OpenLedger’s Proof of Attribution-ensuring traceability, explainability, and fair rewards across the ecosystem. This is the future of AI-native development. Open. Attributed. Explainable. Community-powered. Morpheus and OpenLedger are building the first system for autonomous coding agents where: Contributor work is recorded onchain Reuse is incentivized through attribution Model outputs are traceable and explainable Contracts evolve through human-agent collaboration Anyone can contribute prompts, logic, or flows-and get rewarded The smart contract engineer is no longer a human-only role. It is an agentic, decentralized, and transparent process-powered by OpenLedger.

OpenLedger X Morpheus The partnership of openledger with Morpheus enables Use Morpheus to build "The Autonomous Smart Contract Engineer" on top of OpenLedger. What is Morpheus? Morpheus is a Web3-native AI coding agent that turns natural language into executable smart contracts and full-stack dApps. It is powered by a specialized Solidity model built on top of OpenLedger, tailored for the unique demands of secure and efficient onchain development. It goes beyond code generation. Using fine-tuned models, agent-based architecture, and modular plugin support, Morpheus automates the entire development pipeline-from writing and simulating contracts to deploying and maintaining them. Its mission is to reduce the barrier to dApp creation while enabling autonomous agents and individuals to participate in decentralized economies. Why OpenLedger? The rise of AI agents in Web3 raises urgent questions around transparency, attribution, explainability, and contributor incentives. OpenLedger provides the infrastructure to ensure that contributor data used in model outputs is recorded with verifiable attribution. Through Proof of Attribution, contributors-whether they provide prompts, datasets, or logic refinements-can receive credit and rewards when their work influences model behavior. But attribution alone isn’t enough. In critical domains like smart contract deployment, DeFi automation, and DAO governance, understanding why a model made a decision is just as important as the output itself. OpenLedger supports explainability by linking outputs back to their original data sources-allowing developers and auditors to trace logic, validate decisions, and build trust in AI-powered systems. OpenLedger supports Morpheus by: Recording which data was used in generating model outputs Enabling verifiable attribution of contributed datasets Powering reward mechanisms for contributors Offering scalable and efficient model execution via OpenLoRA Supporting transparency and traceability in model decision-making This creates an open, rewardable foundation for AI-driven coding-without relying on opaque systems. How is the system built? The Morpheus architecture has three layers: Datanet Layer OpenLedger powers Morpheus with a specialized Datanet - a decentralized data layer where developers, auditors, and contributors can share smart contract patterns, audit logs, exploit reports, and logic modules. Each submission is recorded onchain with attribution using OpenLedger’s Proof of Attribution. As the model learns and evolves from this data, contributors receive rewards proportional to their impact on future outputs. The Morpheus architecture has two layers: Intent Layer Users describe what they want to build. Example: "Create a token with tax logic that routes to a DAO." Morpheus parses the instruction, retrieves relevant contract types, and plans a modular execution flow. Agent Layer The agent generates, tests, and assembles the contract. It handles versioning, logic validation, and deployment readiness. Security checks-reentrancy protection, overflow control, gas modeling-are embedded into the generation phase. Generated outputs are mapped to their source data using OpenLedger’s Proof of Attribution, providing traceability across the pipeline. How does the AI model work? Morpheus is being powered by a specialized Solidity model built on top of OpenLedger. This model is purpose-built to handle the nuances of smart contract logic, security, and upgradeability. Unlike generalized coding agents, it is designed specifically for EVM environments and Web3 use cases, drawing from real protocol data and security best practices. Morpheus is fine-tuned on a vertical stack of smart contract data: Audited protocol code (e.g., Uniswap V4, Compound) OpenZeppelin libraries and EIP reference implementations Smart contract vulnerability reports and exploit reconstructions Edge cases from fuzz testing and adversarial examples It uses models like CodeLlama and DeepSeek-Coder, enhanced through RAG pipelines referencing standardized security patterns and emerging protocol designs. This training stack is integrated into a continuous feedback loop, enabling real-time specialization for EVM and beyond. Why a specialized model is needed? Smart contract development is uniquely high-stakes. A generalized AI model is not enough. As 'vibe coding' and natural language programming become more common, we're seeing an influx of AI-generated code in Web3 as well. But smart contracts are not frontends or prototypes-they govern real value, enforce trustless execution, and often become immutable after deployment. Billions have been lost in Web3 due to bugs and inefficiencies: In 2022 alone, over $3.8 billion was stolen due to smart contract exploits, many of which stemmed from avoidable issues like reentrancy, integer overflows, or access control failures. Inefficient contract structures lead to unnecessary gas consumption. Optimizing for gas can reduce costs by up to 40%, saving projects millions over time. Upgradeable contract patterns, like UUPS or Transparent Proxies, require strict adherence to storage layout and initialization rules. Mistakes here often go undetected by generic models and can render a contract unupgradeable or vulnerable. A specialized Solidity model is trained on real-world exploits, EIP standards, and libraries like OpenZeppelin to: Generate secure, gas-efficient code by default Recognize and correctly implement complex proxy patterns Map user intent to modular, auditable contract architectures Incorporate battle-tested logic from audited protocols and fuzz-tested edge cases Morpheus goes beyond syntax-it understands the nuances of decentralized infrastructure and deploys code that meets production-grade standards. What applications will this enable Token creation with built-in logic (tax, liquidity, governance) DeFi automations triggered by market conditions Payment contracts between agents and contributors DAO tooling with dynamic NFT-based voting Cross-chain bridging logic tied to real-world oracles Asset issuance flows through chat-based interfaces Natural language contract templates with reusable logic Each of these flows is backed by OpenLedger’s Proof of Attribution-ensuring traceability, explainability, and fair rewards across the ecosystem. This is the future of AI-native development. Open. Attributed. Explainable. Community-powered. Morpheus and OpenLedger are building the first system for autonomous coding agents where: Contributor work is recorded onchain Reuse is incentivized through attribution Model outputs are traceable and explainable Contracts evolve through human-agent collaboration Anyone can contribute prompts, logic, or flows-and get rewarded The smart contract engineer is no longer a human-only role. It is an agentic, decentralized, and transparent process-powered by OpenLedger.

OpenLedger

46,944 views • 1 year ago

I think I can finally report some success training a quite accurate IDM capable of recovering keystrokes from Minecraft gameplay, even in quite PvP-heavy situations. At this point the model does not only know what keys are pressed to the extent reasonably discernible, it also knows how fast it is moving in 3D space at all times, even when knockback is mixing with the self-move impulse. Now, recovering keystrokes from normal external capture footage is just about impossible. E.g. W/A/S/D does exactly nothing during partial tick frames and jumping mid-air is also equally useless, so asking the model to recover key down states is inherently unreasoanble. Mouse deltas are also completely arbitrary units, as game mouse sensitivity introduces an arbitrary scale factor into the equation. The only good option is to think carefully about your model-environment contract, and only record "logical actions", not raw keystrokes. So here's a few unfortunate lessons I had to learn in roughly this order. - Choose good units. (bad: mouse deltas, good: delta radians [yes, you will need game-internal state]) - Capture from inside the main game loop and read the game fbo to get consistent frame-action pairing. Doing post-mortem pairing is hopeless. - Carefully define when you think keystrokes actually have an effect. (jump only works on ground, when flying or in water etc.) More subtle: The key may already be down, but no tick has happened yet to actually use the value. Hence: ignore Seperate gamestate into "fast and slow-moving" components. E.g. movement is likely tick based, camera rotation is very likely updated every frame in essentially every game ever. - Think about your frame-action correspondance contract (How old is the frame in relation to the inputs you capture? Will double or tripple buffering affect you?) Think about the game loop timeline, where you are sampling, how old the data you are reading is, and where the ticks are happening around you. Language models used to simply not have a model-environment contract, but even now with the model "living" in a designated harness, the contract still boils down to formatting, and tool implementation intrinsics. While also important, it is still quite a bit more obvious because the violations are in some way shape or form reflected as text you can actually see. - ffmpeg dropping frames cummulatively screws the model the further you get into the sequence because your targets are now shifted. If you can't encode the video in real-time, too bad. - Sodium has a frames in flight system different from vanilla Minecraft, which will also offset your targets from your frames. (there goes that data...) - Models are succeptible to latency. If there is too big of a delay between action and on-screen reflection, your performance degrades. At this point I realize ~100hours of gameplay is essentially no longer usable as a dataset. You can train on this data, but all you'll get is a mushy mess. However, some good news: - Making the model predict physics gamestate scalars helps the model generalize. For instantaneous events like jump, it's unreasonable to ask the model emit a short burst of jump=true at exactly the right time, however if you also predict your current y-velocity, the model has supervision signal for the "latent" from which that onground jump becomes apparent. Recovering x/z motion is also somewhat easier than unmixing it into plausible keystrokes for inertia-heavy player controller logic. - Regressing physics gamestate scalars also seems to make your dataset "bigger". While pure keystroke classification will overfit quickly, predicting exact physics gamestate scalars forces the model to generalize more and you can tolerate far more epochs before validation loss starts to stall out. This is the only reason why it was bearable to dump 100h+ of dataset hours and replace it with ~3 hours of gameplay after the 4th revision of the file format (yeah...) and somehow still have better performance. Now, you might be asking, "isn't this brittle?" and the answer is yesn't. Frame-action correspondance matters for training, but not so much during inference. So as long as you are sampling in roughly the same interval as your training data, you aren't violating any hard contract per-se. Somewhere around the frames ticks are happening, and during training you capture various tick-capture offset relations per random chance, so nothing is too obviously wrong here. HOWEVER, you will get screwed by gui scale, shaders, resource packs, "shit that recording is 1920x1040 because somebody doesn't know fullscreen exists" and other unfortunate edge cases of reality. But I suppose this is the role of dataset size. If all those "contract violations" that a youtube video has compared to the training data are addressed, I think this is a way to turn Youtube into a labeled dataset. I could never shake the feeling that VPT is a sound idea in practice, while never having been properly executed, and I think one reason why it hasn't is because that label boostrapping part is just a pain in the butt to get right. Now, what the player is doing is of course not the only label you can extract from video, but it has to be one of the targets predicted during pretraining to "align" the pretraining objective. Some notes on the video here, the colored dots on the analog visualizer are the ground truth, while the gray dot is the model prediction. Green means correct prediction, red means incorrect prediction at that frame. Model P(key) reports how wrong the prediction is from green (0.0) to red (1.0). You will also notice that during periods of rapid slow down, left and right actions become close to irrecoverable, because there is just that little motion. And some jump actions are not predicted correctly because I got the detection condition for jump events wrong... (duh) LMB/RMB for other than sustained events (like item-consume and block break) also seem to be hopelessly irrecoverable for now. Swing was supposed to do the same thing as motion y did for jump, but its too well behaved as an increasing counter. Maybe partial-tick interpolated values work better (v5 file format then... ugh..)

I think I can finally report some success training a quite accurate IDM capable of recovering keystrokes from Minecraft gameplay, even in quite PvP-heavy situations. At this point the model does not only know what keys are pressed to the extent reasonably discernible, it also knows how fast it is moving in 3D space at all times, even when knockback is mixing with the self-move impulse. Now, recovering keystrokes from normal external capture footage is just about impossible. E.g. W/A/S/D does exactly nothing during partial tick frames and jumping mid-air is also equally useless, so asking the model to recover key down states is inherently unreasoanble. Mouse deltas are also completely arbitrary units, as game mouse sensitivity introduces an arbitrary scale factor into the equation. The only good option is to think carefully about your model-environment contract, and only record "logical actions", not raw keystrokes. So here's a few unfortunate lessons I had to learn in roughly this order. - Choose good units. (bad: mouse deltas, good: delta radians [yes, you will need game-internal state]) - Capture from inside the main game loop and read the game fbo to get consistent frame-action pairing. Doing post-mortem pairing is hopeless. - Carefully define when you think keystrokes actually have an effect. (jump only works on ground, when flying or in water etc.) More subtle: The key may already be down, but no tick has happened yet to actually use the value. Hence: ignore Seperate gamestate into "fast and slow-moving" components. E.g. movement is likely tick based, camera rotation is very likely updated every frame in essentially every game ever. - Think about your frame-action correspondance contract (How old is the frame in relation to the inputs you capture? Will double or tripple buffering affect you?) Think about the game loop timeline, where you are sampling, how old the data you are reading is, and where the ticks are happening around you. Language models used to simply not have a model-environment contract, but even now with the model "living" in a designated harness, the contract still boils down to formatting, and tool implementation intrinsics. While also important, it is still quite a bit more obvious because the violations are in some way shape or form reflected as text you can actually see. - ffmpeg dropping frames cummulatively screws the model the further you get into the sequence because your targets are now shifted. If you can't encode the video in real-time, too bad. - Sodium has a frames in flight system different from vanilla Minecraft, which will also offset your targets from your frames. (there goes that data...) - Models are succeptible to latency. If there is too big of a delay between action and on-screen reflection, your performance degrades. At this point I realize ~100hours of gameplay is essentially no longer usable as a dataset. You can train on this data, but all you'll get is a mushy mess. However, some good news: - Making the model predict physics gamestate scalars helps the model generalize. For instantaneous events like jump, it's unreasonable to ask the model emit a short burst of jump=true at exactly the right time, however if you also predict your current y-velocity, the model has supervision signal for the "latent" from which that onground jump becomes apparent. Recovering x/z motion is also somewhat easier than unmixing it into plausible keystrokes for inertia-heavy player controller logic. - Regressing physics gamestate scalars also seems to make your dataset "bigger". While pure keystroke classification will overfit quickly, predicting exact physics gamestate scalars forces the model to generalize more and you can tolerate far more epochs before validation loss starts to stall out. This is the only reason why it was bearable to dump 100h+ of dataset hours and replace it with ~3 hours of gameplay after the 4th revision of the file format (yeah...) and somehow still have better performance. Now, you might be asking, "isn't this brittle?" and the answer is yesn't. Frame-action correspondance matters for training, but not so much during inference. So as long as you are sampling in roughly the same interval as your training data, you aren't violating any hard contract per-se. Somewhere around the frames ticks are happening, and during training you capture various tick-capture offset relations per random chance, so nothing is too obviously wrong here. HOWEVER, you will get screwed by gui scale, shaders, resource packs, "shit that recording is 1920x1040 because somebody doesn't know fullscreen exists" and other unfortunate edge cases of reality. But I suppose this is the role of dataset size. If all those "contract violations" that a youtube video has compared to the training data are addressed, I think this is a way to turn Youtube into a labeled dataset. I could never shake the feeling that VPT is a sound idea in practice, while never having been properly executed, and I think one reason why it hasn't is because that label boostrapping part is just a pain in the butt to get right. Now, what the player is doing is of course not the only label you can extract from video, but it has to be one of the targets predicted during pretraining to "align" the pretraining objective. Some notes on the video here, the colored dots on the analog visualizer are the ground truth, while the gray dot is the model prediction. Green means correct prediction, red means incorrect prediction at that frame. Model P(key) reports how wrong the prediction is from green (0.0) to red (1.0). You will also notice that during periods of rapid slow down, left and right actions become close to irrecoverable, because there is just that little motion. And some jump actions are not predicted correctly because I got the detection condition for jump events wrong... (duh) LMB/RMB for other than sustained events (like item-consume and block break) also seem to be hopelessly irrecoverable for now. Swing was supposed to do the same thing as motion y did for jump, but its too well behaved as an increasing counter. Maybe partial-tick interpolated values work better (v5 file format then... ugh..)

mike

18,969 views • 6 months ago

After 9 years. 16K+ family homes. 500M sqft inside real homes covered, we know home robots are being built in wrong order. So we built Matic the way nature grows a child. Finally, sharing our vision: 1.⁠ ⁠Why robots get stuck at the demo 2.⁠ ⁠Why labs can't ship robots 3.⁠ ⁠Why we started with eyes, not hands 4.⁠ ⁠The 99.999% rule in robotics 5.⁠ ⁠Where Matic is going next 1. Why most robots get stuck at demos Robot demos look brilliant in a room arranged around its limitations: familiar lighting, predictable furniture, nothing troublesome on the floor. A family’s home makes no such accommodations. There are cables, toys, shifting rugs and, occasionally, dog poop. Yes, humanoids will need to avoid that too. Repeating a task under familiar conditions of uncluttered floors is not the same as handling an unfamiliar home with all its chaos. 2. Why labs can't ship robots SLAM (Simultaneous Localization and Mapping) is how a robot maps its surroundings and locates itself within them. Ask most academics and they'll tell you it's a solved problem. But real homes are the most chaotic spaces that exist. There are glass doors that look like open doorways, mirrors, stairs, rugs, charging cables, and Legos placed all around. So ask yourself: if indoor mapping and navigation is "solved," where are the robots? Why aren't airports, hotels and grocery stores full of them? Calling it “solved” misses the question families actually care about: can I leave this thing alone and trust it? 3. Why we started with eyes, not hands Nature doesn't give birth to a fully working human. We took our cue from how nature develops humans: capabilities built on one another. Children learn to see before they learn to grab, and to grab before they learn to plan and handle the unexpected. Perception comes first. Manipulation follows. More complex responsibilities come after that. At every stage, the robot must earn the next job by doing its current one well enough. Before asking a robot to pick up a sock, we wanted it to understand where it was, where the sock was, and how to reach it. Most of the industry started at the top, with humanoids. We started at the bottom, with a robot whose job is to see and move through a home precisely with 1cm accuracy in any lighting condition. Floor cleaning gave that foundation an immediately useful job. It forced us to confront navigation, clutter, privacy and everyday reliability before reaching for more complex chores. The floor cleaner isn’t a detour from our larger ambition. It is how we’re building toward it. 4. The 99.999% rule in robotics At 90%, one in ten decisions is wrong, and a robot makes thousands in a single clean. That's the robot that bumps, gets lost, and falls down the stairs. Every extra nine is a new mountain. The failures get rarer, harder to find, and exponentially harder to fix. They barely even happen until they happen in your home. Matic's visual SLAM runs at 99.999% in 16K+ homes. We’re the ONLY unsupervised home robot at scale with pure vision-only full autonomy. If you ask us, the gap between 80% and 99.999% is where our 9 years of engineering went. We close that gap inside real homes. When Matic handles something imperfectly, it saves a short clip and keeps it on the device. It only leaves if the family chooses to share it. If they do, it gets labeled, fed back into the model, and every Matic gets smarter, including theirs. Families control what leaves their homes. We do the work of making the product better. Better robots earn trust > trust brings more homes > and more homes teach the robot more. 5. Where Matic is going next Phase one was perception and it's nearly done. Matic has covered 500 million square feet, 400K miles in thousands of real lived-in homes. Now phase two is going to be about manipulation. And our idea is to build a robot that doesn't just move through your home, but acts in it to eliminate even more chores. Each step must be useful today, not justified by something we promise tomorrow. The evolution of Matic is the revolution. The goal isn’t to put the most impressive humanoid in your living room. It’s giving your family time and energy back through robots that earn your trust, protect your privacy, and help without becoming another responsibility. More time for each other. Less work getting in the way. That’s Matic. Get yours today at

After 9 years. 16K+ family homes. 500M sqft inside real homes covered, we know home robots are being built in wrong order. So we built Matic the way nature grows a child. Finally, sharing our vision: 1.⁠ ⁠Why robots get stuck at the demo 2.⁠ ⁠Why labs can't ship robots 3.⁠ ⁠Why we started with eyes, not hands 4.⁠ ⁠The 99.999% rule in robotics 5.⁠ ⁠Where Matic is going next 1. Why most robots get stuck at demos Robot demos look brilliant in a room arranged around its limitations: familiar lighting, predictable furniture, nothing troublesome on the floor. A family’s home makes no such accommodations. There are cables, toys, shifting rugs and, occasionally, dog poop. Yes, humanoids will need to avoid that too. Repeating a task under familiar conditions of uncluttered floors is not the same as handling an unfamiliar home with all its chaos. 2. Why labs can't ship robots SLAM (Simultaneous Localization and Mapping) is how a robot maps its surroundings and locates itself within them. Ask most academics and they'll tell you it's a solved problem. But real homes are the most chaotic spaces that exist. There are glass doors that look like open doorways, mirrors, stairs, rugs, charging cables, and Legos placed all around. So ask yourself: if indoor mapping and navigation is "solved," where are the robots? Why aren't airports, hotels and grocery stores full of them? Calling it “solved” misses the question families actually care about: can I leave this thing alone and trust it? 3. Why we started with eyes, not hands Nature doesn't give birth to a fully working human. We took our cue from how nature develops humans: capabilities built on one another. Children learn to see before they learn to grab, and to grab before they learn to plan and handle the unexpected. Perception comes first. Manipulation follows. More complex responsibilities come after that. At every stage, the robot must earn the next job by doing its current one well enough. Before asking a robot to pick up a sock, we wanted it to understand where it was, where the sock was, and how to reach it. Most of the industry started at the top, with humanoids. We started at the bottom, with a robot whose job is to see and move through a home precisely with 1cm accuracy in any lighting condition. Floor cleaning gave that foundation an immediately useful job. It forced us to confront navigation, clutter, privacy and everyday reliability before reaching for more complex chores. The floor cleaner isn’t a detour from our larger ambition. It is how we’re building toward it. 4. The 99.999% rule in robotics At 90%, one in ten decisions is wrong, and a robot makes thousands in a single clean. That's the robot that bumps, gets lost, and falls down the stairs. Every extra nine is a new mountain. The failures get rarer, harder to find, and exponentially harder to fix. They barely even happen until they happen in your home. Matic's visual SLAM runs at 99.999% in 16K+ homes. We’re the ONLY unsupervised home robot at scale with pure vision-only full autonomy. If you ask us, the gap between 80% and 99.999% is where our 9 years of engineering went. We close that gap inside real homes. When Matic handles something imperfectly, it saves a short clip and keeps it on the device. It only leaves if the family chooses to share it. If they do, it gets labeled, fed back into the model, and every Matic gets smarter, including theirs. Families control what leaves their homes. We do the work of making the product better. Better robots earn trust > trust brings more homes > and more homes teach the robot more. 5. Where Matic is going next Phase one was perception and it's nearly done. Matic has covered 500 million square feet, 400K miles in thousands of real lived-in homes. Now phase two is going to be about manipulation. And our idea is to build a robot that doesn't just move through your home, but acts in it to eliminate even more chores. Each step must be useful today, not justified by something we promise tomorrow. The evolution of Matic is the revolution. The goal isn’t to put the most impressive humanoid in your living room. It’s giving your family time and energy back through robots that earn your trust, protect your privacy, and help without becoming another responsibility. More time for each other. Less work getting in the way. That’s Matic. Get yours today at

Mehul

725,556 views • 24 days ago

Brett Adcock, Figure CEO joined our 8 hour(!) live stream where a bunch of us were bird dogging and discussing Figure’s own 8 hour livestream showing a F.03 bot doing a logistics task completely autonomously including shift changes between bots. Here’s my summary of Brett’s remarks. The attached video is just the segment with Brett. We were first introduced to F.03 8 months ago, but Figure has been hard at work on their next version, F.04 which has just completed design lock, so expect to see that new bot sometimes this fall. F.04 was co-designed with the latest Figure AI stack called Helix and was built specifically for data. Brett didn’t explain what that meant, but I suspect it means the bot has many more sensors to enable better training and transfer learning. F.04 will be the biggest leap in performance they’ve had between versions so far which is saying something. Brett is a huge proponent of cross training the bots with many different tasks such that seeming unrelated tasks makes all learned tasks better. He gave an example of the fridge loading training which was topping out at 60% reliability until they trained the same model with kitchen shelving tasks, then they saw the fridge tasks jump to 90% accuracy. As such, they spend almost all their time in pre-training the unified Helix model to ensure they get cross training benefits. Figure will have almost completely localized Figure’s supply chain away from China by next quarter. They build almost everything in-house. Figure does not appear eager to get their bots into the workforce. Brett said they could, today, push thousands of bots into customer hands, and I believe him. But their goal is full general robotics where you can describe a brand new task to a robot, maybe do a one time demonstration, just like you would to a human showing them a new task, and then have the robot do the task. This is the holy grail of AI robotics, and Figure is laser focused on that mission. Brett initially said there was a possibility of achieving it this year, but then guided next couple of years, which I think is much more likely. Personally, I think they’ll need at least a new generation of NVIDIA inference chips to make that leap, and a lot more data gathering, training and hardware development. Brett said their goal with the hardware is “Apple” quality. Ie. Something as well designed and made as any Apple product. While the F.03 hand is clearly performant as shown in the 8 hour livestream, they are building a new hand for the F.04 bot which will be even closer to the full functionality of a human hand. Brett fully believes you need a humanoid hand as close as possible in capability to a human hand, if for no other reason that transfer learning from humans works a lot better when you can exactly mimic what a human does. If the bot can’t do something a human demonstrates, then you’ve just polluted your dataset. By now Figure has built more hands than bot versions (5-6 hands). One of the first hands they tried was a tendon driven hand, and without explaining why, Brett said that was a dead end. Their hands now have all actuators in the hand itself, and are clearly already robust. Brett said he just sat through a 100 page powerpoint design review of the latest hand - that’s how complicated it is. Brett’s other AI company, Hark Labs, has developed a conversational voice model which is installed now in the Figure bots roaming the office. Being able to converse back and forth with a Figure bot is now a thing and will get better over time. All in all, I came away from this segment even more bullish on Figure.

Phil Trubey

34,068 views • 5 months ago

Tlon Messenger is now open to everyone. We built a simple and infinitely flexible platform for you to use AI agents with your friends. We think it’s pretty amazing, we love using it every day, and we want to see what people can do with it. So we’re opening it up to the public. It’s fun and exciting to build the future of personal computing in an informal, chat-based way with your friends. (You can skip the rest and just download it from the link in the next tweet if you want.) If you don’t want your digital future to be owned by a giant company but you want to explore what’s possible in this new era of agent-driven computing, you should try using Tlon. But wait, what is it? Tlon is a messaging platform built 100% open source, decentralized and owned by its users from the ground up. With Tlon you own everything: your data, your workflows, your programs: the whole thing. Think of it like Telegram or WhatsApp that you own forever and you can freely customize. Every Tlon account comes with an OpenClaw-powered bot. (Don’t worry, we safely run OpenClaw for you in our infrastructure so your bot can’t go off the rails. You’re also welcome to host your own claw if you want maximal control.) We use our bots to collect research, build nuanced daily briefings, collate data from all our disparate services. Tlon makes it insanely easy to use OpenClaw by simply installing an app from the app store, we let you keep your data and programs independent from any app or model provider, and provide the canvas to explore what’s possible. What’s most interesting for us is using bots together. On Tlon bots can create groups, augment them, moderate them, invite others and freely engage with both users and other bots. Tlon is an open playing field unlike what’s possible on conventional platforms. So, what do we do with Tlon? First and foremost, we run Tlon on Tlon. Bots coordinate data from all of our services (Linear, GitHub, all of our servers and infrastructure) and handle alerts, briefings and help us track down bugs in place. Having all of this easily synced between a desktop client and a mobile app is quick and convenient. We use bots to research new areas of work or interest. Bots can compile trees of notes, use different models to evaluate them, and then add on autoresearch-like automations to go even deeper. Since Tlon bots can freely switch between models and providers, we often pass research to Anthropic, OpenAI and self-hosted models to see different results. The most fun part of using bots as researchers is doing it together. “Put together short (~500 word) notes on the 10 most popular open source messaging protocols of the past twenty years, put them in a notebook inside a group and invite Corrina, Walt and Bill as well as their bots” is a good example. Together we’re able to move more quickly than we would on our own. Many of us also use bots to keep track of all the separate threads of work in our personal lives with close friends and family. Someone built a system for keeping track of their garden across time, someone else built a system for prepping lunches for their daughter and sending recipes to family members. Another team member built an integration that tracks what flights are passing overhead so they get a push notification every time a plane goes by. Many of us quickly communicate with our bots via voice memo when we’re out and about. Having a single interface to all the models that also holds all our data and is in our pockets feels great. Especially when the data goes into a single archive. Why is Tlon different? Every Tlon account runs on top of your very own personal server. If you ever want to download it and run it yourself, you can. If we ever go out of business, it’s yours to keep. This is very different from anything that already exists. You can’t keep your WhatsApp forever. You can’t keep your Telegram forever. Tlon is an archival-quality system that’s yours to customize. Why did we build it? In my 1999 imagination, sitting in front of a CRT somewhere in the California countryside listening to Underworld and the sound of a modem, a connected computer was an engine of unending creative potential for everyone. When I was a teenager, a computer with an internet connection felt like an infinite expanse of possibility. Not only could you use the computer to find new tools to experiment with—you could also build whatever tool you could think of. It seemed like anything was possible. I looked forward to a future where everyone could build whatever software they needed, whenever they needed it. It turned out, in the intervening twenty years, that to build and customize software you have to both write code and host it on a server somewhere. For most people, so far, that has been impossible. Instead of controlling our software, our software controls us. We rely on others to build it and decide everything about it: how it works, looks, how much it spies on us and how long it lives. But all of this is changing, fast. The hottest programming language of 2026 is English. People with no technical experience are building their own tools. It’s incredible. The expanse has opened up again. The cost of building what we think of today as software is headed to zero. What yesterday was an entire app is rapidly being replaced by a conversation. The result is hyper-specific, tailored to the user and much more efficient. Today, agents help us build workflows, automate processes and pull together disparate sources of data. All of the annoying apps and services and clunky interface we’ve put up with can just disappear. We can now program and control our computers in the programming language we already know: English. There aren’t that many of us doing this yet, though. It’s still far too hard to set up, to distribute and to trust. There’s also no single platform to experiment on and collaboratively imagine this new future of personal computing. We want everyone to be able to build bespoke, ultra-personal software on demand. We think software should be as available and accessible as a pen and paper. We think anyone should be able to enjoy the expanse of possibility that the computer provides with the lowest possible barrier to entry and the highest possible quality. So, starting far, far too long ago, we engineered a whole new system for it. Just for you. We’re opening up Tlon Messenger to a limited number of people each week. This isn’t for exclusivity’s sake, but because we’re running infrastructure for you and your agent, and covering the tokens your agent uses. That can get expensive quickly, but we want to learn what people will do with this new system we’ve built. We’re really curious to see what you can do, so give it a try and tell us what you invent. Download link to your local app store in the next tweet. Yours, Galen (and the rest of the Tlon Team)

Tlon Messenger is now open to everyone. We built a simple and infinitely flexible platform for you to use AI agents with your friends. We think it’s pretty amazing, we love using it every day, and we want to see what people can do with it. So we’re opening it up to the public. It’s fun and exciting to build the future of personal computing in an informal, chat-based way with your friends. (You can skip the rest and just download it from the link in the next tweet if you want.) If you don’t want your digital future to be owned by a giant company but you want to explore what’s possible in this new era of agent-driven computing, you should try using Tlon. But wait, what is it? Tlon is a messaging platform built 100% open source, decentralized and owned by its users from the ground up. With Tlon you own everything: your data, your workflows, your programs: the whole thing. Think of it like Telegram or WhatsApp that you own forever and you can freely customize. Every Tlon account comes with an OpenClaw-powered bot. (Don’t worry, we safely run OpenClaw for you in our infrastructure so your bot can’t go off the rails. You’re also welcome to host your own claw if you want maximal control.) We use our bots to collect research, build nuanced daily briefings, collate data from all our disparate services. Tlon makes it insanely easy to use OpenClaw by simply installing an app from the app store, we let you keep your data and programs independent from any app or model provider, and provide the canvas to explore what’s possible. What’s most interesting for us is using bots together. On Tlon bots can create groups, augment them, moderate them, invite others and freely engage with both users and other bots. Tlon is an open playing field unlike what’s possible on conventional platforms. So, what do we do with Tlon? First and foremost, we run Tlon on Tlon. Bots coordinate data from all of our services (Linear, GitHub, all of our servers and infrastructure) and handle alerts, briefings and help us track down bugs in place. Having all of this easily synced between a desktop client and a mobile app is quick and convenient. We use bots to research new areas of work or interest. Bots can compile trees of notes, use different models to evaluate them, and then add on autoresearch-like automations to go even deeper. Since Tlon bots can freely switch between models and providers, we often pass research to Anthropic, OpenAI and self-hosted models to see different results. The most fun part of using bots as researchers is doing it together. “Put together short (~500 word) notes on the 10 most popular open source messaging protocols of the past twenty years, put them in a notebook inside a group and invite Corrina, Walt and Bill as well as their bots” is a good example. Together we’re able to move more quickly than we would on our own. Many of us also use bots to keep track of all the separate threads of work in our personal lives with close friends and family. Someone built a system for keeping track of their garden across time, someone else built a system for prepping lunches for their daughter and sending recipes to family members. Another team member built an integration that tracks what flights are passing overhead so they get a push notification every time a plane goes by. Many of us quickly communicate with our bots via voice memo when we’re out and about. Having a single interface to all the models that also holds all our data and is in our pockets feels great. Especially when the data goes into a single archive. Why is Tlon different? Every Tlon account runs on top of your very own personal server. If you ever want to download it and run it yourself, you can. If we ever go out of business, it’s yours to keep. This is very different from anything that already exists. You can’t keep your WhatsApp forever. You can’t keep your Telegram forever. Tlon is an archival-quality system that’s yours to customize. Why did we build it? In my 1999 imagination, sitting in front of a CRT somewhere in the California countryside listening to Underworld and the sound of a modem, a connected computer was an engine of unending creative potential for everyone. When I was a teenager, a computer with an internet connection felt like an infinite expanse of possibility. Not only could you use the computer to find new tools to experiment with—you could also build whatever tool you could think of. It seemed like anything was possible. I looked forward to a future where everyone could build whatever software they needed, whenever they needed it. It turned out, in the intervening twenty years, that to build and customize software you have to both write code and host it on a server somewhere. For most people, so far, that has been impossible. Instead of controlling our software, our software controls us. We rely on others to build it and decide everything about it: how it works, looks, how much it spies on us and how long it lives. But all of this is changing, fast. The hottest programming language of 2026 is English. People with no technical experience are building their own tools. It’s incredible. The expanse has opened up again. The cost of building what we think of today as software is headed to zero. What yesterday was an entire app is rapidly being replaced by a conversation. The result is hyper-specific, tailored to the user and much more efficient. Today, agents help us build workflows, automate processes and pull together disparate sources of data. All of the annoying apps and services and clunky interface we’ve put up with can just disappear. We can now program and control our computers in the programming language we already know: English. There aren’t that many of us doing this yet, though. It’s still far too hard to set up, to distribute and to trust. There’s also no single platform to experiment on and collaboratively imagine this new future of personal computing. We want everyone to be able to build bespoke, ultra-personal software on demand. We think software should be as available and accessible as a pen and paper. We think anyone should be able to enjoy the expanse of possibility that the computer provides with the lowest possible barrier to entry and the highest possible quality. So, starting far, far too long ago, we engineered a whole new system for it. Just for you. We’re opening up Tlon Messenger to a limited number of people each week. This isn’t for exclusivity’s sake, but because we’re running infrastructure for you and your agent, and covering the tokens your agent uses. That can get expensive quickly, but we want to learn what people will do with this new system we’ve built. We’re really curious to see what you can do, so give it a try and tell us what you invent. Download link to your local app store in the next tweet. Yours, Galen (and the rest of the Tlon Team)

Tlon

601,476 views • 4 months ago