Загрузка видео...

Не удалось загрузить видео

На главную

💥 A 450M model just beat bigger VLAs on real robot tasks, and it’s 100% open source [📍 bookmark for later] Came across SmolVLA, a new vision-language-action model for robotics that’s compact, fast, and trained entirely on open community datasets from LeRobot via Hugging Face. What stood out to...

IlirAliu_'s profile picture

Ilir Aliu - eu/acc

11,261 subscribers

17,353 просмотров • 11 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSource
0:31

1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSource

Jun Gao

104,471 просмотров • 1 месяц назад

Another win for open-source robotics! 🔥 Hugging Face just released a fully open-source humanoid robot, and you can build one for $2,500. I'm a huge advocate of open-source in robotics space. Why? Robotics is too hard to solve alone. So lowering the entry point is crucial. Also, ROS proved open ecosystems create lasting industry standards. So let's have a look at the new member of humanoids family! LeRobot Humanoid is a 3D-printed, bipedal humanoid platform designed for real robot learning experiments. It's a complete and, buildable robot Anddddd yes, it's a full stack release: → Hardware files, bill of materials and full assembly documentation → Simulation assets and training environments → Runtime tools for calibration and real-world control → Sim-to-real identification pipeline When a part breaks, reprint it. When a design choice doesn't work, modify it and test again. The whole point is fast iteration on real hardware, not treating the robot as a fixed artefact. The timing is not a coincidence. As foundation models for robotics mature, the field desperately needs open physical platforms to train, validate and deploy them on. clem 🤗 LET'S GO!!! Build it here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:20

Another win for open-source robotics! 🔥 Hugging Face just released a fully open-source humanoid robot, and you can build one for $2,500. I'm a huge advocate of open-source in robotics space. Why? Robotics is too hard to solve alone. So lowering the entry point is crucial. Also, ROS proved open ecosystems create lasting industry standards. So let's have a look at the new member of humanoids family! LeRobot Humanoid is a 3D-printed, bipedal humanoid platform designed for real robot learning experiments. It's a complete and, buildable robot Anddddd yes, it's a full stack release: → Hardware files, bill of materials and full assembly documentation → Simulation assets and training environments → Runtime tools for calibration and real-world control → Sim-to-real identification pipeline When a part breaks, reprint it. When a design choice doesn't work, modify it and test again. The whole point is fast iteration on real hardware, not treating the robot as a fixed artefact. The timing is not a coincidence. As foundation models for robotics mature, the field desperately needs open physical platforms to train, validate and deploy them on. clem 🤗 LET'S GO!!! Build it here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

20,361 просмотров • 2 месяцев назад

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:
2:47

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

28,239 просмотров • 1 месяц назад

What if building a mobile dual-arm robot for your home took less than 7 hours and cost under $700? [Github ⬇️ Bookmark for later 📍] XLeRobot… a low-cost, open-source robot for everyday tasks. Built with off-the-shelf parts and powered by the LeRobot ecosystem, it’s made for hackers, students, and researchers. ✅ Built from Lekiwi, SO100, an IKEA cart, and an Anker battery ✅ Fully LeRobot compatible with tons of pre-trained models ✅ Easy to assemble in under 7 hours with global parts ✅ Supported by a 5,000+ member community XLeRobot is more than a fun weekend build, it’s a serious step toward affordable home robotics. Thank you, Vector Wang for sharing!!! 🫶 Github:
0:29

What if building a mobile dual-arm robot for your home took less than 7 hours and cost under $700? [Github ⬇️ Bookmark for later 📍] XLeRobot… a low-cost, open-source robot for everyday tasks. Built with off-the-shelf parts and powered by the LeRobot ecosystem, it’s made for hackers, students, and researchers. ✅ Built from Lekiwi, SO100, an IKEA cart, and an Anker battery ✅ Fully LeRobot compatible with tons of pre-trained models ✅ Easy to assemble in under 7 hours with global parts ✅ Supported by a 5,000+ member community XLeRobot is more than a fun weekend build, it’s a serious step toward affordable home robotics. Thank you, Vector Wang for sharing!!! 🫶 Github:

Ilir Aliu

32,099 просмотров • 1 год назад

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...
2:12

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

kwindla

42,246 просмотров • 1 год назад

“We have a crisis of open source models in the Western world. Outside of China, there are no good open source models. We don't even have any in the US now. The talent, the capital and the focus to be best-in-class at pre-training, mid-training, post-training is an extremely scarce skillset. The answer for a lot of countries may just be to take an open weights model from China, post-train or fine-tune their own version, and have that be what they start from." Everett Randle Why does the West have such poor open source models and is it a national security threat to rely on open-source Chinese models Demis Hassabis Yann LeCun Aidan Gomez Lilian Weng Jan Leike
1:20

“We have a crisis of open source models in the Western world. Outside of China, there are no good open source models. We don't even have any in the US now. The talent, the capital and the focus to be best-in-class at pre-training, mid-training, post-training is an extremely scarce skillset. The answer for a lot of countries may just be to take an open weights model from China, post-train or fine-tune their own version, and have that be what they start from." Everett Randle Why does the West have such poor open source models and is it a national security threat to rely on open-source Chinese models Demis Hassabis Yann LeCun Aidan Gomez Lilian Weng Jan Leike

Harry Stebbings

63,672 просмотров • 1 месяц назад

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER
0:30

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

Ilir Aliu - eu/acc

10,509 просмотров • 1 год назад

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.
2:35

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

AI at Meta

453,260 просмотров • 1 год назад

A new robot policy just cleaned up a kitchen it had never seen before [watch what happens. paper included ⬇️] Pi-0.5 builds on top of Pi-0 and shows how smart co-training with diverse data can unlock real generalization in the home. It doesn’t just learn from one setup but adapts to many, including homes it’s never seen. What it does ✅ Handles new homes without training in them ✅ Follows complex language instructions ✅ Cleans, places dishes, handles spills ✅ Matches in-home training models using cross-embodiment and web data Robots that understand tasks and adapt to new spaces are finally within reach. More in the blog: Read the paper ⬇️ Physical Intelligence, co-founded by UC Berkeley professor Sergey Levine, is a robotics startup developing general-purpose AI foundation models that enable robots to perform a wide variety of real-world tasks with human-like adaptability, recently raising $400 million to advance this vision.
1:22

A new robot policy just cleaned up a kitchen it had never seen before [watch what happens. paper included ⬇️] Pi-0.5 builds on top of Pi-0 and shows how smart co-training with diverse data can unlock real generalization in the home. It doesn’t just learn from one setup but adapts to many, including homes it’s never seen. What it does ✅ Handles new homes without training in them ✅ Follows complex language instructions ✅ Cleans, places dishes, handles spills ✅ Matches in-home training models using cross-embodiment and web data Robots that understand tasks and adapt to new spaces are finally within reach. More in the blog: Read the paper ⬇️ Physical Intelligence, co-founded by UC Berkeley professor Sergey Levine, is a robotics startup developing general-purpose AI foundation models that enable robots to perform a wide variety of real-world tasks with human-like adaptability, recently raising $400 million to advance this vision.

Ilir Aliu - eu/acc

18,120 просмотров • 1 год назад

This is a pretty wild model! You can use it to turn an image into a 3D object with texture. The quality is out of this world! I'm not even a designer, and I've been using this nonstop for the last 2 hours. The model is Hunyuan 3D 2.1. It's open source. You'll find model weights, training/inference code, data pipelines, and architecture on their repository. You can even fine-tune it if you want! GitHub Repository: By the way, the model runs on consumer-grade GPUs. You don't need a datacenter for this! I've been using the model from the HuggingFace demo page: To use it, go to the link and upload an image. That's it! Check out the video I recorded for a couple of examples.
2:42

This is a pretty wild model! You can use it to turn an image into a 3D object with texture. The quality is out of this world! I'm not even a designer, and I've been using this nonstop for the last 2 hours. The model is Hunyuan 3D 2.1. It's open source. You'll find model weights, training/inference code, data pipelines, and architecture on their repository. You can even fine-tune it if you want! GitHub Repository: By the way, the model runs on consumer-grade GPUs. You don't need a datacenter for this! I've been using the model from the HuggingFace demo page: To use it, go to the link and upload an image. That's it! Check out the video I recorded for a couple of examples.

Santiago

44,783 просмотров • 1 год назад

The wait is over. Meet the reBot-DevArm B601—the fully open-source robotic arm. 🦾 Built, broken, and rebuilt for reliability. Here’s what makes it unique: ✅ 1.5kg payload & 650mm reach ✅ 6+1 DOF for complex tasks ✅ Plug-and-play with ROS2, Isaac Sim, LeRobot We tested it endlessly—7x24h runs, overload tests, gravity compensation—to make it a robust platform for embodied AI. And it’s 100% open-source (BOM, CAD, code). Build, hack, and innovate. All files & tutorials are live on GitHub. Check it out and join the build! 🔗Explore the GitHub here:
2:26

The wait is over. Meet the reBot-DevArm B601—the fully open-source robotic arm. 🦾 Built, broken, and rebuilt for reliability. Here’s what makes it unique: ✅ 1.5kg payload & 650mm reach ✅ 6+1 DOF for complex tasks ✅ Plug-and-play with ROS2, Isaac Sim, LeRobot We tested it endlessly—7x24h runs, overload tests, gravity compensation—to make it a robust platform for embodied AI. And it’s 100% open-source (BOM, CAD, code). Build, hack, and innovate. All files & tutorials are live on GitHub. Check it out and join the build! 🔗Explore the GitHub here:

Seeed Studio

29,168 просмотров • 4 месяцев назад

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:
0:47

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

22,812 просмотров • 8 месяцев назад

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.
1:24

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,529 просмотров • 6 месяцев назад

A team tested Pi0, Pi0 Fast, Gr00t, and ACT on real robot arms in manufacturing tasks. (🔖 Bookmark this for later!) The task was precise: place thin rectangular frames from a messy stack into a holder. The team fine-tuned each model on 100 real trajectories and compared training time, inference speed, motion quality, and success rates. ⬇️ Here’s a breakdown of what they found Pi0 (Original) ✅ Strongest overall performance in precise pick-and-place ✅ High success rate even in edge cases ✅ Longest training time (~11 hours, ~$30 per run) ✅ Inference time of 80 ms causes short pauses between actions Despite delays, it handles complex scenarios well… solid for high-precision tasks, but slow to train. Gr00t ✅ Trains fast (~2 hours, ~$5 per run) ✅ Performs almost as well as Pi0 on large-object tasks ✅ Struggles with fine precision; random movement in some trials ✅ More training didn’t fix jitter or random offsets Best suited for tasks where exact precision isn’t critical. Not ready for manufacturing-grade accuracy without more tuning. Pi0 Fast ✅ Promised faster training, but results were underwhelming ✅ Training at 6 hours still showed low success rates ✅ Inference was slower than expected ✅ Not reliable for generalizing even slightly new tasks Currently too unstable for real-world deployment. Doesn’t live up to the “Fast” name yet. ACT (Baseline) ✅ 200MB model—lightweight, but limited ✅ Struggles with stacked objects or ambiguous scenes ✅ Success rates around 70% in best-case setups ✅ Can’t match newer models on precision or generalization Still a solid baseline, but clearly a generation behind in robustness. 🚨 Extra Notes All newer models share a common issue: •Inference takes longer than a frame (80 ms vs 33 ms), so robots “pause” between chunks. •This results in jittery movements, but not a dealbreaker unless tasks are time-sensitive. Language-conditioned tasks also fell short: after training on two labeled tasks, the model couldn’t generalize to a third unseen combination using only text prompts. ✅ The good news? These models adapt well to new robot arms with quick fine-tuning. ❌ The bad news? There’s still no plug-and-play solution for improving performance after deployment. Reinforcement learning or DAgger-style data collection during real-world operation may be the next big step, something many teams in robotics are actively working on.
0:51

A team tested Pi0, Pi0 Fast, Gr00t, and ACT on real robot arms in manufacturing tasks. (🔖 Bookmark this for later!) The task was precise: place thin rectangular frames from a messy stack into a holder. The team fine-tuned each model on 100 real trajectories and compared training time, inference speed, motion quality, and success rates. ⬇️ Here’s a breakdown of what they found Pi0 (Original) ✅ Strongest overall performance in precise pick-and-place ✅ High success rate even in edge cases ✅ Longest training time (~11 hours, ~$30 per run) ✅ Inference time of 80 ms causes short pauses between actions Despite delays, it handles complex scenarios well… solid for high-precision tasks, but slow to train. Gr00t ✅ Trains fast (~2 hours, ~$5 per run) ✅ Performs almost as well as Pi0 on large-object tasks ✅ Struggles with fine precision; random movement in some trials ✅ More training didn’t fix jitter or random offsets Best suited for tasks where exact precision isn’t critical. Not ready for manufacturing-grade accuracy without more tuning. Pi0 Fast ✅ Promised faster training, but results were underwhelming ✅ Training at 6 hours still showed low success rates ✅ Inference was slower than expected ✅ Not reliable for generalizing even slightly new tasks Currently too unstable for real-world deployment. Doesn’t live up to the “Fast” name yet. ACT (Baseline) ✅ 200MB model—lightweight, but limited ✅ Struggles with stacked objects or ambiguous scenes ✅ Success rates around 70% in best-case setups ✅ Can’t match newer models on precision or generalization Still a solid baseline, but clearly a generation behind in robustness. 🚨 Extra Notes All newer models share a common issue: •Inference takes longer than a frame (80 ms vs 33 ms), so robots “pause” between chunks. •This results in jittery movements, but not a dealbreaker unless tasks are time-sensitive. Language-conditioned tasks also fell short: after training on two labeled tasks, the model couldn’t generalize to a third unseen combination using only text prompts. ✅ The good news? These models adapt well to new robot arms with quick fine-tuning. ❌ The bad news? There’s still no plug-and-play solution for improving performance after deployment. Reinforcement learning or DAgger-style data collection during real-world operation may be the next big step, something many teams in robotics are actively working on.

Ilir Aliu

21,844 просмотров • 1 год назад

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵
2:21

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

466,261 просмотров • 1 год назад

Open science is how we continue to push technology forward and today at Meta FAIR we’re sharing eight new AI research artifacts including new models, datasets and code to inspire innovation in the community. More in the video from Joelle Pineau. This work is another important step towards our goal of achieving Advanced Machine Intelligence (AMI). What we’re releasing: • Meta Spirit LM: An open source language model for seamless speech and text integration. • Meta Segment Anything Model 2.1: An updated checkpoint with improved results on visually similar objects, small objects and occlusion handling. Plus a new developer suite to make it easier for developers to build with SAM 2. • Layer Skip: Inference code and fine-tuned checkpoints demonstrating a new method for enhancing LLM performance. • SALSA: New code to enable researchers to benchmark AI-based attacks in support of validating security for post-quantum cryptography. • Meta Lingua: A lightweight and self-contained codebase designed to train language models at scale. • Meta Open Materials: New open source models and the largest dataset of its kind to accelerate AI-driven discovery of new inorganic materials. • MEXMA: A new research paper and code for our novel pre-trained cross-lingual sentence encoder with coverage across 80 languages. • Self-Taught Evaluator: a new method for generating synthetic preference data to train reward models without relying on human annotations. Access to state-of-the-art AI creates opportunities for everyone. We’re excited to share this work and look forward to seeing the community innovation that results from it. Details and access to everything released by FAIR today ➡️
1:36

Open science is how we continue to push technology forward and today at Meta FAIR we’re sharing eight new AI research artifacts including new models, datasets and code to inspire innovation in the community. More in the video from Joelle Pineau. This work is another important step towards our goal of achieving Advanced Machine Intelligence (AMI). What we’re releasing: • Meta Spirit LM: An open source language model for seamless speech and text integration. • Meta Segment Anything Model 2.1: An updated checkpoint with improved results on visually similar objects, small objects and occlusion handling. Plus a new developer suite to make it easier for developers to build with SAM 2. • Layer Skip: Inference code and fine-tuned checkpoints demonstrating a new method for enhancing LLM performance. • SALSA: New code to enable researchers to benchmark AI-based attacks in support of validating security for post-quantum cryptography. • Meta Lingua: A lightweight and self-contained codebase designed to train language models at scale. • Meta Open Materials: New open source models and the largest dataset of its kind to accelerate AI-driven discovery of new inorganic materials. • MEXMA: A new research paper and code for our novel pre-trained cross-lingual sentence encoder with coverage across 80 languages. • Self-Taught Evaluator: a new method for generating synthetic preference data to train reward models without relying on human annotations. Access to state-of-the-art AI creates opportunities for everyone. We’re excited to share this work and look forward to seeing the community innovation that results from it. Details and access to everything released by FAIR today ➡️

AI at Meta

150,222 просмотров • 1 год назад

There is a beautiful story that just happened in AI so let me share it for a lighter tone weekend post among all the doom stories in our AI field this week. It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI model. It started a couple of months ago when a new team in the AI scene released their first model from their headquarters in Paris (France): Mistral 7B. Impressive model, small and very strong performances in the benchmarks, better than all previous models of this size. And open source! So you could build on top of it. Lewis in Bern (Switzerland) and Ed (in Lyon, in the South of France) both from the H4 team, a team of researchers in model fine-tuning and alignment were talking about it over a coffee, in one of these gatherings that often happen at Hugging Face to break the distance between people (literal distance as HF is a remote company). What about fine-tuning it using this new DPO method that a research team from Stanford in California just posted on Arxiv, says one? Hey, that’s a great idea, replies the other. We've just build a great code base (with Nathan, Nazneen, Costa, Younes and all the H4 team and TRL community) let's use it! The next day they start diving in the datasets openly shared on the HF hub and stumble upon two interesting large and good quality fine-tuning datasets recently open-sourced by OpenBMB, a Chinese team from Tsinghua: UltraFeedback and UltraChat. A few rounds of training experiments confirm the intuition, the resulting model is super strong, by far the strongest they have ever seen in their benchmarks from Berkeley and Stanford (LMSYS and Alpaca). Join Clementine, the big boss of the open evaluation leaderboard. Her deep dive into the model capabilities confirms the results: impressive performance. But the H4 team also hosts a famous faculty member, Pr. Sasha Rush, Associate Professor at Cornell University in his daytime, hacker at HF in his nighttime. Joining the conversation, he proposes to quickly draft a research paper to organize and share all the details with the community. A few days later, the model, called Zephyr (a wind like Mistral), paper, and all details are shared with the world. Quickly other companies, everywhere in the world starts to use it. LlamaIndex, a famous data framework and community, shares how the model blew their expectations on real-life use-case benchmarks, while researchers and practitioners discuss the paper and work on the Hugging Face hub. All this happened in just a few weeks catalyzed by open access to knowledge, models, research, and datasets released all over the world (Europe, California, China) and by the idea that people can build upon one another work in AI to bring real-world value with efficient and open models. Stories like this are numerous everywhere around us and make me really proud of the AI community and see how we can build amazingly useful things together. [the video is just me reading this Friday post hahah]
4:30

There is a beautiful story that just happened in AI so let me share it for a lighter tone weekend post among all the doom stories in our AI field this week. It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI model. It started a couple of months ago when a new team in the AI scene released their first model from their headquarters in Paris (France): Mistral 7B. Impressive model, small and very strong performances in the benchmarks, better than all previous models of this size. And open source! So you could build on top of it. Lewis in Bern (Switzerland) and Ed (in Lyon, in the South of France) both from the H4 team, a team of researchers in model fine-tuning and alignment were talking about it over a coffee, in one of these gatherings that often happen at Hugging Face to break the distance between people (literal distance as HF is a remote company). What about fine-tuning it using this new DPO method that a research team from Stanford in California just posted on Arxiv, says one? Hey, that’s a great idea, replies the other. We've just build a great code base (with Nathan, Nazneen, Costa, Younes and all the H4 team and TRL community) let's use it! The next day they start diving in the datasets openly shared on the HF hub and stumble upon two interesting large and good quality fine-tuning datasets recently open-sourced by OpenBMB, a Chinese team from Tsinghua: UltraFeedback and UltraChat. A few rounds of training experiments confirm the intuition, the resulting model is super strong, by far the strongest they have ever seen in their benchmarks from Berkeley and Stanford (LMSYS and Alpaca). Join Clementine, the big boss of the open evaluation leaderboard. Her deep dive into the model capabilities confirms the results: impressive performance. But the H4 team also hosts a famous faculty member, Pr. Sasha Rush, Associate Professor at Cornell University in his daytime, hacker at HF in his nighttime. Joining the conversation, he proposes to quickly draft a research paper to organize and share all the details with the community. A few days later, the model, called Zephyr (a wind like Mistral), paper, and all details are shared with the world. Quickly other companies, everywhere in the world starts to use it. LlamaIndex, a famous data framework and community, shares how the model blew their expectations on real-life use-case benchmarks, while researchers and practitioners discuss the paper and work on the Hugging Face hub. All this happened in just a few weeks catalyzed by open access to knowledge, models, research, and datasets released all over the world (Europe, California, China) and by the idea that people can build upon one another work in AI to bring real-world value with efficient and open models. Stories like this are numerous everywhere around us and make me really proud of the AI community and see how we can build amazingly useful things together. [the video is just me reading this Friday post hahah]

Thomas Wolf

169,127 просмотров • 2 лет назад

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.
16:27

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.

abdel

17,001 просмотров • 3 месяцев назад

We are back. After one year of quiet building. Introducing GENE-26.5, our first robotic brain that takes a major step toward human-level capability. For years, robotics has struggled to learn from the world’s largest and valuable data source: Humans. Solving it means rethinking the whole stack from the ground up: - A robotics-native foundation model. - A 1:1 human-like robotic hand. - A noninvasive data collection glove for motion, force, and touch. - A simulator that turns weeks of experiments into minutes. GENE-26.5 is trained across language, vision, proprioception, tactile, and action. We designed a set of tasks to test how far we can go with this new paradigm. Fully autonomous, 1x speed, one model, same weights. (Enjoy with sound on) We are approaching the endgame for robotics. And this is just a beginning.
3:05

We are back. After one year of quiet building. Introducing GENE-26.5, our first robotic brain that takes a major step toward human-level capability. For years, robotics has struggled to learn from the world’s largest and valuable data source: Humans. Solving it means rethinking the whole stack from the ground up: - A robotics-native foundation model. - A 1:1 human-like robotic hand. - A noninvasive data collection glove for motion, force, and touch. - A simulator that turns weeks of experiments into minutes. GENE-26.5 is trained across language, vision, proprioception, tactile, and action. We designed a set of tasks to test how far we can go with this new paradigm. Fully autonomous, 1x speed, one model, same weights. (Enjoy with sound on) We are approaching the endgame for robotics. And this is just a beginning.

Genesis AI

2,718,226 просмотров • 2 месяцев назад