Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

𝗪𝗵𝗮𝘁 𝗶𝗳 𝘆𝗼𝘂 𝗰𝗼𝘂𝗹𝗱 𝗴𝗼 𝗳𝗿𝗼𝗺 𝘁𝗲𝗹𝗲𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗮 𝗿𝗼𝗯𝗼𝘁 𝘁𝗼 𝗱𝗲𝗽𝗹𝗼𝘆𝗶𝗻𝗴 𝗮 𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗔𝗜 𝗽𝗼𝗹𝗶𝗰𝘆 𝗶𝗻 𝗹𝗲𝘀𝘀 𝘁𝗶𝗺𝗲 𝘁𝗵𝗮𝗻 𝗶𝘁 𝘁𝗮𝗸𝗲𝘀 𝘁𝗼 𝘄𝗮𝘁𝗰𝗵 𝗮 𝗺𝗼𝘃𝗶𝗲? Today, we're excited to open source our complete example showing exactly that: using Neuracore with the Piper robot from AgileX Robotics. What’s included: •... 𝗠𝗲𝘁𝗮 𝗤𝘂𝗲𝘀𝘁 𝘁𝗲𝗹𝗲𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻 – Control your robot with VR • 𝗘𝗻𝗱-𝘁𝗼-𝗲𝗻𝗱 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄 – Demo collection to model training to policy deployment • 𝗥𝗢𝗦 𝘃𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗲𝗿 – See your headset data with a single command • 𝗢𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲 𝗠𝗲𝘁𝗮 𝗤𝘂𝗲𝘀𝘁 𝘁𝗲𝗹𝗲𝗼𝗽 𝗽𝗮𝗰𝗸𝗮𝗴𝗲 – Works with any ROS-enabled robot If you have a Piper robot, grab the example, sign up for Neuracore, and show us what you build. Repo: Learn more: Open Robotics Meta Questshow more

Neuracore

2,196 subscribers

21,894 görüntüleme • 7 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.

Neuracore

40,796 görüntüleme • 9 ay önce

Awesome Robotics! 💾 Building your own robot just got a lot easier. MathWorks have released an open-source GitHub repository packed with robotics resources for anyone interested in getting hands-on. The repo includes examples for robot arms, ground vehicles, and drones, with projects that show how to connect with ROS and ROS2 or even deploy Simulink models directly as ROS nodes. There are also more advanced demos, like modeling off-road environments and testing navigation algorithms in photorealistic simulations. Everything is well-documented, with tutorials and links that make it easy to go from concept to prototype. 📑 Whether you’re a student, researcher, or just curious, there’s material here for every level. And since it’s an open community project, you can not only explore but also share your own contributions. For anyone looking to learn robotics by doing, this is a solid place to start! Here’s the link: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Awesome Robotics! 💾 Building your own robot just got a lot easier. MathWorks have released an open-source GitHub repository packed with robotics resources for anyone interested in getting hands-on. The repo includes examples for robot arms, ground vehicles, and drones, with projects that show how to connect with ROS and ROS2 or even deploy Simulink models directly as ROS nodes. There are also more advanced demos, like modeling off-road environments and testing navigation algorithms in photorealistic simulations. Everything is well-documented, with tutorials and links that make it easy to go from concept to prototype. 📑 Whether you’re a student, researcher, or just curious, there’s material here for every level. And since it’s an open community project, you can not only explore but also share your own contributions. For anyone looking to learn robotics by doing, this is a solid place to start! Here’s the link: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

59,232 görüntüleme • 3 ay önce

Awesome Robotics! 💾 Building your own robot just got a lot easier. MathWorks have released an open-source GitHub repository packed with robotics resources for anyone interested in getting hands-on. The repo includes examples for robot arms, ground vehicles, and drones, with projects that show how to connect with ROS and ROS2 or even deploy Simulink models directly as ROS nodes. There are also more advanced demos, like modeling off-road environments and testing navigation algorithms in photorealistic simulations. Everything is well-documented, with tutorials and links that make it easy to go from concept to prototype. Whether you’re a student, researcher, or just curious, there’s material here for every level. And since it’s an open community project, you can not only explore but also share your own contributions. For anyone looking to learn robotics by doing, this is a solid place to start! Here’s the link:

Awesome Robotics! 💾 Building your own robot just got a lot easier. MathWorks have released an open-source GitHub repository packed with robotics resources for anyone interested in getting hands-on. The repo includes examples for robot arms, ground vehicles, and drones, with projects that show how to connect with ROS and ROS2 or even deploy Simulink models directly as ROS nodes. There are also more advanced demos, like modeling off-road environments and testing navigation algorithms in photorealistic simulations. Everything is well-documented, with tutorials and links that make it easy to go from concept to prototype. Whether you’re a student, researcher, or just curious, there’s material here for every level. And since it’s an open community project, you can not only explore but also share your own contributions. For anyone looking to learn robotics by doing, this is a solid place to start! Here’s the link:

Lukas Ziegler

84,272 görüntüleme • 11 ay önce

Train AI robots without writing a single line of code. 🤖 We just launched LeLab, the official graphical user interface for LeRobot built by Nicolas Rabault. It completely removes the command line from the robot learning workflow, taking you from raw hardware to autonomous movement visually. If you've ever wanted to get into AI robotics but were held back by complex terminal setups, this is for you. - Zero-Terminal Setup: Smart calibration with automatic USB port detection. - Easy Data Collection: Teleoperate your robot and record a dataset. - One-Click GPU Training: Don't have a massive local GPU? Scale your training instantly with Hugging Face Jobs right inside the app. Just plug in your SO-ARM101 and start teaching your robot. We put together a complete, step-by-step video guide showing exactly how to get started and train your first policy. Docs: GitHub:

Train AI robots without writing a single line of code. 🤖 We just launched LeLab, the official graphical user interface for LeRobot built by Nicolas Rabault. It completely removes the command line from the robot learning workflow, taking you from raw hardware to autonomous movement visually. If you've ever wanted to get into AI robotics but were held back by complex terminal setups, this is for you. - Zero-Terminal Setup: Smart calibration with automatic USB port detection. - Easy Data Collection: Teleoperate your robot and record a dataset. - One-Click GPU Training: Don't have a massive local GPU? Scale your training instantly with Hugging Face Jobs right inside the app. Just plug in your SO-ARM101 and start teaching your robot. We put together a complete, step-by-step video guide showing exactly how to get started and train your first policy. Docs: GitHub:

LeRobot

50,616 görüntüleme • 3 ay önce

𝗛𝗲𝗿𝗲’𝘀 𝗮 𝗺𝗶𝘀𝘁𝗮𝗸𝗲 𝗜 𝘀𝗲𝗲 𝗮𝗹𝗹 𝘁𝗵𝗲 𝘁𝗶𝗺𝗲. Teams collect robot data at 30Hz because “that’s what our robot runs at” and then wonder why their models underperform. The truth is that 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝘁𝗮𝘀𝗸𝘀 𝗻𝗲𝗲𝗱 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝘁𝗲𝗺𝗽𝗼𝗿𝗮𝗹 𝗿𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀. Pick-and-place often works best around 10Hz for smooth motions. Dynamic catching requires 30Hz or more. Assembly tasks move slower, so 5Hz can suffice. Precision insertion demands 50Hz or higher for micro-adjustments. Here’s the catch. The 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆 𝗶𝘀𝗻’𝘁 𝘀𝗼𝗺𝗲𝘁𝗵𝗶𝗻𝗴 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗴𝘂𝗲𝘀𝘀. The traditional approach wastes time and demos. You pick a frequency, collect thousands of demos, train a model, get mediocre results, and realize you have to start over. A better way is to 𝗰𝗼𝗹𝗹𝗲𝗰𝘁 𝗱𝗮𝘁𝗮 𝗮𝘁 𝗵𝗶𝗴𝗵 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆, experiment with sync rates, and find what actually works for your task. We’ve seen grasping tasks perform best at 12Hz - not 10Hz, not 15Hz - discovered only through systematic testing. 𝗦𝘆𝗻𝗰𝗵𝗿𝗼𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆 𝗶𝘀 𝗮 𝗵𝘆𝗽𝗲𝗿𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿. 𝗧𝗿𝗲𝗮𝘁 𝗶𝘁 𝗹𝗶𝗸𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗿𝗮𝘁𝗲 𝗼𝗿 𝗯𝗮𝘁𝗰𝗵 𝘀𝗶𝘇𝗲. 𝗗𝗼𝗻’𝘁 𝗱𝗲𝗰𝗶𝗱𝗲 𝗶𝘁 𝗼𝗻𝗰𝗲 𝗮𝗻𝗱 𝗿𝗲𝗴𝗿𝗲𝘁 𝗶𝘁 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. How do you choose your data collection frequency, fixed upfront or experimental?

𝗛𝗲𝗿𝗲’𝘀 𝗮 𝗺𝗶𝘀𝘁𝗮𝗸𝗲 𝗜 𝘀𝗲𝗲 𝗮𝗹𝗹 𝘁𝗵𝗲 𝘁𝗶𝗺𝗲. Teams collect robot data at 30Hz because “that’s what our robot runs at” and then wonder why their models underperform. The truth is that 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝘁𝗮𝘀𝗸𝘀 𝗻𝗲𝗲𝗱 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝘁𝗲𝗺𝗽𝗼𝗿𝗮𝗹 𝗿𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀. Pick-and-place often works best around 10Hz for smooth motions. Dynamic catching requires 30Hz or more. Assembly tasks move slower, so 5Hz can suffice. Precision insertion demands 50Hz or higher for micro-adjustments. Here’s the catch. The 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆 𝗶𝘀𝗻’𝘁 𝘀𝗼𝗺𝗲𝘁𝗵𝗶𝗻𝗴 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗴𝘂𝗲𝘀𝘀. The traditional approach wastes time and demos. You pick a frequency, collect thousands of demos, train a model, get mediocre results, and realize you have to start over. A better way is to 𝗰𝗼𝗹𝗹𝗲𝗰𝘁 𝗱𝗮𝘁𝗮 𝗮𝘁 𝗵𝗶𝗴𝗵 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆, experiment with sync rates, and find what actually works for your task. We’ve seen grasping tasks perform best at 12Hz - not 10Hz, not 15Hz - discovered only through systematic testing. 𝗦𝘆𝗻𝗰𝗵𝗿𝗼𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆 𝗶𝘀 𝗮 𝗵𝘆𝗽𝗲𝗿𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿. 𝗧𝗿𝗲𝗮𝘁 𝗶𝘁 𝗹𝗶𝗸𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗿𝗮𝘁𝗲 𝗼𝗿 𝗯𝗮𝘁𝗰𝗵 𝘀𝗶𝘇𝗲. 𝗗𝗼𝗻’𝘁 𝗱𝗲𝗰𝗶𝗱𝗲 𝗶𝘁 𝗼𝗻𝗰𝗲 𝗮𝗻𝗱 𝗿𝗲𝗴𝗿𝗲𝘁 𝗶𝘁 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. How do you choose your data collection frequency, fixed upfront or experimental?

Stephen James

22,574 görüntüleme • 10 ay önce

Better data for robot hands! 🤲🏼 Embodied reasoning is the bottleneck now. It’s the data we are missing. Specifically, understanding what the robot's hands are doing throughout a task. Perceptron AI new Egocentric offering tracks both hands through an entire video instead of guessing from sampled frames. That matters because a lot happens between frames. Hand positions shift. Contacts change. The robot adjusts grip. You miss all that if you're sampling. 0.280 semantic end-to-end F1 score on WGO-Bench vs 0.158 from the best pipelines built on Gemini Robotics-ER 1.6. Training a robot to manipulate objects requires understanding every moment of hand-object interaction. If your training data has gaps: missing frames, missed contacts, unclear hand positions, the policy learns from incomplete information. You end up with robots that work in demos but fail in real tasks. Continuous hand tracking gives you the full picture. The policy learns from cleaner, more complete data. That compounds into better real-world performance. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Better data for robot hands! 🤲🏼 Embodied reasoning is the bottleneck now. It’s the data we are missing. Specifically, understanding what the robot's hands are doing throughout a task. Perceptron AI new Egocentric offering tracks both hands through an entire video instead of guessing from sampled frames. That matters because a lot happens between frames. Hand positions shift. Contacts change. The robot adjusts grip. You miss all that if you're sampling. 0.280 semantic end-to-end F1 score on WGO-Bench vs 0.158 from the best pipelines built on Gemini Robotics-ER 1.6. Training a robot to manipulate objects requires understanding every moment of hand-object interaction. If your training data has gaps: missing frames, missed contacts, unclear hand positions, the policy learns from incomplete information. You end up with robots that work in demos but fail in real tasks. Continuous hand tracking gives you the full picture. The policy learns from cleaner, more complete data. That compounds into better real-world performance. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

18,348 görüntüleme • 1 ay önce

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

AI at Meta

453,343 görüntüleme • 1 yıl önce

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

466,927 görüntüleme • 1 yıl önce

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Jim Fan

332,238 görüntüleme • 2 yıl önce

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning inside a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation inside a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Jim Fan

227,634 görüntüleme • 6 ay önce

Imagine controlling a real robot from your home… no money, no experience needed. Sounds crazy, right? But it’s already possible. BitRobot 🦾 is building the world’s first open robotics lab powered by crypto incentives. Instead of one company doing everything, it connects people from all over the world to work together on real robotics and AI tasks. The network is made up of specialized subnets, each focused on different missions from collecting real-world data with robots to developing humanoid robots for everyday use. What makes it powerful? It uses crypto rewards to coordinate global resources like compute power, robot fleets, teleoperation time, and even human effort. This allows BitRobot to scale much faster than traditional labs. Now here’s the best part 👇 The easiest way to get involved right now is through TeleArms. You don’t need: – a robot – engineering skills – or any investment – Hardware All you need is a laptop and an internet connection. From your home, you can remotely control a real robotic arm inside BitRobot’s lab using your keyboard or mouse to pick up, move, and place objects. Every action you take helps generate real-world data that trains the next generation of AI to perform useful physical tasks. So you’re not just playing with a robot… You’re actually helping build the future of AI. I’ve been talking about BitRobot for a while, and now TeleArms is live! You can control a real robotic arm from home, but it’s in a private beta with limited access. I’m now an ambassador for BitRobot Network. I’m giving 4 exclusive access codes to my community so they can experience it too. A lot of people want to experience this, but since it’s limited, I decided to do a random giveaway. To participate in this giveaway : 1. Join the BitRobot Network Discord (Link in comments) 2. Come back to this post and comment below, explaining why you want to join TeleArms and how you plan to contribute. Note : Winner will be announced in the last 7 days. Once you do that, you’ll be in the running for one of the codes! Good luck, and I can’t wait to see your ideas!

Imagine controlling a real robot from your home… no money, no experience needed. Sounds crazy, right? But it’s already possible. BitRobot 🦾 is building the world’s first open robotics lab powered by crypto incentives. Instead of one company doing everything, it connects people from all over the world to work together on real robotics and AI tasks. The network is made up of specialized subnets, each focused on different missions from collecting real-world data with robots to developing humanoid robots for everyday use. What makes it powerful? It uses crypto rewards to coordinate global resources like compute power, robot fleets, teleoperation time, and even human effort. This allows BitRobot to scale much faster than traditional labs. Now here’s the best part 👇 The easiest way to get involved right now is through TeleArms. You don’t need: – a robot – engineering skills – or any investment – Hardware All you need is a laptop and an internet connection. From your home, you can remotely control a real robotic arm inside BitRobot’s lab using your keyboard or mouse to pick up, move, and place objects. Every action you take helps generate real-world data that trains the next generation of AI to perform useful physical tasks. So you’re not just playing with a robot… You’re actually helping build the future of AI. I’ve been talking about BitRobot for a while, and now TeleArms is live! You can control a real robotic arm from home, but it’s in a private beta with limited access. I’m now an ambassador for BitRobot Network. I’m giving 4 exclusive access codes to my community so they can experience it too. A lot of people want to experience this, but since it’s limited, I decided to do a random giveaway. To participate in this giveaway : 1. Join the BitRobot Network Discord (Link in comments) 2. Come back to this post and comment below, explaining why you want to join TeleArms and how you plan to contribute. Note : Winner will be announced in the last 7 days. Once you do that, you’ll be in the running for one of the codes! Good luck, and I can’t wait to see your ideas!

Apurba.Eth

36,318 görüntüleme • 5 ay önce

People who've never set foot in a factory will never understand... I watched this three times. For decades, robotics simulation has promised faster deployment. But factories still had to build the real cell to see if it actually worked. Which meant expensive physical prototypes, weeks or months!!! of commissioning, constant surprises between simulation and reality That “sim-to-real gap” has quietly been one of the biggest bottlenecks in manufacturing automation. And it’s exactly what is changing. Today, ABB Robotics announced a partnership with NVIDIA Robotics aimed at closing this gap through the new RobotStudio HyperReality platform: Simulation and real robot behavior can match with near-perfect accuracy. That means manufacturers can design, test, and validate entire production lines before a single robot is installed on the factory floor. The implications are massive: • up to 80% faster setup and commissioning • roughly 40% lower costs by removing physical prototypes • about 50% faster time-to-market for new production lines In other words: Factories can move from trial-and-error engineering to software-driven manufacturing design. Production lines become something you build and validate digitally first. Then deploy physically once everything already works. For an industry that still measures deployment timelines in months or years, this is a major shift. It changes how automation projects are planned, how factories are designed, and how fast manufacturing can adapt to new products. Physical AI actually becomes deployable at an industrial scale. I’ll be at GTC in San Jose next week to see and talk to manufacturers and robotics engineers. If you are into manufacturing like I am, hit me up; my DMs are open!

People who've never set foot in a factory will never understand... I watched this three times. For decades, robotics simulation has promised faster deployment. But factories still had to build the real cell to see if it actually worked. Which meant expensive physical prototypes, weeks or months!!! of commissioning, constant surprises between simulation and reality That “sim-to-real gap” has quietly been one of the biggest bottlenecks in manufacturing automation. And it’s exactly what is changing. Today, ABB Robotics announced a partnership with NVIDIA Robotics aimed at closing this gap through the new RobotStudio HyperReality platform: Simulation and real robot behavior can match with near-perfect accuracy. That means manufacturers can design, test, and validate entire production lines before a single robot is installed on the factory floor. The implications are massive: • up to 80% faster setup and commissioning • roughly 40% lower costs by removing physical prototypes • about 50% faster time-to-market for new production lines In other words: Factories can move from trial-and-error engineering to software-driven manufacturing design. Production lines become something you build and validate digitally first. Then deploy physically once everything already works. For an industry that still measures deployment timelines in months or years, this is a major shift. It changes how automation projects are planned, how factories are designed, and how fast manufacturing can adapt to new products. Physical AI actually becomes deployable at an industrial scale. I’ll be at GTC in San Jose next week to see and talk to manufacturers and robotics engineers. If you are into manufacturing like I am, hit me up; my DMs are open!

Ilir Aliu

68,927 görüntüleme • 5 ay önce

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Jim Fan

216,139 görüntüleme • 1 yıl önce

If you're looking for a weekend project, how about training your own text-to-speech model from scratch on your own GPU, and then running it on any device's CPU? We just open-sourced the entire Pocket TTS training stack: data pipeline, recipes, and evals. It learns pretty damn fast: ~15k steps: babbling starts turning into words ~50k steps: it reads anything you type (WER under 1%) ~200k steps: the voice stops sounding synthetic On a beefy consumer GPU, that's a week of training. On eight H100s: 10-20 hours. A TTS training run will cost you less than $200 if you rent your hardware, and an order of magnitude less if you just pay for power. Some things we'd love to see people try: - Train it in your own language (a few hundred hours of speech gets you surprisingly far). - Add new features to Pocket TTS (Emotion tags? Make it sing?). - Beat us at our own game: make it faster and smaller. Show us what you build! We'll highlight the best models and new languages for the whole community to enjoy. Pocket TTS has already found many use cases, from reading for people with visual impairments to making NPCs in video games talk, and we're sure there's much more to do with it! Here's an example of a Czech Pocket TTS. Try just asking your favorite agent to find data and apply the method, and you can have your own. Get started:

If you're looking for a weekend project, how about training your own text-to-speech model from scratch on your own GPU, and then running it on any device's CPU? We just open-sourced the entire Pocket TTS training stack: data pipeline, recipes, and evals. It learns pretty damn fast: ~15k steps: babbling starts turning into words ~50k steps: it reads anything you type (WER under 1%) ~200k steps: the voice stops sounding synthetic On a beefy consumer GPU, that's a week of training. On eight H100s: 10-20 hours. A TTS training run will cost you less than $200 if you rent your hardware, and an order of magnitude less if you just pay for power. Some things we'd love to see people try: - Train it in your own language (a few hundred hours of speech gets you surprisingly far). - Add new features to Pocket TTS (Emotion tags? Make it sing?). - Beat us at our own game: make it faster and smaller. Show us what you build! We'll highlight the best models and new languages for the whole community to enjoy. Pocket TTS has already found many use cases, from reading for people with visual impairments to making NPCs in video games talk, and we're sure there's much more to do with it! Here's an example of a Czech Pocket TTS. Try just asking your favorite agent to find data and apply the method, and you can have your own. Get started:

kyutai

103,670 görüntüleme • 9 gün önce

🌍𝗔𝗳𝗿𝗶𝗰𝗮 𝗜𝘀 𝗡𝗼𝘁 𝗪𝗮𝗶𝘁𝗶𝗻𝗴 — 𝗜𝘁’𝘀 𝗦𝗵𝗮𝗽𝗶𝗻𝗴 𝘁𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲! Speaking at the Georgia State University Andrew Young School in the United States, during the MandelaWshFellowship, I carried with me the weight of responsibility and generational urgency for Africa’s future — 𝗮 𝗿𝗶𝘀𝗶𝗻𝗴 𝗰𝗼𝗻𝘁𝗶𝗻𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗶𝘀 𝗻𝗼𝘁 𝘄𝗮𝗶𝘁𝗶𝗻𝗴 𝘁𝗼 𝗯𝗲 𝗶𝗻𝘃𝗶𝘁𝗲𝗱 𝗶𝗻𝘁𝗼 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗯𝘂𝘁 𝗶𝘀 𝗮𝗰𝘁𝗶𝘃𝗲𝗹𝘆 𝘀𝗵𝗮𝗽𝗶𝗻𝗴 𝗶𝘁. ◼️𝗧𝗼 𝗴𝗹𝗼𝗯𝗮𝗹 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗼𝗿𝘀 𝗮𝗻𝗱 𝗶𝗻𝘃𝗲𝘀𝘁𝗼𝗿𝘀: The real ROI lies in co-creation. Africa is not just the next frontier — 𝗶𝘁 𝗶𝘀 𝘆𝗼𝘂𝗿 𝗶𝗻𝗱𝗶𝘀𝗽𝗲𝗻𝘀𝗮𝗯𝗹𝗲 𝗰𝗼-𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁 𝗼𝗳 𝘁𝗵𝗲 𝟮𝟭𝘀𝘁 𝗰𝗲𝗻𝘁𝘂𝗿𝘆. From market stalls transforming into data hubs to farms evolving into predictive engines, young African pioneers are not only reimagining AI’s potential — they are engineering blueprints the world will follow. Partner with us. Invest in our talent pipelines. Scale our homegrown breakthroughs. 𝗧𝗵𝗮𝘁 𝗶𝘀 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗳𝗼𝗿𝗲𝘀𝗶𝗴𝗵𝘁. ◼️𝗧𝗼 𝗔𝗳𝗿𝗶𝗰𝗮’𝘀 𝗯𝘂𝗶𝗹𝗱𝗲𝗿𝘀: Your canvas is the world. Code with purpose. Build with vision. Lead with audacity. Scale your vision globally. The world isn’t waiting 𝗳𝗼𝗿 𝘆𝗼𝘂 —it’s waiting 𝗼𝗻 𝘆𝗼𝘂. Don’t catch up—leap ahead. Define what’s next. This is the future we are building at Otic Group. Our 𝗲𝘁𝗵𝗼𝘀 is simple: 𝗧𝗵𝗲 𝗺𝗼𝘀𝘁 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁, 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝗶𝘀 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝗶𝗻 𝘂𝘀. Our work isn't about importing solutions; it's about unlocking that innate brilliance. Do that, and you don't just change a market—you change the century!

🌍𝗔𝗳𝗿𝗶𝗰𝗮 𝗜𝘀 𝗡𝗼𝘁 𝗪𝗮𝗶𝘁𝗶𝗻𝗴 — 𝗜𝘁’𝘀 𝗦𝗵𝗮𝗽𝗶𝗻𝗴 𝘁𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲! Speaking at the Georgia State University Andrew Young School in the United States, during the MandelaWshFellowship, I carried with me the weight of responsibility and generational urgency for Africa’s future — 𝗮 𝗿𝗶𝘀𝗶𝗻𝗴 𝗰𝗼𝗻𝘁𝗶𝗻𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗶𝘀 𝗻𝗼𝘁 𝘄𝗮𝗶𝘁𝗶𝗻𝗴 𝘁𝗼 𝗯𝗲 𝗶𝗻𝘃𝗶𝘁𝗲𝗱 𝗶𝗻𝘁𝗼 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗯𝘂𝘁 𝗶𝘀 𝗮𝗰𝘁𝗶𝘃𝗲𝗹𝘆 𝘀𝗵𝗮𝗽𝗶𝗻𝗴 𝗶𝘁. ◼️𝗧𝗼 𝗴𝗹𝗼𝗯𝗮𝗹 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗼𝗿𝘀 𝗮𝗻𝗱 𝗶𝗻𝘃𝗲𝘀𝘁𝗼𝗿𝘀: The real ROI lies in co-creation. Africa is not just the next frontier — 𝗶𝘁 𝗶𝘀 𝘆𝗼𝘂𝗿 𝗶𝗻𝗱𝗶𝘀𝗽𝗲𝗻𝘀𝗮𝗯𝗹𝗲 𝗰𝗼-𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁 𝗼𝗳 𝘁𝗵𝗲 𝟮𝟭𝘀𝘁 𝗰𝗲𝗻𝘁𝘂𝗿𝘆. From market stalls transforming into data hubs to farms evolving into predictive engines, young African pioneers are not only reimagining AI’s potential — they are engineering blueprints the world will follow. Partner with us. Invest in our talent pipelines. Scale our homegrown breakthroughs. 𝗧𝗵𝗮𝘁 𝗶𝘀 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗳𝗼𝗿𝗲𝘀𝗶𝗴𝗵𝘁. ◼️𝗧𝗼 𝗔𝗳𝗿𝗶𝗰𝗮’𝘀 𝗯𝘂𝗶𝗹𝗱𝗲𝗿𝘀: Your canvas is the world. Code with purpose. Build with vision. Lead with audacity. Scale your vision globally. The world isn’t waiting 𝗳𝗼𝗿 𝘆𝗼𝘂 —it’s waiting 𝗼𝗻 𝘆𝗼𝘂. Don’t catch up—leap ahead. Define what’s next. This is the future we are building at Otic Group. Our 𝗲𝘁𝗵𝗼𝘀 is simple: 𝗧𝗵𝗲 𝗺𝗼𝘀𝘁 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁, 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝗶𝘀 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝗶𝗻 𝘂𝘀. Our work isn't about importing solutions; it's about unlocking that innate brilliance. Do that, and you don't just change a market—you change the century!

Nesta Paul Katende

167,546 görüntüleme • 1 yıl önce

Your phone can build voxel world now! 👾 This is an open-source tool that converts phone video into 3D point clouds. You can run Depth Anything 3 locally. Basically just local GPU inference. The interesting part is how it handles messy input. Video windows overlap, scale gets converted to real meters, drift gets corrected after inference. The system stitches everything together even when footage is noisy. Objects far away might fail strict confidence filters but still show up as recognizable buildings. That's huge for small robots because perception becomes cheap and stays local. Obviously it wouldn't be a phone for robots, but it shows the potential of edge/local perception processing. A robot can understand its environment without offloading to the cloud. Real-world video has artifacts and occlusions. The system doesn't need perfect input to reconstruct scene structure. It works with what's there. For robotics, this means small platforms can build 3D maps autonomously using hardware they already have. Here's the repo: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Your phone can build voxel world now! 👾 This is an open-source tool that converts phone video into 3D point clouds. You can run Depth Anything 3 locally. Basically just local GPU inference. The interesting part is how it handles messy input. Video windows overlap, scale gets converted to real meters, drift gets corrected after inference. The system stitches everything together even when footage is noisy. Objects far away might fail strict confidence filters but still show up as recognizable buildings. That's huge for small robots because perception becomes cheap and stays local. Obviously it wouldn't be a phone for robots, but it shows the potential of edge/local perception processing. A robot can understand its environment without offloading to the cloud. Real-world video has artifacts and occlusions. The system doesn't need perfect input to reconstruct scene structure. It works with what's there. For robotics, this means small platforms can build 3D maps autonomously using hardware they already have. Here's the repo: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,569 görüntüleme • 1 ay önce

I got to ask Jensen a question today at CES 2026. Question: What advice would you give to a new robotics founder for them to choose the right application space or the right idea so that they can have the most impact and most differentiation? Jensen's answer (short summary): The real strategic choice is between a horizontal play and a vertical one. Horizontal competition comes from every direction, but focusing on a specific vertical allows you to solve the hardest problems for a specific industry. Whether it’s EMS manufacturing or surgical robotics, that deep domain expertise is very beneficial. Jensen's full answer: Well, first of all, let's take a step back. As you know, NVIDIA, here we were just talking about AI factories. And that AI factory—our contribution, our chips, systems, infrastructure, which is software, and model technology. Is that right? That's kind of the NVIDIA stack. And that's an AI factory. In order to build robotic systems, you really need three different computers. You need the training computer, which I just described. And then you need another computer for doing simulation, because the robot needs to learn how to practice and be evaluated inside a virtual world that's physically precise, so that it doesn't have to do crazy stuff in the physical world while it's still learning. And so we create a virtual world that obeys the laws of physics, and I've demonstrated it several times, and that virtual world is called Omniverse. And so that's a second computer. And that computer is much more like one of our gaming computers, and the GPU that we use for that is RTX Pro. Basically an RTX. And then the third computer is the computer that goes into the robot. It's the robot brain. And that robot computer we call Orin today, and then the next generation is called Thor. And it has its own stack. So Thor has a super-fast inference stack. It runs a safety operating system, like the safety operating system we have in the car, because you want the robot to stay in its guardrails and not do things that it's not confident in doing. And so, you have your stack, and then you have your model. And the model could be fine articulation, manipulation, and locomotion, and each one of those, it could be system one and system two thinking. The technology necessary to build a robot is incredible. And so I just described for you, in order to be a robotics company, you have to have three computers. You have to understand all three stacks, and you have to build this robotic system, not to mention all the electronics and the mechanicals necessary to do it. It's incredibly hard. However, as you know, these pieces of technology independently have been coming together. Isn't that right? Which is really what happens to a new industry, is when there's enabling technology necessary for the industry itself, but it rides on the contributions of the technology advances in other industry that it doesn't have to worry about. And so the humanoid industry is riding on the work of the AI factories we're building for other mainstream stuff and other AI stuff. And our Omniverse was designed for other applications and different other digital twin capabilities. And so all this stuff is now coming together. The question is for robotics, ultimately, it comes down to a couple of different questions. Do you want to be a horizontal company, or do you want to be a vertically domain-specific company? The benefit of a horizontal company, of course, is that you less worry about the application, you more worry about the technology, and if you succeed, your scale can be quite large. However, horizontal plays are incredibly hard. Your competition comes from every single direction. Now, on domain, if you want to be domain-specific, then you're going to have to understand the particular application quite deeply. So maybe it's something to do with EMS manufacturing, assembly of these AI supercomputers. Maybe it's related to building cars in factories. Whatever the reasons are, your domain expertise—could be surgical robots—domain expertise could really be a benefit. My preference usually—and you asked, so I'll offer—my preference tends to be to go find verticals, but that's kind of my preference. Some companies, some leaders just would prefer to build horizontal capability, and that's fine, too.

I got to ask Jensen a question today at CES 2026. Question: What advice would you give to a new robotics founder for them to choose the right application space or the right idea so that they can have the most impact and most differentiation? Jensen's answer (short summary): The real strategic choice is between a horizontal play and a vertical one. Horizontal competition comes from every direction, but focusing on a specific vertical allows you to solve the hardest problems for a specific industry. Whether it’s EMS manufacturing or surgical robotics, that deep domain expertise is very beneficial. Jensen's full answer: Well, first of all, let's take a step back. As you know, NVIDIA, here we were just talking about AI factories. And that AI factory—our contribution, our chips, systems, infrastructure, which is software, and model technology. Is that right? That's kind of the NVIDIA stack. And that's an AI factory. In order to build robotic systems, you really need three different computers. You need the training computer, which I just described. And then you need another computer for doing simulation, because the robot needs to learn how to practice and be evaluated inside a virtual world that's physically precise, so that it doesn't have to do crazy stuff in the physical world while it's still learning. And so we create a virtual world that obeys the laws of physics, and I've demonstrated it several times, and that virtual world is called Omniverse. And so that's a second computer. And that computer is much more like one of our gaming computers, and the GPU that we use for that is RTX Pro. Basically an RTX. And then the third computer is the computer that goes into the robot. It's the robot brain. And that robot computer we call Orin today, and then the next generation is called Thor. And it has its own stack. So Thor has a super-fast inference stack. It runs a safety operating system, like the safety operating system we have in the car, because you want the robot to stay in its guardrails and not do things that it's not confident in doing. And so, you have your stack, and then you have your model. And the model could be fine articulation, manipulation, and locomotion, and each one of those, it could be system one and system two thinking. The technology necessary to build a robot is incredible. And so I just described for you, in order to be a robotics company, you have to have three computers. You have to understand all three stacks, and you have to build this robotic system, not to mention all the electronics and the mechanicals necessary to do it. It's incredibly hard. However, as you know, these pieces of technology independently have been coming together. Isn't that right? Which is really what happens to a new industry, is when there's enabling technology necessary for the industry itself, but it rides on the contributions of the technology advances in other industry that it doesn't have to worry about. And so the humanoid industry is riding on the work of the AI factories we're building for other mainstream stuff and other AI stuff. And our Omniverse was designed for other applications and different other digital twin capabilities. And so all this stuff is now coming together. The question is for robotics, ultimately, it comes down to a couple of different questions. Do you want to be a horizontal company, or do you want to be a vertically domain-specific company? The benefit of a horizontal company, of course, is that you less worry about the application, you more worry about the technology, and if you succeed, your scale can be quite large. However, horizontal plays are incredibly hard. Your competition comes from every single direction. Now, on domain, if you want to be domain-specific, then you're going to have to understand the particular application quite deeply. So maybe it's something to do with EMS manufacturing, assembly of these AI supercomputers. Maybe it's related to building cars in factories. Whatever the reasons are, your domain expertise—could be surgical robots—domain expertise could really be a benefit. My preference usually—and you asked, so I'll offer—my preference tends to be to go find verticals, but that's kind of my preference. Some companies, some leaders just would prefer to build horizontal capability, and that's fine, too.

The Humanoid Hub

40,043 görüntüleme • 7 ay önce

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.

Andy Zeng

49,443 görüntüleme • 11 ay önce

i don't think people realize what's happening in Chinese robotics. this one manufacturer might be the most impressive AND most concerning company on Earth right now let me explain... Unitree Robotics sells a humanoid robot for $5,900. their robot dog costs $1,600 (Boston Dynamics charges $74,500 for theirs for context). you can literally buy these on Amazon today. so obviously the first question is: how is that even possible? the answer starts with a guy who couldn't pass his English exam. Wang Xingxing grew up in Zhejiang province. for his master's thesis, he decided to build a quadruped robot. budget: about $3,000. for context, $3,000 for this kinda robot is nothing. off-the-shelf servo motors alone would've eaten that twice over. so Wang did the only thing he could: he designed and machined every single component himself. motors, joints, controllers, the frame. all of it. the resulting robot was janky and imperfect. but it worked. and the video went viral globally. after graduating he joined DJI. but he quit after two months, and this is 2016, when DJI was arguably the hottest hardware company in China. walking away from that with no money to start a robotics company is a... specific kind of stubborn. he launches Unitree with $280K from a single angel investor. tiny office in Hangzhou. 50 square meters. but the money runs out fast. he can't make payroll for three years. the company almost dies in 2017. but emergency government funding arrives with days to spare. he survives, barely, and keeps building. this is where it gets really fascinating IMO. this founding constraint, building everything yourself because you literally cannot afford to buy parts, never went away. even after funding rounds started landing. even after revenue kicked in. it just became the company's permanent DNA. Unitree now manufactures 90%+ of its core components in-house. motors, reducers, controllers, encoders, LiDAR, etc the founder's $3,000 robot thesis ended up being an architectural decision that turned out to be structurally superior. think about what that means in practice. Boston Dynamics needs a better motor? they negotiate with a supplier, wait on lead times, qualify the part. but when Unitree needs one, they design theirs internally and have a new version in production within weeks. that gap compounds every cycle. Unitree shipped three separate humanoid platforms in 18 months. Figure AI has shipped one. Tesla has shipped zero commercially. the results are getting hard to dismiss. 23,700 robot dogs shipped in 2024 (roughly 70% of the entire global market). 7,000+ humanoids deployed. over 600 industrial sites running their quadrupeds. $140M+ revenue, profitable every year since 2020. for perspective: no Western humanoid competitor is profitable. not one. OK. now here's where the "most concerning" part of this starts... if you watched the DJI story unfold, you already recognize the shape. affordable Chinese hardware quietly saturates global markets. years later, the national security questions arrive, after the install base is already massive. drones, then EVs, then AI. now robots. Unitree is running this exact playbook in real time. in April 2025, researchers found an undocumented backdoor in their Go1 robot. a remote tunnel letting anyone control the robot and stream its camera feed. default password: pi/123. 1,919 vulnerable units exposed globally. including machines at MIT, Princeton, and Carnegie Mellon. but it gets worse. every Unitree robot shares the same hardcoded encryption key. encrypt the word "unitree" and you get root access to any of them. one compromised robot can spread to every Unitree robot in Bluetooth range automatically. a literal robot botnet. the G1 quietly transmits sensor data to Chinese servers every five minutes. audio, video, GPS, LiDAR spatial mapping, with no notification, no consent, no opt-out. PLA footage has shown Go2 robots with mounted weapons. Ukrainian forces literally deployed weaponized units on the actual frontline. and every member of the bipartisan House China Committee signed a letter calling for Unitree's military company designation. Wang signed a 2022 pledge alongside Boston Dynamics not to weaponize robots. but pledges don't survive contact with shipping hardware to open markets. and under China's 2025 rules restricting military-related speech, Unitree couldn't publicly confirm PLA use even if they wanted to. 50,000+ of these robots are now deployed globally. some at institutions that probably should've asked harder questions before connecting them to their networks. the security stuff is real and people should know about it. but i also think it's important not to let that overshadow what's actually been built here. a 35-year-old who failed his English exam created a robotics company that's outshipping and outpricing every Western competitor while being the only profitable humanoid maker on Earth. most impressive and most concerning company in the world right now.

i don't think people realize what's happening in Chinese robotics. this one manufacturer might be the most impressive AND most concerning company on Earth right now let me explain... Unitree Robotics sells a humanoid robot for $5,900. their robot dog costs $1,600 (Boston Dynamics charges $74,500 for theirs for context). you can literally buy these on Amazon today. so obviously the first question is: how is that even possible? the answer starts with a guy who couldn't pass his English exam. Wang Xingxing grew up in Zhejiang province. for his master's thesis, he decided to build a quadruped robot. budget: about $3,000. for context, $3,000 for this kinda robot is nothing. off-the-shelf servo motors alone would've eaten that twice over. so Wang did the only thing he could: he designed and machined every single component himself. motors, joints, controllers, the frame. all of it. the resulting robot was janky and imperfect. but it worked. and the video went viral globally. after graduating he joined DJI. but he quit after two months, and this is 2016, when DJI was arguably the hottest hardware company in China. walking away from that with no money to start a robotics company is a... specific kind of stubborn. he launches Unitree with $280K from a single angel investor. tiny office in Hangzhou. 50 square meters. but the money runs out fast. he can't make payroll for three years. the company almost dies in 2017. but emergency government funding arrives with days to spare. he survives, barely, and keeps building. this is where it gets really fascinating IMO. this founding constraint, building everything yourself because you literally cannot afford to buy parts, never went away. even after funding rounds started landing. even after revenue kicked in. it just became the company's permanent DNA. Unitree now manufactures 90%+ of its core components in-house. motors, reducers, controllers, encoders, LiDAR, etc the founder's $3,000 robot thesis ended up being an architectural decision that turned out to be structurally superior. think about what that means in practice. Boston Dynamics needs a better motor? they negotiate with a supplier, wait on lead times, qualify the part. but when Unitree needs one, they design theirs internally and have a new version in production within weeks. that gap compounds every cycle. Unitree shipped three separate humanoid platforms in 18 months. Figure AI has shipped one. Tesla has shipped zero commercially. the results are getting hard to dismiss. 23,700 robot dogs shipped in 2024 (roughly 70% of the entire global market). 7,000+ humanoids deployed. over 600 industrial sites running their quadrupeds. $140M+ revenue, profitable every year since 2020. for perspective: no Western humanoid competitor is profitable. not one. OK. now here's where the "most concerning" part of this starts... if you watched the DJI story unfold, you already recognize the shape. affordable Chinese hardware quietly saturates global markets. years later, the national security questions arrive, after the install base is already massive. drones, then EVs, then AI. now robots. Unitree is running this exact playbook in real time. in April 2025, researchers found an undocumented backdoor in their Go1 robot. a remote tunnel letting anyone control the robot and stream its camera feed. default password: pi/123. 1,919 vulnerable units exposed globally. including machines at MIT, Princeton, and Carnegie Mellon. but it gets worse. every Unitree robot shares the same hardcoded encryption key. encrypt the word "unitree" and you get root access to any of them. one compromised robot can spread to every Unitree robot in Bluetooth range automatically. a literal robot botnet. the G1 quietly transmits sensor data to Chinese servers every five minutes. audio, video, GPS, LiDAR spatial mapping, with no notification, no consent, no opt-out. PLA footage has shown Go2 robots with mounted weapons. Ukrainian forces literally deployed weaponized units on the actual frontline. and every member of the bipartisan House China Committee signed a letter calling for Unitree's military company designation. Wang signed a 2022 pledge alongside Boston Dynamics not to weaponize robots. but pledges don't survive contact with shipping hardware to open markets. and under China's 2025 rules restricting military-related speech, Unitree couldn't publicly confirm PLA use even if they wanted to. 50,000+ of these robots are now deployed globally. some at institutions that probably should've asked harder questions before connecting them to their networks. the security stuff is real and people should know about it. but i also think it's important not to let that overshadow what's actually been built here. a 35-year-old who failed his English exam created a robotics company that's outshipping and outpricing every Western competitor while being the only profitable humanoid maker on Earth. most impressive and most concerning company in the world right now.

Ole Lehmann

122,620 görüntüleme • 6 ay önce