Загрузка видео...

Не удалось загрузить видео

На главную

AI for robotics has some monumental challenges, like high dimensionality and reasoning with the physical world. Ashok Elluswamy, AI lead at Tesla, thinks the hardest problem is Evaluation. - Loss isn't a perfect indicator of Policy Neural Network's quality. - Testing in the real world for all the long-tail...

TheHumanoidHub's profile picture

The Humanoid Hub

110,123 subscribers

16,591 просмотров • 5 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,900 просмотров • 8 месяцев назад

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation
2:52

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation

AGIBOT

28,219 просмотров • 3 месяцев назад

NEWS: Waymo has introduced the Waymo World Model, "a frontier generative model built on Google DeepMind’s Genie 3 that sets a new bar for large-scale, hyper-realistic autonomous driving simulation." "By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. This combination of broad world knowledge, fine-grained controllability, and multi-modal realism enhances Waymo’s ability to safely scale our service across more places and new driving environments. In the following sections we showcase the Waymo World Model in action, featuring simulations of the Waymo Driver navigating diverse rare edge-case scenarios." More info below.
1:03

NEWS: Waymo has introduced the Waymo World Model, "a frontier generative model built on Google DeepMind’s Genie 3 that sets a new bar for large-scale, hyper-realistic autonomous driving simulation." "By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. This combination of broad world knowledge, fine-grained controllability, and multi-modal realism enhances Waymo’s ability to safely scale our service across more places and new driving environments. In the following sections we showcase the Waymo World Model in action, featuring simulations of the Waymo Driver navigating diverse rare edge-case scenarios." More info below.

Sawyer Merritt

133,168 просмотров • 5 месяцев назад

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.
0:40

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 просмотров • 4 месяцев назад

Robotics has a massive, silent bottleneck. It isn’t just data collection—it’s the brutal 1x speed of the physical world. Genesis AI Genesis AI just unveiled Genesis World 1.0, and they are attempting to turn the notorious Sim2Real gap into a pure compute problem. Evaluating a robotics foundation model across edge cases usually means hundreds of hours of physical lab testing. With Genesis World 1.0, what traditionally takes nearly a week of continuous, real-world operation is being compressed into 30 minutes in simulation. What makes this different from just dropping a robot model into an off-the-shelf game engine? 1️⃣ Nyx Renderer: A custom, real-time path-traced engine rendering noise-free 1080p frames in under 4ms. Game engines use rasterization tricks that confuse AI; Nyx uses physically accurate multi-bounce lighting so the model's "eyes" see exactly what real sensors see. 2️⃣ Quadrants Compiler: A custom Python-to-GPU compiler to run heavily parallelized multi-physics simulations (rigid bodies, fluids, deformables) natively across architectures. 3️⃣ Evaluation First: They aren't rushing to train on synthetic data. They are using this purely for closed-loop evaluation to perfect the physics first, currently claiming an impressive 89% correlation with real-world hardware tests. If the industry can accurately evaluate models in simulation without the physical world bottleneck, humanoid development stops moving at wall-clock time and starts scaling with compute.
1:36

Robotics has a massive, silent bottleneck. It isn’t just data collection—it’s the brutal 1x speed of the physical world. Genesis AI Genesis AI just unveiled Genesis World 1.0, and they are attempting to turn the notorious Sim2Real gap into a pure compute problem. Evaluating a robotics foundation model across edge cases usually means hundreds of hours of physical lab testing. With Genesis World 1.0, what traditionally takes nearly a week of continuous, real-world operation is being compressed into 30 minutes in simulation. What makes this different from just dropping a robot model into an off-the-shelf game engine? 1️⃣ Nyx Renderer: A custom, real-time path-traced engine rendering noise-free 1080p frames in under 4ms. Game engines use rasterization tricks that confuse AI; Nyx uses physically accurate multi-bounce lighting so the model's "eyes" see exactly what real sensors see. 2️⃣ Quadrants Compiler: A custom Python-to-GPU compiler to run heavily parallelized multi-physics simulations (rigid bodies, fluids, deformables) natively across architectures. 3️⃣ Evaluation First: They aren't rushing to train on synthetic data. They are using this purely for closed-loop evaluation to perfect the physics first, currently claiming an impressive 89% correlation with real-world hardware tests. If the industry can accurately evaluate models in simulation without the physical world bottleneck, humanoid development stops moving at wall-clock time and starts scaling with compute.

Humanoids daily

17,240 просмотров • 1 месяц назад

Self-learning worked in AlphaGo because the world was a clean rulebook, says Demis Hassabis. Real life has no perfect rulebook, so long-term planning needs a strong “world model” to simulate what happens next. Without that simulation, a robot has to guess in the real world—where mistakes can be costly and irreversible. Better world models are the first missing piece, says Demis Hassabis.
1:38

Self-learning worked in AlphaGo because the world was a clean rulebook, says Demis Hassabis. Real life has no perfect rulebook, so long-term planning needs a strong “world model” to simulate what happens next. Without that simulation, a robot has to guess in the real world—where mistakes can be costly and irreversible. Better world models are the first missing piece, says Demis Hassabis.

Wes Roth

15,757 просмотров • 5 месяцев назад

The next frontier of autonomous driving is unlocked by reasoning models. NVIDIA Alpamayo brings together open AI models with reasoning capabilities, closed-loop simulation tools, and massive real-world driving datasets. Alpamayo 1 is a vision–language–action model that explains its own decisions through explicit reasoning traces, enabling trustworthy, humanlike decision-making. Together with NVIDIA’s Physical AI dataset and AlpaSim simulation, Alpamayo provides the tools and scale required to enable level 4 autonomous vehicles. ▶️ Watch now:
0:45

The next frontier of autonomous driving is unlocked by reasoning models. NVIDIA Alpamayo brings together open AI models with reasoning capabilities, closed-loop simulation tools, and massive real-world driving datasets. Alpamayo 1 is a vision–language–action model that explains its own decisions through explicit reasoning traces, enabling trustworthy, humanlike decision-making. Together with NVIDIA’s Physical AI dataset and AlpaSim simulation, Alpamayo provides the tools and scale required to enable level 4 autonomous vehicles. ▶️ Watch now:

NVIDIA DRIVE

35,324 просмотров • 6 месяцев назад

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.
0:29

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

Xueyan Zou

38,164 просмотров • 1 месяц назад

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.
1:52

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.

Humanoids daily

12,391 просмотров • 3 месяцев назад

It’s easy to watch this video and see a pre-recorded clip, but what you’re really watching is a frame-by-frame, interactive simulation of the world. Odyssey-2 Pro—a general-purpose world model—is prompted and then rolls out a minutes-long, open-ended simulation.
1:41

It’s easy to watch this video and see a pre-recorded clip, but what you’re really watching is a frame-by-frame, interactive simulation of the world. Odyssey-2 Pro—a general-purpose world model—is prompted and then rolls out a minutes-long, open-ended simulation.

Odyssey

32,420 просмотров • 7 месяцев назад

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 просмотров • 4 месяцев назад

NVIDIA CEO Jensen Huang on Tesla FSD at CES: “Tesla’s FSD stack is completely world-class. They’ve been working on it for quite some time. It’s world-class. Not only in the number of miles it has, but it’s world-class in the way that it’s designed, the way they do training to data collection and curation, synthetic data generation, all of their simulation technologies. Of course, the latest generation is end-to-end full self-driving, meaning it’s just one large model that is end-to-end trained. Elon’s AV system is, in every way, 100% state-of-the-art. And so I’m really quite impressed by the technology. I have it and I drive it in our house and it works incredibly well.”
1:11

NVIDIA CEO Jensen Huang on Tesla FSD at CES: “Tesla’s FSD stack is completely world-class. They’ve been working on it for quite some time. It’s world-class. Not only in the number of miles it has, but it’s world-class in the way that it’s designed, the way they do training to data collection and curation, synthetic data generation, all of their simulation technologies. Of course, the latest generation is end-to-end full self-driving, meaning it’s just one large model that is end-to-end trained. Elon’s AV system is, in every way, 100% state-of-the-art. And so I’m really quite impressed by the technology. I have it and I drive it in our house and it works incredibly well.”

Nic Cruz Patane

52,826 просмотров • 6 месяцев назад

I improved the VRChat cloth simulation, now it is actually rendered as a mesh and finally can handle self collision for proper folds. Also I moved the simulation to CRTs so it kinda works for non friends(the simulation is on the avatar!). But the collision still requires cameras.
1:49

I improved the VRChat cloth simulation, now it is actually rendered as a mesh and finally can handle self collision for proper folds. Also I moved the simulation to CRTs so it kinda works for non friends(the simulation is on the avatar!). But the collision still requires cameras.

Mykhailo Moroz

363,768 просмотров • 3 лет назад

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->
1:15

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

33,766 просмотров • 1 месяц назад

Robot trained entirely in simulation! 🤯 Sudo AI just released #sudo R1. This could be the moment the data bottleneck in robotics finally breaks, a major step forward for getting simulation-trained robots to actually work in the real world. #sudo R1 is a manipulation foundation model that reliably picks objects it has never seen: glass, fabric, metal, irregular shapes. All with zero real-world training data. No tele-op. No demos. No manual labeling. Pure simulation. And picking isn't just any skill, it's the foundation. Almost everything we want robots to do starts with being able to reliably pick something up. Data collection is the bottleneck holding the whole field back. Every new task means hours of tele-op and armies of labelers. If simulation alone can actually deliver production-grade picking, that bottleneck starts to disappear. Really exciting to see! If the results hold, the path to general-purpose robots just got a whole lot shorter. 👇 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:59

Robot trained entirely in simulation! 🤯 Sudo AI just released #sudo R1. This could be the moment the data bottleneck in robotics finally breaks, a major step forward for getting simulation-trained robots to actually work in the real world. #sudo R1 is a manipulation foundation model that reliably picks objects it has never seen: glass, fabric, metal, irregular shapes. All with zero real-world training data. No tele-op. No demos. No manual labeling. Pure simulation. And picking isn't just any skill, it's the foundation. Almost everything we want robots to do starts with being able to reliably pick something up. Data collection is the bottleneck holding the whole field back. Every new task means hours of tele-op and armies of labelers. If simulation alone can actually deliver production-grade picking, that bottleneck starts to disappear. Really exciting to see! If the results hold, the path to general-purpose robots just got a whole lot shorter. 👇 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

25,718 просмотров • 2 месяцев назад

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.
16:27

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.

abdel

16,916 просмотров • 2 месяцев назад

ATTENTION: The World Health Organization, John Hopkins University and the Bill & Melinda Gates Foundation completed a new pandemic simulation back in October of 2022 in Belgium, Germany. These are the same exact people who ran event 201, which was a pandemic simulation they ran for Covid six months before the real Covid Pandemic hit. According to the simulation this new pandemic will originate in Brazil in 2025 and the virus is called #SEERS (Severe, Epidemic, Enterovirus, Respiratory, Syndrome). Reports in the simulation stated there’d be a billion cases worldwide with 20 million deaths, including 15 million children, with countless millions left with paralysis. SEERS is an enterovirus, like Polio, that can not only effect the nerves in the spinal cord, and cause paralysis, but it can also effect the brain stem, and make people brain dead. According to the simulation we have approx a year and half to get as ready as possible. Plan accordingly. #pandemic #pandemic2025 #Genocide
6:09

ATTENTION: The World Health Organization, John Hopkins University and the Bill & Melinda Gates Foundation completed a new pandemic simulation back in October of 2022 in Belgium, Germany. These are the same exact people who ran event 201, which was a pandemic simulation they ran for Covid six months before the real Covid Pandemic hit. According to the simulation this new pandemic will originate in Brazil in 2025 and the virus is called #SEERS (Severe, Epidemic, Enterovirus, Respiratory, Syndrome). Reports in the simulation stated there’d be a billion cases worldwide with 20 million deaths, including 15 million children, with countless millions left with paralysis. SEERS is an enterovirus, like Polio, that can not only effect the nerves in the spinal cord, and cause paralysis, but it can also effect the brain stem, and make people brain dead. According to the simulation we have approx a year and half to get as ready as possible. Plan accordingly. #pandemic #pandemic2025 #Genocide

Lion of Judah

581,666 просмотров • 3 лет назад

2. NVIDIA is building the GPT of humanoid robots. They just launched Isaac GR00T N1.5 - a foundation model for general purpose robotics. Here’s how it works: → A human demos the task once → Cosmos (their physics AI model) generates 1,000s of variations → Omniverse simulates the motions in high fidelity → The robot trains entirely in simulation → Then fine-tunes itself in the real world Robots can now learn general skills across tasks, tools, even body types with just one human demo. AI isn’t just thinking in text anymore. It’s perceiving. Reasoning. Moving. Physical AI is here and it’s training itself.
2:21

2. NVIDIA is building the GPT of humanoid robots. They just launched Isaac GR00T N1.5 - a foundation model for general purpose robotics. Here’s how it works: → A human demos the task once → Cosmos (their physics AI model) generates 1,000s of variations → Omniverse simulates the motions in high fidelity → The robot trains entirely in simulation → Then fine-tunes itself in the real world Robots can now learn general skills across tasks, tools, even body types with just one human demo. AI isn’t just thinking in text anymore. It’s perceiving. Reasoning. Moving. Physical AI is here and it’s training itself.

Shruti

70,359 просмотров • 1 год назад

To push self-driving into situations wilder than reality, we built a neural network world simulator that can create entirely synthetic worlds for the Tesla to drive in. Video below is fully generated & not a real video
1:15

To push self-driving into situations wilder than reality, we built a neural network world simulator that can create entirely synthetic worlds for the Tesla to drive in. Video below is fully generated & not a real video

Tesla

1,698,151 просмотров • 8 месяцев назад

Physical AI is here and it’s scaling fast. 👏 From simulation of robotics and autonomous systems, a full stack of advanced libraries is powering the next generation of intelligent, real-world machines. Watch the #NVIDIAGTC montage: Catch up on all Physical AI Days sessions:
0:26

Physical AI is here and it’s scaling fast. 👏 From simulation of robotics and autonomous systems, a full stack of advanced libraries is powering the next generation of intelligent, real-world machines. Watch the #NVIDIAGTC montage: Catch up on all Physical AI Days sessions:

NVIDIA Omniverse

12,459 просмотров • 3 месяцев назад