正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Fundamental spatial and temporal understanding is the bedrock upon which robots will learn motor control. These types of Embodied Reasoning capabilities enable policy learning but also inference abilities like image or video conditioning. We just released a new SOTA ER model!

Ted Xiao

136,119 subscribers

40,855 次观看 • 11 个月前 •via X (Twitter)

科学技术教育艺术

Anya Rossi• Live Now

Private livecam show

9 条评论

Ted Xiao 的头像

Ted Xiao11 个月前

Our team has pushed the Gemini Robotics-ER 1.5 model to full production, where anyone in the world can unlock world class Embodied Reasoning at Flash-scale cost and speed. Try it today!

Dhruv Diddi 的头像

Dhruv Diddi11 个月前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 Nice! Open source?

Priyesh Gandhi 的头像

Priyesh Gandhi4 个月前

ER is the missing middleware between perception and policy. This framing unlocks so much. One observation from the data side: ER models scale with the diversity of spatial/temporal contexts they've seen, not just volume. Most existing robot datasets are the same 5 labs filmed 10,000 times. We're collecting egocentric manipulation across thousands of physically different environments to exactly this thesis. 5K+ hrs and growing.

Noiz Ai 的头像

Noiz Ai2 天前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 The spatial and temporal point feels important here. A model can recognize a scene once, but control depends on keeping the state coherent as the robot acts and the view changes.

Min Chon Chi 的头像

Min Chon Chi11 个月前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 Embodied reasoning for video conditioning sounds promising.

web3石榴酱^^ 的头像

web3石榴酱^^4 个月前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 Hi Ted, we sent you a collaboration invite via DM. Please check your inbox when you have a moment.

Meng Meng 的头像

Meng Meng11 个月前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 This is so exciting!! Would you guys consider doing user requested demo videos or a twitch-plays-Pokémon style stream? E.g. replenishing convenience store drinks but on a smaller scale. The Japanese drink shelves are slanted down and are restocked from the back

Kylie 的头像

Kylie22 天前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 Hi Ted, I’d love to collaborate on a potential piece. Could you check your DMs when you have a moment? Thank you.

黄强的头像

黄强3 个月前

@xf1280 @lgraesser3 @ks_konyushkova @m0hitsharma @jackyliang42 Hi，我是UBTECH这边的高招经理，目前我们正在全球范围寻找具身智能首席科学家，我们的薪资范围在1500W-1.24亿，期待您的来信，详细交流我的邮箱是[email protected]

相关视频

Sam Altman says AI reasoning is still at the GPT-2 stage but the improvement curve is steep and the new o1 model represents a new paradigm of AI development which will enable rapid progress in capabilities

Sam Altman says AI reasoning is still at the GPT-2 stage but the improvement curve is steep and the new o1 model represents a new paradigm of AI development which will enable rapid progress in capabilities

Tsarathustra

419,003 次观看 • 2 年前

PaLM-E is the largest VLM reported to date. We observe emergent capabilities like multimodal chain of thought reasoning, and multi-image inference, despite being trained on only single-image prompts. Though not the focus of our work, PaLM-E sets a new SOTA on OK-VQA benchmark.

PaLM-E is the largest VLM reported to date. We observe emergent capabilities like multimodal chain of thought reasoning, and multi-image inference, despite being trained on only single-image prompts. Though not the focus of our work, PaLM-E sets a new SOTA on OK-VQA benchmark.

Danny Driess

12,291 次观看 • 3 年前

We're releasing HY-Embodied-0.5, a family of foundation models for real-world embodied agents. The 2B model is now open source. It strengthens spatial-temporal perception and embodied reasoning for prediction, interaction, and planning. 🤖 The suite includes: 🔹 2B for edge deployment 🔹 32B for complex reasoning Key innovations: 🔹 Mixture-of-Transformers (MoT) architecture for modality-specific computation 🔹 Latent tokens for improved perceptual representation 🔹 Self-evolving post-training 🔹 On-policy distillation from large to small models Across 22 benchmarks, the 2B model outperforms similarly sized SOTA systems on 16 tasks. The 32B model approaches frontier-level performance. 🔗 GitHub: 🤗 Hugging Face:

We're releasing HY-Embodied-0.5, a family of foundation models for real-world embodied agents. The 2B model is now open source. It strengthens spatial-temporal perception and embodied reasoning for prediction, interaction, and planning. 🤖 The suite includes: 🔹 2B for edge deployment 🔹 32B for complex reasoning Key innovations: 🔹 Mixture-of-Transformers (MoT) architecture for modality-specific computation 🔹 Latent tokens for improved perceptual representation 🔹 Self-evolving post-training 🔹 On-policy distillation from large to small models Across 22 benchmarks, the 2B model outperforms similarly sized SOTA systems on 16 tasks. The 32B model approaches frontier-level performance. 🔗 GitHub: 🤗 Hugging Face:

Tencent Hy

34,317 次观看 • 5 个月前

EXCLUSIVE TOUR OF NEW HUMANOID ROBOT STARTUP: ChatGPT, after watching this video tour of Foundation Robotics's CEO/founder, Sankaet, tell me the state of the art on humanoid robots, wrote this: "Wow, what a deep dive—killer interview. You got a full masterclass in next-gen robotics and AI from people who are not only building but deploying humanoid robots in the real world. The fact that Foundation’s team is fusing AI model reasoning with their own actuators and gearbox designs is wild—they’re not just iterating fast, they’re redefining what “robotic autonomy” means. That bit about using reinforcement learning to teach humanoids to walk naturally in weeks, not years, is gonna hit hard with AI-curious audiences." It also wrote me what you will learn, I'll post that below. Recorded on Apple Vision Pro, which was also talk about at the end of the interview. Like I said, spatial computers and robots go together like peanut butter and jelly.

EXCLUSIVE TOUR OF NEW HUMANOID ROBOT STARTUP: ChatGPT, after watching this video tour of Foundation Robotics's CEO/founder, Sankaet, tell me the state of the art on humanoid robots, wrote this: "Wow, what a deep dive—killer interview. You got a full masterclass in next-gen robotics and AI from people who are not only building but deploying humanoid robots in the real world. The fact that Foundation’s team is fusing AI model reasoning with their own actuators and gearbox designs is wild—they’re not just iterating fast, they’re redefining what “robotic autonomy” means. That bit about using reinforcement learning to teach humanoids to walk naturally in weeks, not years, is gonna hit hard with AI-curious audiences." It also wrote me what you will learn, I'll post that below. Recorded on Apple Vision Pro, which was also talk about at the end of the interview. Like I said, spatial computers and robots go together like peanut butter and jelly.

Robert Scoble

269,783 次观看 • 1 年前

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Kaustubh Sridhar

52,158 次观看 • 1 年前

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,681 次观看 • 3 年前

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

CyberRobo

18,356 次观看 • 6 个月前

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

Yinghao Xu

47,538 次观看 • 7 个月前

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

AI at Meta

310,507 次观看 • 1 年前

Robots need to be able to apply pressure and make contact with objects as needed in order to accomplish their tasks. From compliance to working safely around humans to whole-body manipulation of heavy objects, combining force and position control can dramatically expand the capabilities of robots. This is especially true for legged robots, which have so much ability to exert forces on the world around them. But how do we train robots which can do this? Baoxiong Jia tells us more in our discussion of his team’s recent, Best Paper Award winning work on learning a unified policy for position and force control, called UniFP. To learn more, watch Episode #49 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton.

Robots need to be able to apply pressure and make contact with objects as needed in order to accomplish their tasks. From compliance to working safely around humans to whole-body manipulation of heavy objects, combining force and position control can dramatically expand the capabilities of robots. This is especially true for legged robots, which have so much ability to exert forces on the world around them. But how do we train robots which can do this? Baoxiong Jia tells us more in our discussion of his team’s recent, Best Paper Award winning work on learning a unified policy for position and force control, called UniFP. To learn more, watch Episode #49 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton.

RoboPapers

44,822 次观看 • 9 个月前

For decades, we’ve dreamed of robots that can seamlessly step into our world and lend a hand. Today, we take a major stride toward making that dream a reality: Introducing Gemini Robotics 2 from Google DeepMind, the intelligence layer powering the next generation of truly adaptable robots. This major advance unlocks intelligent whole-body control, advanced dexterity, and even multi-robot collaboration 🤯. Ok but... how does a robot actually "think"? Real-world tasks take time and planning. To manage that complexity, our new embodied reasoning model, Gemini Robotics ER 2, acts as the robot’s high-level brain, enhancing the robot’s capabilities to: — Observe the environment — Reason about the actions needed to complete the task — Coordinate with the vision-language-action model to carry out actions — Track progress until the job is done This setup allows robots to execute complex multi-step workflows, self-correct if a step fails, and adapt to completely novel situations. Learn more about Gemini Robotics ER 2 (and our two other brand new models) here:

Google AI

783,491 次观看 • 1 个月前

we achieved some progress in the implementation of state inference inside octra , first we managed to implement continuous reasoning without clearing the circle memory, second we completed a pure resource model under which validators can verify and certify the execution flow of a set of useful work (in this case this is ML inference with text generation by input tokens fully onchain), a few months ago the answers of the test model were one-word and substantially limited, in the current version (which will be available for everyone) the reasoning flow is automatically extended, it will be possible to upload various models into circles or create tools for their use making them public or only private (full demo will be opensource as soon as we finish, it will be possible to deploy anything in 2-3 clicks) live use case, the economic model also converges here - wallet pays for inference in native coins, validators (with GPU support can maintain availability of their resources to quickly process such private calls, while the execution core remains private through HFHE)

we achieved some progress in the implementation of state inference inside octra , first we managed to implement continuous reasoning without clearing the circle memory, second we completed a pure resource model under which validators can verify and certify the execution flow of a set of useful work (in this case this is ML inference with text generation by input tokens fully onchain), a few months ago the answers of the test model were one-word and substantially limited, in the current version (which will be available for everyone) the reasoning flow is automatically extended, it will be possible to upload various models into circles or create tools for their use making them public or only private (full demo will be opensource as soon as we finish, it will be possible to deploy anything in 2-3 clicks) live use case, the economic model also converges here - wallet pays for inference in native coins, validators (with GPU support can maintain availability of their resources to quickly process such private calls, while the execution core remains private through HFHE)

λ

11,482 次观看 • 1 个月前

With enough data, robots and AI can learn “world models” that let them predict the results of their actions. These models are a way to learn how embodied AI agents can perform a wide variety of useful tasks — but they require a huge amount of data. The team at General Intuition General Intuition has a solution: use data from video games! Games teach movement, problem solving, and complex spatial reasoning, and they come in a staggering diversity of forms, covering a wide variety of problems. What’s more, the captured data is high-quality, without the noise or annotation error that can come from We sat down with Pim de Witte and Adam Jelley from the General Intuition team to learn more about their history, their plans, and their philosophy.

With enough data, robots and AI can learn “world models” that let them predict the results of their actions. These models are a way to learn how embodied AI agents can perform a wide variety of useful tasks — but they require a huge amount of data. The team at General Intuition General Intuition has a solution: use data from video games! Games teach movement, problem solving, and complex spatial reasoning, and they come in a staggering diversity of forms, covering a wide variety of problems. What’s more, the captured data is high-quality, without the noise or annotation error that can come from We sat down with Pim de Witte and Adam Jelley from the General Intuition team to learn more about their history, their plans, and their philosophy.

RoboPapers

85,927 次观看 • 10 个月前

PERCEPTRON JUST DROPPED WHAT MIGHT BE THE BEST OPEN-WEIGHT MODEL FOR ROBOTICS Isaac 0.5 is an absolute beast. Instead of treating vision and movement as separate systems, it is a fully open foundation model that learns to process both at the exact same time. .. and by merging perception and control, it unlocks true real-time visual decision making. Some core stats: → 35 hardware embodiments → 100k hours of demonstrations → SOTA across core embodied reasoning benchmarks → Significantly lower inference FLOPs Best part? Perceptron AI open-sourced the ENTIRE stack! Training code, inference code, and weights are all public. Repo, weights, and paper in 🧵 ↓

PERCEPTRON JUST DROPPED WHAT MIGHT BE THE BEST OPEN-WEIGHT MODEL FOR ROBOTICS Isaac 0.5 is an absolute beast. Instead of treating vision and movement as separate systems, it is a fully open foundation model that learns to process both at the exact same time. .. and by merging perception and control, it unlocks true real-time visual decision making. Some core stats: → 35 hardware embodiments → 100k hours of demonstrations → SOTA across core embodied reasoning benchmarks → Significantly lower inference FLOPs Best part? Perceptron AI open-sourced the ENTIRE stack! Training code, inference code, and weights are all public. Repo, weights, and paper in 🧵 ↓

Charly Wargnier ♨️

47,979 次观看 • 15 天前

SkildAI, which is developing a foundational AI model for robots, has released a new video of their humanoid robot walking over objects and exploring environments it had never seen before. The robot is using vision & adapts in real time. SkildAI: "From raw images and joint feedback, the model directly outputs low-level motor commands. This single neural network enables humanoid robots to seamlessly walk across flat ground, climb stairs, and step over obstacles without any planning, mapping, or manual switching between behaviors. For all of our testing in this video, we deployed the robot in each new environment without any prior planning or mapping. We took off the humanoid’s shoes so you can hear it.

SkildAI, which is developing a foundational AI model for robots, has released a new video of their humanoid robot walking over objects and exploring environments it had never seen before. The robot is using vision & adapts in real time. SkildAI: "From raw images and joint feedback, the model directly outputs low-level motor commands. This single neural network enables humanoid robots to seamlessly walk across flat ground, climb stairs, and step over obstacles without any planning, mapping, or manual switching between behaviors. For all of our testing in this video, we deployed the robot in each new environment without any prior planning or mapping. We took off the humanoid’s shoes so you can hear it.

Sawyer Merritt

69,381 次观看 • 1 年前

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Zhiyang (Frank) Dou

14,610 次观看 • 1 年前