Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

PaLM-E is a generalist, embodied language model for robotics. 🤖 It solves many tasks on 𝘮𝘶𝘭𝘵𝘪𝘱𝘭𝘦 types of robots and for 𝘮𝘶𝘭𝘵𝘪𝘱𝘭𝘦 modalities, including images and neural scene representations. Hear more from researchers at #ICML2023: 📍Booth #109 ⌚10.30 HST

Google DeepMind

1,519,283 subscribers

185,398 views • 3 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

10 Comments

Google DeepMind3 years ago

🔵 PaLM-E combines the large language model PaLM together with one of the most advanced vision models, ViT-22B. The largest instantiation of this approach is PaLM-E-562B, which set a new state of the art on the visual-language OK-VQA benchmark.

Google DeepMind3 years ago

🖼️ As a generally-capable vision-language model, PaLM-E can perform tasks such as describing images, detecting objects, or classifying scenes. It’s also proficient at language tasks, like quoting poetry, solving math equations or generating code. ⬇️

Yair Lavi3 years ago

Is there like a bet between you guys and openAI on who destroys the human race first?

Kevin Carter3 years ago

It seems that the software is ahead of the hardware for now. I wonder how long until it's truly viable to mass produce robots for everyday help

Maurus Balmer3 years ago

MORE, MUCH MORE ...

Cryptking.eth 👑 🦍3 years ago

@Scobleizer the future is coming fast 🏆

Marcos Leite3 years ago

Sensational,the future now! Check in supervision in erro path alternative!

C. ↙️⚠️3 years ago

@DynamicWebPaige It crunched my chips, and took much longer. Dinner service is behind already

Viaja Tech3 years ago

@OpenAI already uploaded chatgpt to @GooglePlay When can we see Bard in its own shop?

Xieba (Beatrix) De La Cruz3 years ago

Palm Cam Bridge e No

Related Videos

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

Danny Driess

1,272,806 views • 3 years ago

LingBot-VLA 2.0 is an impressive new embodied model. Open source and is trained across diverse robot configurations, from single-arm robots to humanoid platforms. It packs 60K hours of curated robot and human data into one generalist policy. It improves robots on difficult long-horizon tasks. Great release by Robbyant.

LingBot-VLA 2.0 is an impressive new embodied model. Open source and is trained across diverse robot configurations, from single-arm robots to humanoid platforms. It packs 60K hours of curated robot and human data into one generalist policy. It improves robots on difficult long-horizon tasks. Great release by Robbyant.

elvis

10,464 views • 2 months ago

Today, we announced 𝗥𝗧-𝟮: a first of its kind vision-language-action model to control robots. 🤖 It learns from both web and robotics data and translates this knowledge into generalised instructions. Find out more:

Today, we announced 𝗥𝗧-𝟮: a first of its kind vision-language-action model to control robots. 🤖 It learns from both web and robotics data and translates this knowledge into generalised instructions. Find out more:

Google DeepMind

537,902 views • 3 years ago

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

Galbot

40,796 views • 4 months ago

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Jiafei Duan

117,636 views • 4 months ago

🤖 Ever seen a robot get confused by a glass door? That era is ending. Today, we're open-sourcing LingBot-Depth, a new spatial perception model that lets robots truly *see* the physical world, including transparent & reflective objects. It even surpasses top-tier industrial depth cameras in core metrics. A new sense of sight for embodied AI. #EmbodiedAI #Robotics #ComputerVision #OpenSource #DepthCamera

🤖 Ever seen a robot get confused by a glass door? That era is ending. Today, we're open-sourcing LingBot-Depth, a new spatial perception model that lets robots truly see the physical world, including transparent & reflective objects. It even surpasses top-tier industrial depth cameras in core metrics. A new sense of sight for embodied AI. #EmbodiedAI #Robotics #ComputerVision #OpenSource #DepthCamera

Robbyant

320,159 views • 7 months ago

We're releasing HY-Embodied-0.5, a family of foundation models for real-world embodied agents. The 2B model is now open source. It strengthens spatial-temporal perception and embodied reasoning for prediction, interaction, and planning. 🤖 The suite includes: 🔹 2B for edge deployment 🔹 32B for complex reasoning Key innovations: 🔹 Mixture-of-Transformers (MoT) architecture for modality-specific computation 🔹 Latent tokens for improved perceptual representation 🔹 Self-evolving post-training 🔹 On-policy distillation from large to small models Across 22 benchmarks, the 2B model outperforms similarly sized SOTA systems on 16 tasks. The 32B model approaches frontier-level performance. 🔗 GitHub: 🤗 Hugging Face:

We're releasing HY-Embodied-0.5, a family of foundation models for real-world embodied agents. The 2B model is now open source. It strengthens spatial-temporal perception and embodied reasoning for prediction, interaction, and planning. 🤖 The suite includes: 🔹 2B for edge deployment 🔹 32B for complex reasoning Key innovations: 🔹 Mixture-of-Transformers (MoT) architecture for modality-specific computation 🔹 Latent tokens for improved perceptual representation 🔹 Self-evolving post-training 🔹 On-policy distillation from large to small models Across 22 benchmarks, the 2B model outperforms similarly sized SOTA systems on 16 tasks. The 32B model approaches frontier-level performance. 🔗 GitHub: 🤗 Hugging Face:

Tencent Hy

34,317 views • 5 months ago

🤖Robots that think ahead and act in real time. LingBot-VA 2.0 — the first embodied-native foundation model. Not fine-tuned from a video generator. Built from scratch for the physical world. ✅ 93.6% success on bimanual tasks ⚡ 150 Hz single-GPU inference 🎯 20 demos to generalize. This is what happens when you stop adapting and start building natively. 🧵👇

🤖Robots that think ahead and act in real time. LingBot-VA 2.0 — the first embodied-native foundation model. Not fine-tuned from a video generator. Built from scratch for the physical world. ✅ 93.6% success on bimanual tasks ⚡ 150 Hz single-GPU inference 🎯 20 demos to generalize. This is what happens when you stop adapting and start building natively. 🧵👇

Robbyant

895,079 views • 2 months ago

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

61,021 views • 8 months ago

Startup School is back! Hear from Jensen Huang, Sam Altman, Alexandr Wang, Jeff Dean, and more. Join a hand-selected group of top CS students, researchers, and engineers for two days of talks, sessions with YC partners, and hands-on robotics demos, right here in San Francisco.

Startup School is back! Hear from Jensen Huang, Sam Altman, Alexandr Wang, Jeff Dean, and more. Join a hand-selected group of top CS students, researchers, and engineers for two days of talks, sessions with YC partners, and hands-on robotics demos, right here in San Francisco.

Y Combinator

531,297 views • 6 months ago

New release from Meta FAIR — Meta Motivo is a first-of-its-kind behavioral foundation model for controlling virtual physics-based humanoid agents for a wide range of complex whole-body tasks. The model is capable of expressing human-like behaviors and achieves performance competitive with task-specific methods and outperforms state-of-the-art unsupervised RL and model-based baselines. Try the demo ➡️ Get the model and code ➡️ We’re excited about how this research could pave the way for fully embodied agents, leading to more lifelike NPCs, democratization of character animation and new types of immersive experiences.

New release from Meta FAIR — Meta Motivo is a first-of-its-kind behavioral foundation model for controlling virtual physics-based humanoid agents for a wide range of complex whole-body tasks. The model is capable of expressing human-like behaviors and achieves performance competitive with task-specific methods and outperforms state-of-the-art unsupervised RL and model-based baselines. Try the demo ➡️ Get the model and code ➡️ We’re excited about how this research could pave the way for fully embodied agents, leading to more lifelike NPCs, democratization of character animation and new types of immersive experiences.

AI at Meta

129,283 views • 1 year ago

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 views • 3 years ago

Pretraining is essential for good performance on a wide variety of robotics tasks, and so most vision-language-action models build off of a vision language model (VLM) trained on a wide variety of image-language data. But how does the choice of VLM translate to downstream robotics performance? Jianke Zhang and @GYanjiang join us to talk about this key part of the robot policy, looking at a wide variety of different VLMs and how they perform. Interestingly, they see that performance on auxiliary tasks like quesiton answering did not lead to downstream improvements in control. To learn more, watch episode 65 of RoboPapers now, with Chris Paxton and Jiafei Duan!

Pretraining is essential for good performance on a wide variety of robotics tasks, and so most vision-language-action models build off of a vision language model (VLM) trained on a wide variety of image-language data. But how does the choice of VLM translate to downstream robotics performance? Jianke Zhang and @GYanjiang join us to talk about this key part of the robot policy, looking at a wide variety of different VLMs and how they perform. Interestingly, they see that performance on auxiliary tasks like quesiton answering did not lead to downstream improvements in control. To learn more, watch episode 65 of RoboPapers now, with Chris Paxton and Jiafei Duan!

RoboPapers

23,905 views • 6 months ago

11 years of humanoid robotics in one video. 🤖 In 2015, humanoid robots were still struggling with basic tasks like walking, climbing stairs and recovering from falls during the DARPA Robotics Challenge. Fast-forward to 2026, and humanoids are running, jumping, boxing, playing football, handling industrial tasks and operating in real-world environments. The World Humanoid Robot Games featured 2,056 robots across 51 events, including practical tasks in factories, hotels and homes. The progress is obvious. But the real question is not how impressive they look in a demo. What will humanoid robots be capable of in 2036? 👀 🎥 Media: Defense Advanced Research Projects Agency (DARPA) / Unitree Robotics ⚠️ This content is shared for informational purposes only. CTO Robotics Media is a media platform and does not own or develop the technologies shown. Credit belongs to the original creators. #HumanoidRobots #Robotics #PhysicalAI #AI #DARPA #RobotEvolution #FutureOfRobotics #EmbodiedAI #Automation #RobotTechnology #CTORoboticsMedia

11 years of humanoid robotics in one video. 🤖 In 2015, humanoid robots were still struggling with basic tasks like walking, climbing stairs and recovering from falls during the DARPA Robotics Challenge. Fast-forward to 2026, and humanoids are running, jumping, boxing, playing football, handling industrial tasks and operating in real-world environments. The World Humanoid Robot Games featured 2,056 robots across 51 events, including practical tasks in factories, hotels and homes. The progress is obvious. But the real question is not how impressive they look in a demo. What will humanoid robots be capable of in 2036? 👀 🎥 Media: Defense Advanced Research Projects Agency (DARPA) / Unitree Robotics ⚠️ This content is shared for informational purposes only. CTO Robotics Media is a media platform and does not own or develop the technologies shown. Credit belongs to the original creators. #HumanoidRobots #Robotics #PhysicalAI #AI #DARPA #RobotEvolution #FutureOfRobotics #EmbodiedAI #Automation #RobotTechnology #CTORoboticsMedia

CTO ROBOTICS Media

322,062 views • 7 days ago

Xiaomi’s humanoid robot at the 2026 World Robot Conference. 🤖🌹 On Chinese Valentine’s Day, the robot is part of the flower-gifting experience on the show floor. From picking a flower and adding decorative confetti to handing it to a visitor, it handles the entire sequence autonomously. Powered by an embodied model, it completes the interaction without remote human teleoperation. Note: Developed as a non-commercial prototype for internal use at Xiaomi EV Factory. Not for sale. #HumanoidRobots #EmbodiedAI #Robotics #XiaomiRobotics #WRC2026

Xiaomi’s humanoid robot at the 2026 World Robot Conference. 🤖🌹 On Chinese Valentine’s Day, the robot is part of the flower-gifting experience on the show floor. From picking a flower and adding decorative confetti to handing it to a visitor, it handles the entire sequence autonomously. Powered by an embodied model, it completes the interaction without remote human teleoperation. Note: Developed as a non-commercial prototype for internal use at Xiaomi EV Factory. Not for sale. #HumanoidRobots #EmbodiedAI #Robotics #XiaomiRobotics #WRC2026

Xiaomi Tech

20,474 views • 1 month ago

🤖 China has launched a factory where robots build robots UBTECH Robotics has opened a “smart” factory in Liuzhou for the mass production of humanoid robots. Robots are already being used on the production line to help assemble other robots. The facility is designed to produce more than 10,000 humanoids a year, with a target rate of roughly one finished robot every 10 minutes. For now, it all looks like industrial automation. Skynet, please take a number and wait your turn.

🤖 China has launched a factory where robots build robots UBTECH Robotics has opened a “smart” factory in Liuzhou for the mass production of humanoid robots. Robots are already being used on the production line to help assemble other robots. The facility is designed to produce more than 10,000 humanoids a year, with a target rate of roughly one finished robot every 10 minutes. For now, it all looks like industrial automation. Skynet, please take a number and wait your turn.

NEXTA

44,262 views • 2 days ago