Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

⭐ The first foundational model available on LeRobot ⭐ Pi0 is the most advanced Vision Language Action model. It takes natural language commands as input and directly output autonomous behavior. It was trained by Physical Intelligence and ported to pytorch by Pablo Montalvo 👇🧵

Remi Cadene

20,749 subscribers

130,941 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

11 Kommentare

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

Take a look at the code:

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

You can now download and use the pretrained policy just like transformers: policy = Pi0Policy.from_pretrained("lerobot/pi0") action =

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

Even cooler, you can finetune it directly on your dataset. Here is an example with @DAubakirovaa 's dataset: python lerobot/scripts/train.py \ --policy.path=lerobot/pi0 \ --dataset.repo_id=danaaubakirova/koch_test Visualize her dataset here:

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

The original code in Jax is available over here:

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

We have full blog post with more details!

Profilbild von AssemblyAI

AssemblyAIvor 1 Jahr

Announcing: Our most advanced speech-to-text model goes beyond accuracy to capture the real-world complexity of human conversation and deliver reliable, source-of-truth audio data. Explore Universal-2 updates 👇

Profilbild von Muhammad Ahmed

Muhammad Ahmedvor 1 Jahr

@LeRobotHF @physical_int @m_olbap that is so cool , Thanks @LeRobotHF ! Is this model somewhat similar to OpenVLA (which is also on hugging face) ? For instance with OpenVLA we get action output tokens for a single armed robot. Can i simply replace OpenVLA with this without changing the inference pipeline ?

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

@LeRobotHF @physical_int @m_olbap Pi0 was trained with much more data! Yes you should be as our policy includes all the preprocessing and post processing. Only requirements: images need to be in float32 [0,1] range.

Profilbild von Igor Beaver

Igor Beavervor 1 Jahr

@LeRobotHF @physical_int @m_olbap Incredible gift!

Profilbild von Venky

Venkyvor 1 Jahr

@LeRobotHF @physical_int @m_olbap Super. The LLM moment for Robotics is here. The gap is closing fast. Eagerly awaiting for the weights. Also, would be great to have some blog or tutorial on how one can do this from scratch.

Profilbild von Remi Cadene

Remi Cadenevor 1 Jahr

@LeRobotHF @physical_int @m_olbap We ported the weights already. We are working on a tutorial to reproduce this on more affordable robots

Ähnliche Videos

📢 First contact between a frontier model and robots! Gemini Robotics is a SOTA generalist Vision-Language-Action model bringing frontier model intelligence to the physical world. It's an extremely capable model enabling dexterous, steerable, and general robot control. 🧵⬇️

📢 First contact between a frontier model and robots! Gemini Robotics is a SOTA generalist Vision-Language-Action model bringing frontier model intelligence to the physical world. It's an extremely capable model enabling dexterous, steerable, and general robot control. 🧵⬇️

Ted Xiao

152,445 Aufrufe • vor 1 Jahr

Pi0 from Physical Intelligence is one of the best generalist Vision-Language-Action (VLA) models for robotics (and predecessor of Pi0.5). 🧵 Here's how it works + how you can fine-tune it on your own robot for a simple task:

Pi0 from Physical Intelligence is one of the best generalist Vision-Language-Action (VLA) models for robotics (and predecessor of Pi0.5). 🧵 Here's how it works + how you can fine-tune it on your own robot for a simple task:

Ilia

32,956 Aufrufe • vor 10 Monaten

Vision RAG with vector database is all you need. It uses vision language model to embed pages of PDF as directly vectors, without the tedious chunking process. 100% Opensource code.

Vision RAG with vector database is all you need. It uses vision language model to embed pages of PDF as directly vectors, without the tedious chunking process. 100% Opensource code.

Shubham Saboo

106,731 Aufrufe • vor 1 Jahr

We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵

We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵

Google DeepMind

819,379 Aufrufe • vor 1 Jahr

A new interesting model from Hailuo AI (MiniMax) It's called T2V-01-Director, and as the name says, you can use natural language or specific commands to guide the camera. Available in early preview for the creative partners. Here's how it works:

A new interesting model from Hailuo AI (MiniMax) It's called T2V-01-Director, and as the name says, you can use natural language or specific commands to guide the camera. Available in early preview for the creative partners. Here's how it works:

Alex Patrascu

53,808 Aufrufe • vor 1 Jahr

🕹️ Is this the smallest language model in the world? I just managed to squeeze JAM, real artificial intelligence into 30 kilobytes, running on a 1979 Atari 800. Just A Model. Fully generative, deterministic language model, powered by a neural network and built to run on 8‑bit hardware. Atari Forever.

🕹️ Is this the smallest language model in the world? I just managed to squeeze JAM, real artificial intelligence into 30 kilobytes, running on a 1979 Atari 800. Just A Model. Fully generative, deterministic language model, powered by a neural network and built to run on 8‑bit hardware. Atari Forever.

Marek Spanel

113,623 Aufrufe • vor 3 Monaten

Exciting progress on Vision-Language-Action models from a collaboration between San Francisco-based Physical Intelligence (π) and China’s AGIBOT: (π)’s single model can autonomously perform diverse tasks on the AGIBOT G1 robot, using both humanoid hands and two-finger grippers.

Exciting progress on Vision-Language-Action models from a collaboration between San Francisco-based Physical Intelligence (π) and China’s AGIBOT: (π)’s single model can autonomously perform diverse tasks on the AGIBOT G1 robot, using both humanoid hands and two-finger grippers.

The Humanoid Hub

32,575 Aufrufe • vor 1 Jahr

Below is our open-source arm assembled by one of the discord members! Also, today Physical Intelligence shared their incredible top-notch robotics model Pi0. And I just want to remind you that you can use our arm together with this model and with LeRobot The arm has the same DOF as the arm used by Pi0, but has smaller size and cost starts from 200$ per arm. You can use better motors like we did in our demos, then it will be more precise and stable but the price will be about $600 per arm. Printable files, purchase list, and help with the assembly - in our discord channel.

Below is our open-source arm assembled by one of the discord members! Also, today Physical Intelligence shared their incredible top-notch robotics model Pi0. And I just want to remind you that you can use our arm together with this model and with LeRobot The arm has the same DOF as the arm used by Pi0, but has smaller size and cost starts from 200$ per arm. You can use better motors like we did in our demos, then it will be more precise and stable but the price will be about $600 per arm. Printable files, purchase list, and help with the assembly - in our discord channel.

Igor Kulakov

53,711 Aufrufe • vor 1 Jahr

VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relevant dynamics. During training, the VLA leverages V-JEPA2 by conditioning its predictor. This clever trick adds a world modeling objective to the training, which also allows pretraining on human videos. At inference, the world model is dropped entirely, keeping only a standard VLA architecture: Qwen backbone and action head. The demo here was only fine-tuned on 13 examples, showing great pretraining capability and running in real time on NVIDIA Robotics DGX Spark! VLA-JEPA is the first world model to be ported to LeRobot, and I feel like it won't be the last 🚀 Thomas Wolf clem 🤗

VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relevant dynamics. During training, the VLA leverages V-JEPA2 by conditioning its predictor. This clever trick adds a world modeling objective to the training, which also allows pretraining on human videos. At inference, the world model is dropped entirely, keeping only a standard VLA architecture: Qwen backbone and action head. The demo here was only fine-tuned on 13 examples, showing great pretraining capability and running in real time on NVIDIA Robotics DGX Spark! VLA-JEPA is the first world model to be ported to LeRobot, and I feel like it won't be the last 🚀 Thomas Wolf clem 🤗

LeRobot

318,029 Aufrufe • vor 1 Monat

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

60,893 Aufrufe • vor 5 Monaten

We're releasing Paris 2.0, which, to our knowledge, is the world's first decentralized trained video generation model. We benchmarked it against a monolithic model trained on the same data and compute budget, and Paris 2.0 outperformed the monolithic by ~2x on FVD benchmark.

We're releasing Paris 2.0, which, to our knowledge, is the world's first decentralized trained video generation model. We benchmarked it against a monolithic model trained on the same data and compute budget, and Paris 2.0 outperformed the monolithic by ~2x on FVD benchmark.

Bidhan Roy

454,964 Aufrufe • vor 1 Monat

Introducing Realtime TTS-2, a new generation of voice model built for realtime conversation. It is the first voice model that hears the conversation, takes natural-language voice direction, holds one voice identity across over 100 languages, and speaks like a person who is paying attention. The result is voice AI that feels as good as it sounds. Try it out: Learn More:

Introducing Realtime TTS-2, a new generation of voice model built for realtime conversation. It is the first voice model that hears the conversation, takes natural-language voice direction, holds one voice identity across over 100 languages, and speaks like a person who is paying attention. The result is voice AI that feels as good as it sounds. Try it out: Learn More:

Inworld AI

325,540 Aufrufe • vor 2 Monaten

It’s time to go beyond language models. Introducing Odyssey-2 Max, our most powerful world model yet. It materially advances the SOTA in physical accuracy. This is a big step toward models that simulate and interact with the world in real time. A new intelligence entirely!

It’s time to go beyond language models. Introducing Odyssey-2 Max, our most powerful world model yet. It materially advances the SOTA in physical accuracy. This is a big step toward models that simulate and interact with the world in real time. A new intelligence entirely!

Odyssey

2,183,089 Aufrufe • vor 2 Monaten

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Google AI

202,556 Aufrufe • vor 3 Monaten

1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇

1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇

Fangchen Liu

68,353 Aufrufe • vor 1 Jahr

Agentic trading is now live on Gemini 🤖 Connect your AI model of choice directly to the exchange, train it with plain language, and run your own trading systems directly on the exchange Built to get you trading in minutes

Agentic trading is now live on Gemini 🤖 Connect your AI model of choice directly to the exchange, train it with plain language, and run your own trading systems directly on the exchange Built to get you trading in minutes

Gemini

166,094 Aufrufe • vor 3 Monaten

The next frontier of autonomous driving is unlocked by reasoning models. NVIDIA Alpamayo brings together open AI models with reasoning capabilities, closed-loop simulation tools, and massive real-world driving datasets. Alpamayo 1 is a vision–language–action model that explains its own decisions through explicit reasoning traces, enabling trustworthy, humanlike decision-making. Together with NVIDIA’s Physical AI dataset and AlpaSim simulation, Alpamayo provides the tools and scale required to enable level 4 autonomous vehicles. ▶️ Watch now:

The next frontier of autonomous driving is unlocked by reasoning models. NVIDIA Alpamayo brings together open AI models with reasoning capabilities, closed-loop simulation tools, and massive real-world driving datasets. Alpamayo 1 is a vision–language–action model that explains its own decisions through explicit reasoning traces, enabling trustworthy, humanlike decision-making. Together with NVIDIA’s Physical AI dataset and AlpaSim simulation, Alpamayo provides the tools and scale required to enable level 4 autonomous vehicles. ▶️ Watch now:

NVIDIA DRIVE

35,324 Aufrufe • vor 6 Monaten

Today I am extremely proud to introduce the Fury Autonomous Vehicle Orchestrator 🇺🇸 For the past 12 months, the Scout AI team has been building this quietly. Now we’re finally ready to unveil it to the world Just like AI agents are transforming the digital world, Fury Orchestrator brings that same agentic intelligence into the physical world for the U.S. warfighter It is the first agentic system that can take a commander’s high-level mission intent, spoken or typed in natural language, and turn it into coordinated autonomous action across a mixed fleet of unmanned assets More in 🧵

Today I am extremely proud to introduce the Fury Autonomous Vehicle Orchestrator 🇺🇸 For the past 12 months, the Scout AI team has been building this quietly. Now we’re finally ready to unveil it to the world Just like AI agents are transforming the digital world, Fury Orchestrator brings that same agentic intelligence into the physical world for the U.S. warfighter It is the first agentic system that can take a commander’s high-level mission intent, spoken or typed in natural language, and turn it into coordinated autonomous action across a mixed fleet of unmanned assets More in 🧵

Colby Adcock

213,365 Aufrufe • vor 4 Monaten

Today we're releasing Not Diamond… The world’s most powerful AI model router. Not Diamond maximizes LLM output quality by automatically recommending the best LLM on every request at lower cost and latency. And it takes <5m to set up. Watch this to see how to start using it:

Today we're releasing Not Diamond… The world’s most powerful AI model router. Not Diamond maximizes LLM output quality by automatically recommending the best LLM on every request at lower cost and latency. And it takes <5m to set up. Watch this to see how to start using it:

Tomas Hernando Kofman

75,821 Aufrufe • vor 1 Jahr