Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Flow reversal steering allows "steering" diffusion-based VLAs with high-level actions, for example from VLM reasoning. This also lets us run RL in the diffusion noise space with exploration guided by high-level reasoning: think through a task, then practice it! 👇

svlevine's profile picture

Sergey Levine

141,362 subscribers

90,281 Aufrufe • vor 4 Monaten •via X (Twitter)

BildungGesundheit & WellnessWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

6 Kommentare

Profilbild von Sergey Levine
Sergey Levinevor 4 Monaten

Check out the paper here: A fun collaboration w/ Andy Tang, @verityw_ , @ajwagenmaker, @chelseabfinn

Profilbild von Nurvai - The Data Layer for Physical AI
Nurvai - The Data Layer for Physical AIvor 4 Monaten

Saw this a few months back, and the idea of guiding diffusion policies through higher-level reasoning is still really interesting. Giving models a way to “think through” a task while exploring the action space could be a powerful bridge between planning and execution in robotics.

Profilbild von Shubham Nagar
Shubham Nagarvor 4 Monaten

That’s really inspiring work, I have fun reading about latest research in robotics. My current interest is in using FF-JEPA world models and using RL agents to plan it in the latent space. I have been talking to a professor about this at my university!

Profilbild von Masa.S
Masa.Svor 4 Monaten

技術は毎年進化する。 しかし未来を支えるのは、優しさと温もり、そして人と人とのつながりではないでしょうか。 ヒューマノイドが「助ける」だけでなく、「安心を届ける」存在になれたらどうでしょう。 Technology may become smarter every year. But the future will belong to those who remember kindness, warmth, and human connection. What if a humanoid could offer not only assistance, but also reassurance?

Profilbild von Maximilian Alexander
Maximilian Alexandervor 4 Monaten

What robotic arms do you recommend us buying to try this out?

Profilbild von YJTimeless
YJTimelessvor 4 Monaten

✨ “Think first, then practice” perfectly sums up the core advantage of Flow Reverse Steering!

Ähnliche Videos

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇
0:39

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

Sergey Levine

153,232 Aufrufe • vor 1 Jahr

Large multimodal models often lack precise low-level perception needed for high-level reasoning, even with simple vector graphics. We bridge this gap by proposing an intermediate symbolic representation that leverages LLMs for text-based reasoning. 🧵1/4
1:34

Large multimodal models often lack precise low-level perception needed for high-level reasoning, even with simple vector graphics. We bridge this gap by proposing an intermediate symbolic representation that leverages LLMs for text-based reasoning. 🧵1/4

Zhenhailong Wang

37,320 Aufrufe • vor 2 Jahren

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.
5:11

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,681 Aufrufe • vor 3 Jahren

Reasoning = step-by-step thinking. Usually. Diffusion reasons in parallel and refines natively. Better quality without the latency tax. Samar Khanna on how diffusion reasoning works. We're hiring →
1:18

Reasoning = step-by-step thinking. Usually. Diffusion reasons in parallel and refines natively. Better quality without the latency tax. Samar Khanna on how diffusion reasoning works. We're hiring →

Inception

16,490 Aufrufe • vor 7 Monaten

One core bottleneck in VLA models is action representation. Discrete tokens scale beautifully with VLM pretraining—but lose precision. Continuous actions are precise—but often break VLM reasoning. In our new work, we resolve this tension at the representation level. 🧵
0:41

One core bottleneck in VLA models is action representation. Discrete tokens scale beautifully with VLM pretraining—but lose precision. Continuous actions are precise—but often break VLM reasoning. In our new work, we resolve this tension at the representation level. 🧵

Jianlan Luo

47,993 Aufrufe • vor 9 Monaten

How can robots autonomously handle ambiguous situations that require commonsense reasoning? *VLM-PC* provides adaptive high-level planning, so robots can get unstuck by exploring multiple strategies. Paper:
0:49

How can robots autonomously handle ambiguous situations that require commonsense reasoning? *VLM-PC* provides adaptive high-level planning, so robots can get unstuck by exploring multiple strategies. Paper:

Annie Chen

24,336 Aufrufe • vor 1 Jahr

We figured out how to train VLAs with diffusion outputs much faster (7.5x faster), inheriting better language following from the VLM, and leading to better results. The key: protect the VLM backbone during training with knowledge insulation. Let’s talk about what we learned👇
0:31

We figured out how to train VLAs with diffusion outputs much faster (7.5x faster), inheriting better language following from the VLM, and leading to better results. The key: protect the VLM backbone during training with knowledge insulation. Let’s talk about what we learned👇

Physical Intelligence

101,406 Aufrufe • vor 1 Jahr

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀
0:50

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀

Abhishek Gupta

22,354 Aufrufe • vor 1 Jahr

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior abs: project page:
0:22

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior abs: project page:

AK

163,868 Aufrufe • vor 3 Jahren

We present MMaDA, first diffusion that unifies text reasoning, multimodal understanding, and image generation through Mixed Long-CoT, and unified RL - UniGRPO. 📚 Paper: 💻 Code: 📦 Model:
0:25

We present MMaDA, first diffusion that unifies text reasoning, multimodal understanding, and image generation through Mixed Long-CoT, and unified RL - UniGRPO. 📚 Paper: 💻 Code: 📦 Model:

Ling Yang

87,734 Aufrufe • vor 1 Jahr

🚀 Introducing 𝐈𝐃𝐌-𝐕𝐓𝐎𝐍 : A novel diffusion model for image-based virtual try-on! 👗 😍 Improves garment fidelity and generates authentic visuals. 🔧 Uses two modules to encode garment semantics: visual encoder for high-level & parallel UNet for low-level. Links below👇
0:30

🚀 Introducing 𝐈𝐃𝐌-𝐕𝐓𝐎𝐍 : A novel diffusion model for image-based virtual try-on! 👗 😍 Improves garment fidelity and generates authentic visuals. 🔧 Uses two modules to encode garment semantics: visual encoder for high-level & parallel UNet for low-level. Links below👇

Gradio

104,492 Aufrufe • vor 2 Jahren

Masked diffusion is a discrete analog of continuous diffusion. Continuous diffusion gradually adds Gaussian noise, then learns to reverse that process to recover data. Masked diffusion instead progressively replaces tokens with a mask token, then learns to fill them back in.
0:25

Masked diffusion is a discrete analog of continuous diffusion. Continuous diffusion gradually adds Gaussian noise, then learns to reverse that process to recover data. Masked diffusion instead progressively replaces tokens with a mask token, then learns to fill them back in.

Alec Helbling

24,757 Aufrufe • vor 2 Monaten

They really do think that Europe was originally populated by Blacks and Browns. There's no reasoning with this level of stupid.
0:52

They really do think that Europe was originally populated by Blacks and Browns. There's no reasoning with this level of stupid.

Holly Grayle

10,867 Aufrufe • vor 6 Monaten

Diffusion Policy was one of the big breakthroughs that has enabled an explosion in real-world robot learning. However, it’s always had a weakness, which is that it works by computing a final action trajectory from random noise, which leads to high latency when predicting a final action sequence. Instead, why not initialize the search based on previous actions? This allows for incredibly fast policy inference and in many cases improved generalization, generating high-quality predictions with sub-ms latency. Jindou Jia and Jianfei Yang join us to explain. Learn more on Episode 95 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton!
49:19

Diffusion Policy was one of the big breakthroughs that has enabled an explosion in real-world robot learning. However, it’s always had a weakness, which is that it works by computing a final action trajectory from random noise, which leads to high latency when predicting a final action sequence. Instead, why not initialize the search based on previous actions? This allows for incredibly fast policy inference and in many cases improved generalization, generating high-quality predictions with sub-ms latency. Jindou Jia and Jianfei Yang join us to explain. Learn more on Episode 95 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

28,629 Aufrufe • vor 2 Monaten

Update on the new reasoning popover in ChatGPT web app prompt composer - there's now even a keyboard shortcut to cycle through reasoning levels, and it looks like these levels correspond to "Quick" (low) = GPT-4o, "Think a little" (medium) = o3-mini, and "Think harder" (high) = o3-mini-high
0:25

Update on the new reasoning popover in ChatGPT web app prompt composer - there's now even a keyboard shortcut to cycle through reasoning levels, and it looks like these levels correspond to "Quick" (low) = GPT-4o, "Think a little" (medium) = o3-mini, and "Think harder" (high) = o3-mini-high

Tibor Blaho

63,466 Aufrufe • vor 1 Jahr

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion Video Model, aligning them in the representation space. > Let the LLM control the VM on-the-fly. Instruction tuning maximizes the controllability. > Autoregressive LLM empowers VM to generate indefinitely long videos: Starting with a VM for 2-second videos, Pandora extends it for 8-second videos. Would #Sora+#GPT4 under Pandora produce hours-long videos? 📽️ > World Model is beyond just video generation. It’s sensory-level information processing + concept-level reasoning and reflection. Pandora bridges both, with the concept- / language-level backbone (LLM) managing & steering the sensory-level VM functionalities. 👁️🧠 Check out for a bunch of interesting results:
1:35

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion Video Model, aligning them in the representation space. > Let the LLM control the VM on-the-fly. Instruction tuning maximizes the controllability. > Autoregressive LLM empowers VM to generate indefinitely long videos: Starting with a VM for 2-second videos, Pandora extends it for 8-second videos. Would #Sora+#GPT4 under Pandora produce hours-long videos? 📽️ > World Model is beyond just video generation. It’s sensory-level information processing + concept-level reasoning and reflection. Pandora bridges both, with the concept- / language-level backbone (LLM) managing & steering the sensory-level VM functionalities. 👁️🧠 Check out for a bunch of interesting results:

Zhiting Hu

63,477 Aufrufe • vor 2 Jahren

Meet Alpamayo 2 Super, now commercially available for robotaxis and autonomous vehicles. Built for complex real-world driving, this open reasoning model adds 360° awareness, high-level driving decisions and automated reasoning labels. We built it, so you can build on it.
0:51

Meet Alpamayo 2 Super, now commercially available for robotaxis and autonomous vehicles. Built for complex real-world driving, this open reasoning model adds 360° awareness, high-level driving decisions and automated reasoning labels. We built it, so you can build on it.

NVIDIA

131,872 Aufrufe • vor 2 Monaten

Netflix just dropped Go-with-the-Flow Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
4:10

Netflix just dropped Go-with-the-Flow Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise

AK

95,564 Aufrufe • vor 1 Jahr

With Cybertruck, we introduced & scaled many automotive firsts 48V architecture: 4x lower current that enables thinner, lighter wiring True Steer-by-Wire with no mechanical steering link, enabling variable steering response Combined with rear-wheel steering, this also brought coordinated front + rear steering for greater low speed maneuverability & high speed stability Structural stainless-steel exoskeleton: allows exterior panels to carry structural loads, resulting in far higher torsional stiffness Full-hard stainless air bending: forms ultra-hard exterior panels without damaging the visible surface Etherloop: gigabit vehicle network that eliminates ~70% of cross-vehicle wiring
0:32

With Cybertruck, we introduced & scaled many automotive firsts 48V architecture: 4x lower current that enables thinner, lighter wiring True Steer-by-Wire with no mechanical steering link, enabling variable steering response Combined with rear-wheel steering, this also brought coordinated front + rear steering for greater low speed maneuverability & high speed stability Structural stainless-steel exoskeleton: allows exterior panels to carry structural loads, resulting in far higher torsional stiffness Full-hard stainless air bending: forms ultra-hard exterior panels without damaging the visible surface Etherloop: gigabit vehicle network that eliminates ~70% of cross-vehicle wiring

Tesla

1,338,922 Aufrufe • vor 1 Monat