正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Flow reversal steering allows "steering" diffusion-based VLAs with high-level actions, for example from VLM reasoning. This also lets us run RL in the diffusion noise space with exploration guided by high-level reasoning: think through a task, then practice it! 👇

Sergey Levine

129,974 subscribers

62,094 次观看 • 5 天前 •via X (Twitter)

教育健康养生科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

Sergey Levine

152,824 次观看 • 11 个月前

So you’ve trained your favorite diffusion/flow based policy, but it’s just not good enough 0-shot. Worry not, in our new work DSRL - we show how to *steer* pre-trained diffusion policies with off-policy RL, improving behavior efficiently enough for direct training in the real world! DSRL retains nice exploration from the base policy, but allows for quick improvement beyond this base policy with RL. The method is frustratingly simple, and super easy to throw on top of your favorite pretrained policy (VLA/diffusion policy, etc). Let’s think about how it works, 🧵 (1/10)

So you’ve trained your favorite diffusion/flow based policy, but it’s just not good enough 0-shot. Worry not, in our new work DSRL - we show how to steer pre-trained diffusion policies with off-policy RL, improving behavior efficiently enough for direct training in the real world! DSRL retains nice exploration from the base policy, but allows for quick improvement beyond this base policy with RL. The method is frustratingly simple, and super easy to throw on top of your favorite pretrained policy (VLA/diffusion policy, etc). Let’s think about how it works, 🧵 (1/10)

Abhishek Gupta

19,035 次观看 • 11 个月前

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 次观看 • 2 年前

One core bottleneck in VLA models is action representation. Discrete tokens scale beautifully with VLM pretraining—but lose precision. Continuous actions are precise—but often break VLM reasoning. In our new work, we resolve this tension at the representation level. 🧵

One core bottleneck in VLA models is action representation. Discrete tokens scale beautifully with VLM pretraining—but lose precision. Continuous actions are precise—but often break VLM reasoning. In our new work, we resolve this tension at the representation level. 🧵

Jianlan Luo

47,954 次观看 • 5 个月前

How can robots autonomously handle ambiguous situations that require commonsense reasoning? *VLM-PC* provides adaptive high-level planning, so robots can get unstuck by exploring multiple strategies. Paper:

How can robots autonomously handle ambiguous situations that require commonsense reasoning? VLM-PC provides adaptive high-level planning, so robots can get unstuck by exploring multiple strategies. Paper:

Annie Chen

24,112 次观看 • 1 年前

We figured out how to train VLAs with diffusion outputs much faster (7.5x faster), inheriting better language following from the VLM, and leading to better results. The key: protect the VLM backbone during training with knowledge insulation. Let’s talk about what we learned👇

We figured out how to train VLAs with diffusion outputs much faster (7.5x faster), inheriting better language following from the VLM, and leading to better results. The key: protect the VLM backbone during training with knowledge insulation. Let’s talk about what we learned👇

Physical Intelligence

100,142 次观看 • 1 年前

Google DeepMind introduces Gemini Robotics 1.5, enabling robots to perceive, plan, think, use tools, and act on complex tasks. The agentic framework comprises: ⦿ Gemini Robotics-ER 1.5 (VLM): Orchestrates high-level embodied reasoning and planning. ⦿ Gemini Robotics 1.5 (VLA): Converts visuals and instructions provided by ER 1.5 into actions.

Google DeepMind introduces Gemini Robotics 1.5, enabling robots to perceive, plan, think, use tools, and act on complex tasks. The agentic framework comprises: ⦿ Gemini Robotics-ER 1.5 (VLM): Orchestrates high-level embodied reasoning and planning. ⦿ Gemini Robotics 1.5 (VLA): Converts visuals and instructions provided by ER 1.5 into actions.

The Humanoid Hub

65,928 次观看 • 8 个月前

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀

Abhishek Gupta

22,354 次观看 • 8 个月前

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior abs: project page:

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior abs: project page:

AK

163,868 次观看 • 3 年前

They really do think that Europe was originally populated by Blacks and Browns. There's no reasoning with this level of stupid.

They really do think that Europe was originally populated by Blacks and Browns. There's no reasoning with this level of stupid.

Holly Grayle

10,867 次观看 • 1 个月前

🚀 Introducing 𝐈𝐃𝐌-𝐕𝐓𝐎𝐍 : A novel diffusion model for image-based virtual try-on! 👗 😍 Improves garment fidelity and generates authentic visuals. 🔧 Uses two modules to encode garment semantics: visual encoder for high-level & parallel UNet for low-level. Links below👇

🚀 Introducing 𝐈𝐃𝐌-𝐕𝐓𝐎𝐍 : A novel diffusion model for image-based virtual try-on! 👗 😍 Improves garment fidelity and generates authentic visuals. 🔧 Uses two modules to encode garment semantics: visual encoder for high-level & parallel UNet for low-level. Links below👇

Gradio

104,492 次观看 • 2 年前

Update on the new reasoning popover in ChatGPT web app prompt composer - there's now even a keyboard shortcut to cycle through reasoning levels, and it looks like these levels correspond to "Quick" (low) = GPT-4o, "Think a little" (medium) = o3-mini, and "Think harder" (high) = o3-mini-high

Update on the new reasoning popover in ChatGPT web app prompt composer - there's now even a keyboard shortcut to cycle through reasoning levels, and it looks like these levels correspond to "Quick" (low) = GPT-4o, "Think a little" (medium) = o3-mini, and "Think harder" (high) = o3-mini-high

Tibor Blaho

63,219 次观看 • 1 年前

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion Video Model, aligning them in the representation space. > Let the LLM control the VM on-the-fly. Instruction tuning maximizes the controllability. > Autoregressive LLM empowers VM to generate indefinitely long videos: Starting with a VM for 2-second videos, Pandora extends it for 8-second videos. Would #Sora+#GPT4 under Pandora produce hours-long videos? 📽️ > World Model is beyond just video generation. It’s sensory-level information processing + concept-level reasoning and reflection. Pandora bridges both, with the concept- / language-level backbone (LLM) managing & steering the sensory-level VM functionalities. 👁️🧠 Check out for a bunch of interesting results:

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion Video Model, aligning them in the representation space. > Let the LLM control the VM on-the-fly. Instruction tuning maximizes the controllability. > Autoregressive LLM empowers VM to generate indefinitely long videos: Starting with a VM for 2-second videos, Pandora extends it for 8-second videos. Would #Sora+#GPT4 under Pandora produce hours-long videos? 📽️ > World Model is beyond just video generation. It’s sensory-level information processing + concept-level reasoning and reflection. Pandora bridges both, with the concept- / language-level backbone (LLM) managing & steering the sensory-level VM functionalities. 👁️🧠 Check out for a bunch of interesting results:

Zhiting Hu

62,689 次观看 • 2 年前

Netflix just dropped Go-with-the-Flow Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise

Netflix just dropped Go-with-the-Flow Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise

AK

95,542 次观看 • 1 年前

New research from Meta FAIR: Large Concept Models (LCM) is a fundamentally different paradigm for language modeling that decouples reasoning from language representation, inspired by how humans can plan high-level thoughts to communicate.

New research from Meta FAIR: Large Concept Models (LCM) is a fundamentally different paradigm for language modeling that decouples reasoning from language representation, inspired by how humans can plan high-level thoughts to communicate.

AI at Meta

531,486 次观看 • 1 年前

Wow, we can steer diffusion models at inference time! Introducing Diffusion Tree Sampling (DTS): a search-based approach inspired by Monte Carlo Tree Search that turns inference into an anytime, reward-guided optimization process. Diffusion Tree Sampling (DTS) produces asymptotically exact samples from the target distribution in the limit of infinite rollouts, and its greedy variant, Diffusion Tree Search (DTS⋆), performs a global search for high reward samples. The results are pretty impressive: - On MNIST and CIFAR-10 class-conditional generation, DTS matches the FID of the best-performing baseline with up to 10× less compute. - In text-to-image generation and language completion tasks, DTS⋆ effectively searches for high reward samples that match best-of-N with up to 5× less compute.

Wow, we can steer diffusion models at inference time! Introducing Diffusion Tree Sampling (DTS): a search-based approach inspired by Monte Carlo Tree Search that turns inference into an anytime, reward-guided optimization process. Diffusion Tree Sampling (DTS) produces asymptotically exact samples from the target distribution in the limit of infinite rollouts, and its greedy variant, Diffusion Tree Search (DTS⋆), performs a global search for high reward samples. The results are pretty impressive: - On MNIST and CIFAR-10 class-conditional generation, DTS matches the FID of the best-performing baseline with up to 10× less compute. - In text-to-image generation and language completion tasks, DTS⋆ effectively searches for high reward samples that match best-of-N with up to 5× less compute.

机器之心 JIQIZHIXIN

19,037 次观看 • 11 个月前

Diffusion Policies and VLAs with Formal Safety Guarantees🛡️ We just released our new paper: “From Demonstrations to Safe Deployment: Path-Consistent Safety Filtering for Diffusion Policies” 📑Paper: 🌐Website:

Diffusion Policies and VLAs with Formal Safety Guarantees🛡️ We just released our new paper: “From Demonstrations to Safe Deployment: Path-Consistent Safety Filtering for Diffusion Policies” 📑Paper: 🌐Website:

Learning Systems and Robotics Lab (is hiring!)

20,356 次观看 • 7 个月前

Memory-V2V by Adobe: diffusion model with memory for consistent iterative editing; - ReCamMaster+LucyEdit; - reduces FLOPs by 30-90%; outperforming FIFO-Diffusion. high fidelity across long durations like consistent style transfer

Memory-V2V by Adobe: diffusion model with memory for consistent iterative editing; - ReCamMaster+LucyEdit; - reduces FLOPs by 30-90%; outperforming FIFO-Diffusion. high fidelity across long durations like consistent style transfer

Wildminder

10,248 次观看 • 4 个月前

FasterCache Training-Free Video Diffusion Model Acceleration with High Quality

FasterCache Training-Free Video Diffusion Model Acceleration with High Quality

AK

91,259 次观看 • 1 年前

Meet Higgsfield CINEMA STUDIO 3.0 with UP TO 65% OFF. This is Cinema Studio at its peak. Happy birthday to us. Cinematic reasoning & Ultimate realism. Next-level AI filmmaking with native audio. A faster workflow made for high-quality production. Available exclusively on Business Plan.

Meet Higgsfield CINEMA STUDIO 3.0 with UP TO 65% OFF. This is Cinema Studio at its peak. Happy birthday to us. Cinematic reasoning & Ultimate realism. Next-level AI filmmaking with native audio. A faster workflow made for high-quality production. Available exclusively on Business Plan.

Higgsfield AI 🧩

1,754,399 次观看 • 2 个月前