Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Look Only When Needed Certificate-Guided Visual Routing for Efficient Diffusion VLMs paper:

AK

538,835 subscribers

20,862 views • 7 days ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

7 Comments

harpreet7 days ago

@dan_gural can’t help but think of you every time I see this coco pic 😂

The SI Maximalist7 days ago

Certificate-guided visual routing is a delightfully formal answer to “do I really need to look?”

花花大人7 days ago

Diffusion VLM 按需看图这条路由思路挺干净。

Saksham Jain7 days ago

wouldn't have guessed the image only flips the unmasked token on 22.8% of steps. using the no-image pass as its own check is clever, nothing to tune

Shinka - AI7 days ago

Certificate-guided routing is how diffusion VLMs stop wasting compute on background tokens.

Ethan Satoshi7 days ago

Smarter visual routing could make VLMs much more efficient.

Matthew Broadstreet7 days ago

⏫

Related Videos

FP4 Explore, BF16 Train Diffusion Reinforcement Learning via Efficient Rollout Scaling paper:

FP4 Explore, BF16 Train Diffusion Reinforcement Learning via Efficient Rollout Scaling paper:

AK

12,817 views • 7 months ago

MolmoPoint Better Pointing for VLMs with Grounding Tokens paper: models: app:

MolmoPoint Better Pointing for VLMs with Grounding Tokens paper: models: app:

AK

20,057 views • 7 months ago

👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: 📰Paper: 💻Code: 1/n

👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: 📰Paper: 💻Code: 1/n

Zirui "Colin" Wang ✈️EMNLP

63,924 views • 5 months ago

Diffusion Policies and VLAs with Formal Safety Guarantees🛡️ We just released our new paper: “From Demonstrations to Safe Deployment: Path-Consistent Safety Filtering for Diffusion Policies” 📑Paper: 🌐Website:

Diffusion Policies and VLAs with Formal Safety Guarantees🛡️ We just released our new paper: “From Demonstrations to Safe Deployment: Path-Consistent Safety Filtering for Diffusion Policies” 📑Paper: 🌐Website:

Learning Systems and Robotics Lab (is hiring!)

20,534 views • 1 year ago

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.

Yunzhi Zhang

49,225 views • 1 year ago

What makes a maze look like a maze? Humans can reason about infinitely many instantiations of mazes—made of candy canes, sticks, icing, yarn, etc. But VLMs often struggle to make sense of such visual abstractions. We improve VLMs' ability to interpret these abstract concepts.

What makes a maze look like a maze? Humans can reason about infinitely many instantiations of mazes—made of candy canes, sticks, icing, yarn, etc. But VLMs often struggle to make sense of such visual abstractions. We improve VLMs' ability to interpret these abstract concepts.

Joy Hsu

43,528 views • 2 years ago

Excited to share our #CVPR2023 on synthesizing new views along a camera trajectory from a **single image**! How? 💡 The good old epipolar constraints in a pose-guided diffusion model! Paper: Project:

Excited to share our #CVPR2023 on synthesizing new views along a camera trajectory from a single image! How? 💡 The good old epipolar constraints in a pose-guided diffusion model! Paper: Project:

Jia-Bin Huang

94,199 views • 3 years ago

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 views • 1 year ago

Excited to share Norman Müller's DiffRF: Rendering-guided 3D Radiance Field Diffusion #CVPR2023 highlight! 2D diffusion is great, but what about 3D? We show radiance field diffusion with rendering guidance for consistent and editable 3D synthesis. Vid:

Excited to share Norman Müller's DiffRF: Rendering-guided 3D Radiance Field Diffusion #CVPR2023 highlight! 2D diffusion is great, but what about 3D? We show radiance field diffusion with rendering guidance for consistent and editable 3D synthesis. Vid:

Matthias Niessner

118,156 views • 3 years ago

Introducing Wave Link 3.0 (full public release). Audio mixing software built for all creators that delivers broadcast-level routing and effects. 🆕 Now free for Windows and Mac 🫵 Use your own mic 💡 Reimagined interface 👀 Visual routing table 🎙️ Support for multiple mics + ∞ software inputs 🔊 5 independent output mixes Now free for all, no Elgato hardware needed:

Introducing Wave Link 3.0 (full public release). Audio mixing software built for all creators that delivers broadcast-level routing and effects. 🆕 Now free for Windows and Mac 🫵 Use your own mic 💡 Reimagined interface 👀 Visual routing table 🎙️ Support for multiple mics + ∞ software inputs 🔊 5 independent output mixes Now free for all, no Elgato hardware needed:

Elgato

44,009 views • 8 months ago

When appliances run unnecessarily, your electricity bill keeps growing.Use them only when needed and adopt smart habits for a comfortable and efficient summer. #UPPCL #SmartUsage #EnergyAwareness #SaveElectricity CM Office, GoUP Energy Minister Office UP A K Sharma MDUPPCL

When appliances run unnecessarily, your electricity bill keeps growing.Use them only when needed and adopt smart habits for a comfortable and efficient summer. #UPPCL #SmartUsage #EnergyAwareness #SaveElectricity CM Office, GoUP Energy Minister Office UP A K Sharma MDUPPCL

UPPCL

28,266 views • 4 months ago

Making paper airplanes with efficient flight

Making paper airplanes with efficient flight

Today I Learned

241,866 views • 1 year ago

EasyControl just dropped on Hugging Face Adding Efficient and Flexible Control for Diffusion Transformer Free chatgpt style ghibli image generation with easy control

EasyControl just dropped on Hugging Face Adding Efficient and Flexible Control for Diffusion Transformer Free chatgpt style ghibli image generation with easy control

AK

222,239 views • 1 year ago

Vision-language models (VLMs) can see well, but they struggle to reason. In this episode, Antonia Wüst (PhD researcher, TU Darmstadt) explains how combining VLMs with program synthesis yields more reliable visual reasoning, with fewer tokens than chain-of-thought.

Vision-language models (VLMs) can see well, but they struggle to reason. In this episode, Antonia Wüst (PhD researcher, TU Darmstadt) explains how combining VLMs with program synthesis yields more reliable visual reasoning, with fewer tokens than chain-of-thought.

Ndea

22,130 views • 9 months ago

Continuous diffusion had a good run—now it’s time for Discrete diffusion! Introducing Anchored Posterior Sampling (APS) APS outperforms discrete and continuous baselines in terms of performance & scaling on inverse problems, stylization, and text-guided editing.

Continuous diffusion had a good run—now it’s time for Discrete diffusion! Introducing Anchored Posterior Sampling (APS) APS outperforms discrete and continuous baselines in terms of performance & scaling on inverse problems, stylization, and text-guided editing.

Litu Rout

40,315 views • 1 year ago

I built a RAG Routing Agent that exactly knows where to look. It automatically route queries to the right knowledge base and falls back to web search when needed. 100% Opensource Code with step-by-step tutorial.

I built a RAG Routing Agent that exactly knows where to look. It automatically route queries to the right knowledge base and falls back to web search when needed. 100% Opensource Code with step-by-step tutorial.

Shubham Saboo

80,579 views • 1 year ago

Introducing 🔥BLIP-Diffusion🔥, a novel method for enabling Text-to-image Diffusion models with multimodal controllable generation/editing, powered by BLIP-2 pre-trained text-aligned subject representation. Paper: Project: (1/n)

Introducing 🔥BLIP-Diffusion🔥, a novel method for enabling Text-to-image Diffusion models with multimodal controllable generation/editing, powered by BLIP-2 pre-trained text-aligned subject representation. Paper: Project: (1/n)

Steven Hoi

40,615 views • 3 years ago