正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Large multimodal models often lack precise low-level perception needed for high-level reasoning, even with simple vector graphics. We bridge this gap by proposing an intermediate symbolic representation that leverages LLMs for text-based reasoning. 🧵1/4

Zhenhailong Wang

1,294 subscribers

37,320 次观看 • 2 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 条评论

Zhenhailong Wang 的头像

Zhenhailong Wang2 年前

Introducing Visually Descriptive Language Model (VDLM), a text-based model for visual reasoning on vector graphics. VDLM leverages SVG as a more precise initial perception of input images, and learns Primal Visual Descriptions to directly harness off-the-shelf LLMs.🧵2/4

Zhenhailong Wang 的头像

Zhenhailong Wang2 年前

VDLM achieves stronger zero-shot performance compared to SOTA LMMs, such as GPT-4V, in various low-level multimodal perception and reasoning tasks on vector graphics. VDLM also offers better interpretability due to its disentangled perception and reasoning processes. 🧵3/4

Zhenhailong Wang 的头像

Zhenhailong Wang2 年前

Special thanks to the exceptional collaborators and mentors! @zhenhailongW @joycjhsu @xingyaow_ @kuanhaoh_ @ManlingLi_ @jiajunwu_cs @hengjinlp Code: Paper: 🧵4/4

Runsen Xu 的头像

Runsen Xu2 年前

Good video demo! Very clear!

Money&abc 的头像

Money&abc2 年前

@PublicAI_ #AI

相关视频

One core bottleneck in VLA models is action representation. Discrete tokens scale beautifully with VLM pretraining—but lose precision. Continuous actions are precise—but often break VLM reasoning. In our new work, we resolve this tension at the representation level. 🧵

One core bottleneck in VLA models is action representation. Discrete tokens scale beautifully with VLM pretraining—but lose precision. Continuous actions are precise—but often break VLM reasoning. In our new work, we resolve this tension at the representation level. 🧵

Jianlan Luo

47,993 次观看 • 9 个月前

Excited to share RoCo: Dialectic Multi-Robot Collaboration with Large Language Models. We propose a novel approach to multi-robot collaboration that leverages LLMs for both high-level communication and low-level path planning. w/ Shreeya Jain, Shuran Song

Excited to share RoCo: Dialectic Multi-Robot Collaboration with Large Language Models. We propose a novel approach to multi-robot collaboration that leverages LLMs for both high-level communication and low-level path planning. w/ Shreeya Jain, Shuran Song

Mandi Zhao

88,722 次观看 • 3 年前

New research from Meta FAIR: Large Concept Models (LCM) is a fundamentally different paradigm for language modeling that decouples reasoning from language representation, inspired by how humans can plan high-level thoughts to communicate.

New research from Meta FAIR: Large Concept Models (LCM) is a fundamentally different paradigm for language modeling that decouples reasoning from language representation, inspired by how humans can plan high-level thoughts to communicate.

AI at Meta

531,765 次观看 • 1 年前

Flow reversal steering allows "steering" diffusion-based VLAs with high-level actions, for example from VLM reasoning. This also lets us run RL in the diffusion noise space with exploration guided by high-level reasoning: think through a task, then practice it! 👇

Flow reversal steering allows "steering" diffusion-based VLAs with high-level actions, for example from VLM reasoning. This also lets us run RL in the diffusion noise space with exploration guided by high-level reasoning: think through a task, then practice it! 👇

Sergey Levine

90,281 次观看 • 4 个月前

Have vision-language models achieved human-level compositional reasoning? Our research suggests: not quite yet. We’re excited to present CREPE – a large-scale Compositional REPresentation Evaluation benchmark for vision-language models – as a 🌟highlight🌟at #CVPR2023. 🧵1/7

Have vision-language models achieved human-level compositional reasoning? Our research suggests: not quite yet. We’re excited to present CREPE – a large-scale Compositional REPresentation Evaluation benchmark for vision-language models – as a 🌟highlight🌟at #CVPR2023. 🧵1/7

Zixian Ma

21,382 次观看 • 3 年前

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀

Abhishek Gupta

22,354 次观看 • 1 年前

1/N Introducing🌈PRISM-Physics, a process-level, rule-based benchmark for complex physics reasoning. Each solution is modeled as a Directed Acyclic Graph of formulas, capturing causal relations between steps. A rule-based symbolic equivalence checker ensures consistent evaluation

1/N Introducing🌈PRISM-Physics, a process-level, rule-based benchmark for complex physics reasoning. Each solution is modeled as a Directed Acyclic Graph of formulas, capturing causal relations between steps. A rule-based symbolic equivalence checker ensures consistent evaluation

Wanjia Zhao

57,765 次观看 • 11 个月前

Meet Alpamayo 2 Super, now commercially available for robotaxis and autonomous vehicles. Built for complex real-world driving, this open reasoning model adds 360° awareness, high-level driving decisions and automated reasoning labels. We built it, so you can build on it.

Meet Alpamayo 2 Super, now commercially available for robotaxis and autonomous vehicles. Built for complex real-world driving, this open reasoning model adds 360° awareness, high-level driving decisions and automated reasoning labels. We built it, so you can build on it.

NVIDIA

131,872 次观看 • 2 个月前

1/4 LLMs solve research grade math problems but struggle with basic calculations. We bridge this gap by turning them to computers. We built a computer INSIDE a transformer that can run programs for millions of steps in seconds solving even the hardest Sudokus with 100% accuracy

1/4 LLMs solve research grade math problems but struggle with basic calculations. We bridge this gap by turning them to computers. We built a computer INSIDE a transformer that can run programs for millions of steps in seconds solving even the hardest Sudokus with 100% accuracy

Christos Tzamos

1,832,398 次观看 • 7 个月前

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Thomas Kipf

98,036 次观看 • 2 年前

They really do think that Europe was originally populated by Blacks and Browns. There's no reasoning with this level of stupid.

They really do think that Europe was originally populated by Blacks and Browns. There's no reasoning with this level of stupid.

Holly Grayle

10,867 次观看 • 6 个月前

I’m excited to announce that 💫StarVector has been accepted at CVPR 2025! Over a year in the making, StarVector opens a new paradigm for Scalable Vector Graphics (SVG) generation by harnessing multimodal LLMs to generate SVG code that aesthetically mirrors input images and text. With this milestone, we’re also releasing StarVector on Hugging Face! 🥳🚀

I’m excited to announce that 💫StarVector has been accepted at CVPR 2025! Over a year in the making, StarVector opens a new paradigm for Scalable Vector Graphics (SVG) generation by harnessing multimodal LLMs to generate SVG code that aesthetically mirrors input images and text. With this milestone, we’re also releasing StarVector on Hugging Face! 🥳🚀

Joan Rodriguez

16,410 次观看 • 1 年前

Demis Hassabis on the next 12 Months: - Full multimodal convergence: Models like Gemini will seamlessly take in and output text, images, audio, and video, with cross-pollination that boosts reasoning + creativity. - Breakthrough visual intelligence: Image models like Nano Banana Pro will produce highly accurate infographics and show near-human visual understanding. - Language + video fusion: Video models integrated with LLMs unlock richer analysis, storytelling, and step-by-step visual reasoning. - World models go mainstream like Genie 3 - Agents become reliable

Demis Hassabis on the next 12 Months: - Full multimodal convergence: Models like Gemini will seamlessly take in and output text, images, audio, and video, with cross-pollination that boosts reasoning + creativity. - Breakthrough visual intelligence: Image models like Nano Banana Pro will produce highly accurate infographics and show near-human visual understanding. - Language + video fusion: Video models integrated with LLMs unlock richer analysis, storytelling, and step-by-step visual reasoning. - World models go mainstream like Genie 3 - Agents become reliable

Chubby♨️

89,236 次观看 • 10 个月前

Introducing 🔥BLIP-Diffusion🔥, a novel method for enabling Text-to-image Diffusion models with multimodal controllable generation/editing, powered by BLIP-2 pre-trained text-aligned subject representation. Paper: Project: (1/n)

Introducing 🔥BLIP-Diffusion🔥, a novel method for enabling Text-to-image Diffusion models with multimodal controllable generation/editing, powered by BLIP-2 pre-trained text-aligned subject representation. Paper: Project: (1/n)

Steven Hoi

40,615 次观看 • 3 年前

[1/n] Do distinct large models admit a simple map that aligns their embedding spaces? We show that across multimodal contrastive models—trained on different data and architectures—an orthogonal map aligns image embeddings. Strikingly, the same map also aligns text embeddings.

[1/n] Do distinct large models admit a simple map that aligns their embedding spaces? We show that across multimodal contrastive models—trained on different data and architectures—an orthogonal map aligns image embeddings. Strikingly, the same map also aligns text embeddings.

Sharut Gupta

37,899 次观看 • 8 个月前

Multimodal Reasoning AI Agents are here with Gemini 2.0 Flash Thinking I built a multimodal AI agent that can reason and understand images using gemini flash reasoning LLM. 100% Opensource Code with step-by-step tutorial.

Multimodal Reasoning AI Agents are here with Gemini 2.0 Flash Thinking I built a multimodal AI agent that can reason and understand images using gemini flash reasoning LLM. 100% Opensource Code with step-by-step tutorial.

Shubham Saboo

36,596 次观看 • 1 年前

introducing Seedream 5.0 Pro. this new multimodal model brings advance reasoning, precise image editing, and support for up to 10 image references. try it now on Krea.

introducing Seedream 5.0 Pro. this new multimodal model brings advance reasoning, precise image editing, and support for up to 10 image references. try it now on Krea.

Krea

38,896 次观看 • 3 个月前

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,681 次观看 • 3 年前