Large multimodal models often lack precise low-level perception needed for high-level reasoning, even with simple vector graphics. We bridge this gap by proposing an intermediate symbolic representation that leverages LLMs for text-based reasoning. 🧵1/4

Introducing Visually Descriptive Language Model (VDLM), a text-based model for visual reasoning on vector graphics. VDLM leverages SVG as a more precise initial perception of input images, and learns Primal Visual Descriptions to directly harness off-the-shelf LLMs.🧵2/4

VDLM achieves stronger zero-shot performance compared to SOTA LMMs, such as GPT-4V, in various low-level multimodal perception and reasoning tasks on vector graphics. VDLM also offers better interpretability due to its disentangled perception and reasoning processes. 🧵3/4

Special thanks to the exceptional collaborators and mentors! @zhenhailongW @joycjhsu @xingyaow_ @kuanhaoh_ @ManlingLi_ @jiajunwu_cs @hengjinlp Code: Paper: 🧵4/4

Good video demo! Very clear!

@PublicAI_ #AI