Zirui "Colin" Wang ✈️EMNLP
@zwcolin • 1,495 subscribers
Research Intern @MetaAI; CS PhD Student @Berkeley_AI and @BerkeleySky; prev @Princeton_NLP, @HDSIUCSD, @VoioInc multimodal interaction
Videos
👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: 📰Paper: 💻Code: 1/n
Zirui "Colin" Wang ✈️EMNLP63,924 views • 5 months ago
![🎮 We release VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (w/ Junyi Zhang @aomaru_21490) 🌐 With 17 environments across multiple domains, we show systematically the brittleness of VLMs in visual interaction, and what training leads to. 🧵[1/8]](https://image.24vids.com/tw-2015812327338287227/amplify_video_thumb/2015812237521547264/img/mQ2OuiGsLYkGmhrM.jpg)
🎮 We release VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (w/ Junyi Zhang @aomaru_21490) 🌐 With 17 environments across multiple domains, we show systematically the brittleness of VLMs in visual interaction, and what training leads to. 🧵[1/8]
Zirui "Colin" Wang41,381 views • 9 months ago
🤨 Are Multimodal Large Language Models really as 𝐠𝐨𝐨𝐝 at 𝐜𝐡𝐚𝐫𝐭 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 as existing benchmarks such as ChartQA suggest? 🚫 Our ℂ𝕙𝕒𝕣𝕏𝕚𝕧 benchmark suggests NO! 🥇Humans achieve ✨𝟖𝟎+% correctness. 🥈Sonnet 3.5 outperforms GPT-4o by 10+ points, reaching 🌟𝟔𝟎% correctness. 🥉Open-weight models are capped at ⭐𝟑𝟐% correctness. 🪜 Leaderboard: 📜 Preprint: 📊 Charxiv is ✨𝟏𝟎𝟎% handcrafted with rigorous human validation, and it reveals substantial gaps among Multimodal Large Language Models and humans in chart understanding. 🎥👇 80 second video (🎶sound on!). 🧶 1/6
Zirui "Colin" Wang ✈️CoRL48,494 views • 2 years ago
No more content to load