Gabriel Sarch
@GabrielSarch • 1,145 subscribers
Postdoctoral Fellow @PrincetonPLI. Ph.D. @mldcmu @cmuneurosci. Prev. @yutori_ai @MSFTResearch.
Videos
How can we get VLMs to move their eyes—and reason step-by-step in visually grounded ways? 👀 We introduce ViGoRL, a RL method that anchors reasoning to image regions. 🎯 It outperforms vanilla GRPO and SFT across grounding, spatial tasks, and visual search (86.4% on V*). 👇🧵
Gabriel Sarch76,824 просмотров • 1 год назад
Achieve SOTA on multimodal, complex tasks with minimal supervision! Introducing In-Context Abstraction Learning (ICAL): a method that extracts insights from visual demos and human feedback to continually teach multimodal LLM agents 🚀🤖🧠 ICAL learns from sub-optimal demonstrations and human feedback by transforming noisy trajectories into optimized, annotated examples that improve multimodal decision-making in new domains. ICAL is the first to show state-of-the-art performance across three challenging domains: we obtain a 34% relative improvement on TEACh household robot instruction following from dialogue, a 59% relative improvement on Visual Web Arena for autonomous visual web agents, and beat few-shot GPT-4V and supervised baselines on Ego4D action forecasting.
Gabriel Sarch19,340 просмотров • 2 лет назад
Больше нет контента для загрузки