Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:

22,589 Aufrufe • vor 29 Tagen •via X (Twitter)

7 Kommentare

Profilbild von Yu (Bryan) Zhou
Yu (Bryan) Zhouvor 28 Tagen

Thanks for sharing our work! Here are some links for reference: Paper: Website: Code: Demo:

Profilbild von Van0SS
Van0SSvor 29 Tagen

Woah supercool, would be super helpful with CUA work

Profilbild von DANY
DANYvor 29 Tagen

agents with memory, finally

Profilbild von The SI Maximalist
The SI Maximalistvor 29 Tagen

OpenVLHarness brings multimodal memory to real-world visual agentic tool-use.

Profilbild von chebyte
chebytevor 29 Tagen

the part i'd steal is keeping tool outputs as addressable artifacts instead of pasting them back into context. medium effort + harness beating max effort at ~37% of the tokens, my guess is a lot of the think-harder budget was the model re-reading crops it already had

Profilbild von Aleksandar Janca
Aleksandar Jancavor 29 Tagen

visual tool use is where I would trust an agent last, I always reach for the api first because a screen breaks the day someone moves a button

Profilbild von Yu (Bryan) Zhou
Yu (Bryan) Zhouvor 28 Tagen

For more details, check out our thread:

Ähnliche Videos

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,481 Aufrufe • vor 3 Jahren