Загрузка видео...

Не удалось загрузить видео

На главную

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:

_akhaliq's profile picture

AK

538,286 subscribers

22,193 просмотров • 16 дней назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 7

Фото профиля Yu (Bryan) Zhou
Yu (Bryan) Zhou16 дней назад

Thanks for sharing our work! Here are some links for reference: Paper: Website: Code: Demo:

Фото профиля Van0SS
Van0SS16 дней назад

Woah supercool, would be super helpful with CUA work

Фото профиля DANY
DANY16 дней назад

agents with memory, finally

Фото профиля The SI Maximalist
The SI Maximalist16 дней назад

OpenVLHarness brings multimodal memory to real-world visual agentic tool-use.

Фото профиля chebyte
chebyte16 дней назад

the part i'd steal is keeping tool outputs as addressable artifacts instead of pasting them back into context. medium effort + harness beating max effort at ~37% of the tokens, my guess is a lot of the think-harder budget was the model re-reading crops it already had

Фото профиля Aleksandar Janca
Aleksandar Janca16 дней назад

visual tool use is where I would trust an agent last, I always reach for the api first because a screen breaks the day someone moves a button

Фото профиля Yu (Bryan) Zhou
Yu (Bryan) Zhou16 дней назад

For more details, check out our thread:

Похожие видео

EgoTools Towards Tool-Centric Reasoning in Real-World Egocentric Videos paper:
0:30

EgoTools Towards Tool-Centric Reasoning in Real-World Egocentric Videos paper:

AK

22,224 просмотров • 1 месяц назад

Wait Xiaomi has also released the best 9B model in the world?! This model is based on Qwen-9B... And you can easily run it even on 8GB of memory! - MIT license - Multimodal (image + text) - Solid for coding/agentic tasks Perfect to use locally and offline... even on a laptop with unified memory!
0:37

Wait Xiaomi has also released the best 9B model in the world?! This model is based on Qwen-9B... And you can easily run it even on 8GB of memory! - MIT license - Multimodal (image + text) - Solid for coding/agentic tasks Perfect to use locally and offline... even on a laptop with unified memory!

Paul Couvert

85,496 просмотров • 1 месяц назад

Welcome to the world, Gemini 2.0 ✨ our most capable AI model yet. We're first releasing an experimental version of 2.0 Flash ⚡ It has better performance, new multimodal output, Google tool use - and paves the way for new agentic experiences. 🧵
2:52

Welcome to the world, Gemini 2.0 ✨ our most capable AI model yet. We're first releasing an experimental version of 2.0 Flash ⚡ It has better performance, new multimodal output, Google tool use - and paves the way for new agentic experiences. 🧵

Google DeepMind

365,769 просмотров • 1 год назад

Gemini 3.7 Flash: "Play 1 round of Wordle" using my Android emulator via ADB. The latency and visual reasoning make Gemini 3.7 Flash exceptionally good for multimodal agentic use case like mobile control and Computer Use.
1:23

Gemini 3.7 Flash: "Play 1 round of Wordle" using my Android emulator via ADB. The latency and visual reasoning make Gemini 3.7 Flash exceptionally good for multimodal agentic use case like mobile control and Computer Use.

Philipp Schmid

18,290 просмотров • 2 месяцев назад

Excited to introduce Reka Vision, an agentic visual understanding and search platform. Transform your unstructured multimodal data into insights and actions.
0:34

Excited to introduce Reka Vision, an agentic visual understanding and search platform. Transform your unstructured multimodal data into insights and actions.

Reka

485,951 просмотров • 1 год назад

1/ muse spark 1.2 is a very strong multimodal model—it can do visual coding, robotics planning, and audio-visual understanding that all come together through agentic tools.
0:19

1/ muse spark 1.2 is a very strong multimodal model—it can do visual coding, robotics planning, and audio-visual understanding that all come together through agentic tools.

Alexandr Wang

185,121 просмотров • 2 месяцев назад

Machine Mental Imagery Empower Multimodal Reasoning with Latent Visual Tokens
0:34

Machine Mental Imagery Empower Multimodal Reasoning with Latent Visual Tokens

AK

43,552 просмотров • 1 год назад

Introducing GLM-5V-Turbo: Vision Coding Model - Native Multimodal Coding: Natively understands multimodal inputs including images, videos, design drafts, and document layouts. - Balanced Visual and Programming Capabilities: Achieves leading performance across core benchmarks for multimodal coding, tool use, and GUI Agents. - Deep Adaptation for Claude Code and Claw Scenarios: Works in deep synergy with Agents like Claude Code and OpenClaw. Try it now: API: Coding Plan trial applications:
1:17

Introducing GLM-5V-Turbo: Vision Coding Model - Native Multimodal Coding: Natively understands multimodal inputs including images, videos, design drafts, and document layouts. - Balanced Visual and Programming Capabilities: Achieves leading performance across core benchmarks for multimodal coding, tool use, and GUI Agents. - Deep Adaptation for Claude Code and Claw Scenarios: Works in deep synergy with Agents like Claude Code and OpenClaw. Try it now: API: Coding Plan trial applications:

Z.ai

1,971,219 просмотров • 7 месяцев назад

Gemma 4 is live on Cerebras, the fastest multimodal inference ever! Running on Gemma 4 31B open-weight model at a blistering 1,500+ tokens/sec. That's a 15x speedup, unlocking real-time visual and agentic loops without the GPU lag.
1:33

Gemma 4 is live on Cerebras, the fastest multimodal inference ever! Running on Gemma 4 31B open-weight model at a blistering 1,500+ tokens/sec. That's a 15x speedup, unlocking real-time visual and agentic loops without the GPU lag.

Google Gemma

114,159 просмотров • 3 месяцев назад

Visual agentic intelligence
3:47

Visual agentic intelligence

Qwen

113,347 просмотров • 3 месяцев назад

Introducing Large Visual Memory Model (LVMM) 2.0, in partnership with Qualcomm Advanced visual AI is now on your phone, and in your home cameras. AI that understands your photos and videos, runs 100% privately and securely. ALL on-device. The beginning of on-device multimodal AI.
6:46

Introducing Large Visual Memory Model (LVMM) 2.0, in partnership with Qualcomm Advanced visual AI is now on your phone, and in your home cameras. AI that understands your photos and videos, runs 100% privately and securely. ALL on-device. The beginning of on-device multimodal AI.

Shawn Shen

1,204,959 просмотров • 1 год назад

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.
0:22

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,481 просмотров • 3 лет назад

Text-to-SQL is dead 💀 The next generation of querying is agentic - and it’s already here. This paper ( introduces a type of agentic querying called 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗖𝗮𝗹𝗹𝗶𝗻𝗴 that uses an LLM to structure queries using predefined function calls in JSON format, with optional arguments for search, filters, aggregation, and grouping. Along with that, it also tests out a bunch of different models with a new dataset, DBGorilla, designed to evaluate agentic querying techniques on real-world use cases. Weaviate AI Database also just released a 𝗤𝘂𝗲𝗿𝘆 𝗔𝗴𝗲𝗻𝘁, designed based on some of the work in this paper, to handle advanced agentic querying out of the box, find out more here:
3:32

Text-to-SQL is dead 💀 The next generation of querying is agentic - and it’s already here. This paper ( introduces a type of agentic querying called 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗖𝗮𝗹𝗹𝗶𝗻𝗴 that uses an LLM to structure queries using predefined function calls in JSON format, with optional arguments for search, filters, aggregation, and grouping. Along with that, it also tests out a bunch of different models with a new dataset, DBGorilla, designed to evaluate agentic querying techniques on real-world use cases. Weaviate AI Database also just released a 𝗤𝘂𝗲𝗿𝘆 𝗔𝗴𝗲𝗻𝘁, designed based on some of the work in this paper, to handle advanced agentic querying out of the box, find out more here:

Victoria Slocum

189,664 просмотров • 1 год назад

Muse Spark 1.2 supports a broad range of multimodal tasks, from turning visuals into working code to translating perception into physical action. It also brings robust audio-visual understanding to enable video-heavy workflows common in real-world enterprise use. Today, we’re sharing new evals and demos that illustrate the breadth of the model’s visual understanding and reasoning capabilities. Let’s start with a demo that shows how Muse Spark parses multimodal observations and calls tools to guide a robot to navigate in an unstructured environment to find a rubber duck. 🧵👇
0:43

Muse Spark 1.2 supports a broad range of multimodal tasks, from turning visuals into working code to translating perception into physical action. It also brings robust audio-visual understanding to enable video-heavy workflows common in real-world enterprise use. Today, we’re sharing new evals and demos that illustrate the breadth of the model’s visual understanding and reasoning capabilities. Let’s start with a demo that shows how Muse Spark parses multimodal observations and calls tools to guide a robot to navigate in an unstructured environment to find a rubber duck. 🧵👇

AI at Meta

59,390 просмотров • 2 месяцев назад