Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Despite having vision, most AI agents still struggle to see. General-purpose multimodal models are powerful, but they’re expensive for every visual task. We built something better: Perceptron's MCP gives any agent stronger vision capabilities through Isaac with far lower cost.

Perceptron AI

3,592 subscribers

335,448 views • 5 months ago •via X (Twitter)

News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

The teams shipping AI agents right now are bleeding money on the dumbest possible expense: teaching a 400B-parameter model to read a file name. Every time an AI agent needs to "see" something today, it routes an image through a frontier model. OCR, object detection, checking if a button exists on screen. You're paying GPT-4o or Claude pricing for tasks that require perception, not reasoning. One agent workflow processing a few thousand screenshots per day can burn through more on vision calls than on the actual thinking. Perceptron's Isaac is 2B parameters. Built by the team that created Meta's Chameleon multimodal models. On perceptive benchmarks, it matches or beats models 50x its size. The VQA, OCR, and object detection scores are competitive with models running on infrastructure that costs orders of magnitude more. The MCP wrapper is the distribution play. One install command and every Claude Code agent can offload vision tasks to a model that runs on a single consumer GPU. The agent keeps its reasoning in the frontier model and routes perception to a specialist. That split is how you get vision-heavy agent workflows from "technically possible but expensive" to "cheap enough to run on everything." This is the same pattern that won in every other compute-intensive stack. General-purpose handles orchestration. Specialists handle the heavy lifting. Graphics went through it. Audio went through it. Video encoding went through it. Vision in AI agents is next. The teams building agents that see 10,000 images a day will care about this before anyone else does.

The teams shipping AI agents right now are bleeding money on the dumbest possible expense: teaching a 400B-parameter model to read a file name. Every time an AI agent needs to "see" something today, it routes an image through a frontier model. OCR, object detection, checking if a button exists on screen. You're paying GPT-4o or Claude pricing for tasks that require perception, not reasoning. One agent workflow processing a few thousand screenshots per day can burn through more on vision calls than on the actual thinking. Perceptron's Isaac is 2B parameters. Built by the team that created Meta's Chameleon multimodal models. On perceptive benchmarks, it matches or beats models 50x its size. The VQA, OCR, and object detection scores are competitive with models running on infrastructure that costs orders of magnitude more. The MCP wrapper is the distribution play. One install command and every Claude Code agent can offload vision tasks to a model that runs on a single consumer GPU. The agent keeps its reasoning in the frontier model and routes perception to a specialist. That split is how you get vision-heavy agent workflows from "technically possible but expensive" to "cheap enough to run on everything." This is the same pattern that won in every other compute-intensive stack. General-purpose handles orchestration. Specialists handle the heavy lifting. Graphics went through it. Audio went through it. Video encoding went through it. Vision in AI agents is next. The teams building agents that see 10,000 images a day will care about this before anyone else does.

Aakash Gupta

55,978 views • 5 months ago

🤯 Building computer-vision applications is never going to be the same again. As Andrej Karpathy says, "English is going to be the new programming-language", and we just made that happen for programming visual AI. 👨‍💻 This past month, the VLM Run team cooked up something magical that I'm especially proud of - a natural-language interface to build and run computer vision workflows (object-detection, OCR, parsing, and so much more) – all via our new VLM Run MCP server. 🔌 It works with any MCP-compatible AI agent (Claude, OpenAI, Cursor, etc.) and abstracts complex visual workflows into simple English commands. No need to write Python or fiddle with bespoke computer-vision code – just say what you want your agent to do, and it does it. Here’s a glimpse into what you can build today ( ☑️ Image AI agents – run detection, tagging, blurring, cropping, and more traditional CV tasks with English ☑️ Document AI agents – extract totals from invoices, search visual content, or act on what’s found ☑️ Video AI agents – understand scene content, trim clips, add subtitles, and create ready-to-share edits ☑️ And best-of-all, it's fully programmable with English. 🧑‍🍳 MCP Showcase: 📚 MCP Cookbooks: 📚 MCP Docs: #computervision #ai #agents #llm #vlm #mcp #vibecoding #genai

🤯 Building computer-vision applications is never going to be the same again. As Andrej Karpathy says, "English is going to be the new programming-language", and we just made that happen for programming visual AI. 👨‍💻 This past month, the VLM Run team cooked up something magical that I'm especially proud of - a natural-language interface to build and run computer vision workflows (object-detection, OCR, parsing, and so much more) – all via our new VLM Run MCP server. 🔌 It works with any MCP-compatible AI agent (Claude, OpenAI, Cursor, etc.) and abstracts complex visual workflows into simple English commands. No need to write Python or fiddle with bespoke computer-vision code – just say what you want your agent to do, and it does it. Here’s a glimpse into what you can build today ( ☑️ Image AI agents – run detection, tagging, blurring, cropping, and more traditional CV tasks with English ☑️ Document AI agents – extract totals from invoices, search visual content, or act on what’s found ☑️ Video AI agents – understand scene content, trim clips, add subtitles, and create ready-to-share edits ☑️ And best-of-all, it's fully programmable with English. 🧑‍🍳 MCP Showcase: 📚 MCP Cookbooks: 📚 MCP Docs: #computervision #ai #agents #llm #vlm #mcp #vibecoding #genai

Sudeep Pillai

17,789 views • 1 year ago

Today we are launching something BIG! The most accurate and fastest web search APIs for AI agents Something which took us months to build Something so powerful that it lets your AI agent to go from "guessing" -> "doing" Combined with our powerful people and company APIs, your AI agent can now find actionable answer to any question and finally act with confidence Whether you are building an AI agent in sales, recruitment, investment, insurance or just adding small automation which needs access to web data in structured manner, this is for you. Live on Producthunt now! Do upvote and support (link in comments)

Today we are launching something BIG! The most accurate and fastest web search APIs for AI agents Something which took us months to build Something so powerful that it lets your AI agent to go from "guessing" -> "doing" Combined with our powerful people and company APIs, your AI agent can now find actionable answer to any question and finally act with confidence Whether you are building an AI agent in sales, recruitment, investment, insurance or just adding small automation which needs access to web data in structured manner, this is for you. Live on Producthunt now! Do upvote and support (link in comments)

Abhilash Chowdhary

32,914 views • 7 months ago

THIS GUY BUILT A COMMAND CENTER FOR MANAGING AI AGENT TEAMS he was running 20+ agents through openclaw. all from terminal tabs and Discord. 12 tabs open. still losing track of who's doing what. which cron failed. so he built a dashboard that gives you one place to see everything: > org map showing the full agent hierarchy, who reports to who and what tools each agent has > chat with any agent through a real UI instead of bouncing between terminals > kanban board for tracking tasks across all agents > cron monitor with pipeline visualizations so you can see which jobs are failing > memory browser to read through all agent knowledge files in one place you can even point it at your openclaw workspace and it auto-discovers every agent from their config files. zero setup. AND ITS OPEN SOURCE TOO.

THIS GUY BUILT A COMMAND CENTER FOR MANAGING AI AGENT TEAMS he was running 20+ agents through openclaw. all from terminal tabs and Discord. 12 tabs open. still losing track of who's doing what. which cron failed. so he built a dashboard that gives you one place to see everything: > org map showing the full agent hierarchy, who reports to who and what tools each agent has > chat with any agent through a real UI instead of bouncing between terminals > kanban board for tracking tasks across all agents > cron monitor with pipeline visualizations so you can see which jobs are failing > memory browser to read through all agent knowledge files in one place you can even point it at your openclaw workspace and it auto-discovers every agent from their config files. zero setup. AND ITS OPEN SOURCE TOO.

Om Patel

46,780 views • 5 months ago

Today we are launching SEC Agent v0 . A specialized AI agent purpose built to dig through SEC filings, 10-Ks, and earnings transcripts, to find you an edge. As a part of the larger Xynth agent suite, it can be paired directly with other data like options flow, darkpool feeds, politician trades, and of course, regular live market data. We are on a mission to equip every trader with the most powerful alpha generation tools. RT + Comment “Alpha” and we’ll send it to you

Today we are launching SEC Agent v0 . A specialized AI agent purpose built to dig through SEC filings, 10-Ks, and earnings transcripts, to find you an edge. As a part of the larger Xynth agent suite, it can be paired directly with other data like options flow, darkpool feeds, politician trades, and of course, regular live market data. We are on a mission to equip every trader with the most powerful alpha generation tools. RT + Comment “Alpha” and we’ll send it to you

Xynth

102,801 views • 4 months ago

Modern AI is confined to the digital world. At Skild AI, we are building towards AGI for the real world, unconstrained by robot type or task — a single, omni-bodied brain. Today, we are sharing our journey, starting with early milestones, with more to come in the weeks ahead. Our Mission: Artificial General Intelligence grounded in the physical world. We believe AGI that can truly understand and reason in the real world can only be built through grounding in the physical world. Our Vision: Any robot, Any task, One brain. We tackle robotics in its full generality – building a continually improving, omni-bodied brain that can control any hardware for any task. Who are we? A passionate group of scientists & engineers driven by our shared vision. We have been researching AI and robotics for more than a decade. Our team includes pioneers of self-supervised learning, curiosity-driven exploration, end-to-end sim2real for visual locomotion, dexterous manipulation, learning from human videos, robot parkour, and many more. Many of these works have won awards at top-tier AI and Robotics conferences. Our team has also built production-ready systems at Anduril, Tesla, Nvidia, Meta, Kitty Hawk, Google, Everyday Robotics, and Amazon. Join us in our mission to build the robot brains of tomorrow.

Modern AI is confined to the digital world. At Skild AI, we are building towards AGI for the real world, unconstrained by robot type or task — a single, omni-bodied brain. Today, we are sharing our journey, starting with early milestones, with more to come in the weeks ahead. Our Mission: Artificial General Intelligence grounded in the physical world. We believe AGI that can truly understand and reason in the real world can only be built through grounding in the physical world. Our Vision: Any robot, Any task, One brain. We tackle robotics in its full generality – building a continually improving, omni-bodied brain that can control any hardware for any task. Who are we? A passionate group of scientists & engineers driven by our shared vision. We have been researching AI and robotics for more than a decade. Our team includes pioneers of self-supervised learning, curiosity-driven exploration, end-to-end sim2real for visual locomotion, dexterous manipulation, learning from human videos, robot parkour, and many more. Many of these works have won awards at top-tier AI and Robotics conferences. Our team has also built production-ready systems at Anduril, Tesla, Nvidia, Meta, Kitty Hawk, Google, Everyday Robotics, and Amazon. Join us in our mission to build the robot brains of tomorrow.

Skild AI

385,589 views • 1 year ago

Introducing LobeHub: Agent teammates that grow with you. LobeHub is the ultimate space for work and life: to find, build, and collaborate with agent teammates that grow with you. We’re building the world’s first and largest human–agent co-evolving network. Two years ago, we built LobeChat, an open-source interface for using different AI models. Today, LobeChat has 70k+ GitHub stars and serves 6M+ users worldwide. How to fully unlock the power of models has always been a shared mission between us and the community. We started with interaction — a fundamentally new, agent-first experience. Agents are no longer passive tools invoked in a single conversation. They should be proactive, always-on units of work. Treating agents as the minimal atomic unit is also the core of our agent harness infra. Today’s agents are mostly one-off executors. Even with memory, it’s often global — and hallucinates. We build long-term agent teammates that evolve with users. Each agent has its own dedicated memory space, editable by users, allowing humans and agents to co-evolve over time. This, in turn, allows us to design clearer rewards for reinforcement learning and create cleaner environments for continual learning. Agent teammates can work in groups. Through a multi-agent system, agent groups operate faster, more cost-effective, and go beyond what single-agent systems can achieve. For example, a single agent often requires heavy user involvement to proceed step by step, whereas LobeHub can execute the same work from a single instruction, with a supervisor orchestrating agents that run in parallel or debate to produce better results. We are building the collaboration network among agent teammates — and between humans and agent teammates as well. Ease of use matters. AI intelligence and shared human intelligence are equally important. With simple instructions and tool selection, you can effortlessly build and team up with agent coworkers to deliver complex, systematic work — even assembling a quant team to execute trades. Through the LobeHub community, anyone can discover, reuse, and remix agents and agent groups, customizing them to fit their own workflows, preferences, and needs. Last but not least, our vision started with LobeChat: multi-model support is the most efficient approach for users. We believe different models excel in different scenarios. By routing across multiple models, LobeHub improves cost efficiency and unlocks capabilities that a single-model setup cannot easily support.

Introducing LobeHub: Agent teammates that grow with you. LobeHub is the ultimate space for work and life: to find, build, and collaborate with agent teammates that grow with you. We’re building the world’s first and largest human–agent co-evolving network. Two years ago, we built LobeChat, an open-source interface for using different AI models. Today, LobeChat has 70k+ GitHub stars and serves 6M+ users worldwide. How to fully unlock the power of models has always been a shared mission between us and the community. We started with interaction — a fundamentally new, agent-first experience. Agents are no longer passive tools invoked in a single conversation. They should be proactive, always-on units of work. Treating agents as the minimal atomic unit is also the core of our agent harness infra. Today’s agents are mostly one-off executors. Even with memory, it’s often global — and hallucinates. We build long-term agent teammates that evolve with users. Each agent has its own dedicated memory space, editable by users, allowing humans and agents to co-evolve over time. This, in turn, allows us to design clearer rewards for reinforcement learning and create cleaner environments for continual learning. Agent teammates can work in groups. Through a multi-agent system, agent groups operate faster, more cost-effective, and go beyond what single-agent systems can achieve. For example, a single agent often requires heavy user involvement to proceed step by step, whereas LobeHub can execute the same work from a single instruction, with a supervisor orchestrating agents that run in parallel or debate to produce better results. We are building the collaboration network among agent teammates — and between humans and agent teammates as well. Ease of use matters. AI intelligence and shared human intelligence are equally important. With simple instructions and tool selection, you can effortlessly build and team up with agent coworkers to deliver complex, systematic work — even assembling a quant team to execute trades. Through the LobeHub community, anyone can discover, reuse, and remix agents and agent groups, customizing them to fit their own workflows, preferences, and needs. Last but not least, our vision started with LobeChat: multi-model support is the most efficient approach for users. We believe different models excel in different scenarios. By routing across multiple models, LobeHub improves cost efficiency and unlocks capabilities that a single-model setup cannot easily support.

LobeHub

185,401 views • 7 months ago

This one is for builders who want an agent that can operate their computer. Today, we're releasing HoloDesktop CLI. Powered by Holo3 models, it brings H Agent directly into the agent harnesses you already use, including Claude R Perrin Code, Hermes Trismegistus, Cursor, and others. It runs locally on your device with low latency, full privacy, and no runtime cost. Most AI tooling assumes work happens through APIs. In reality, much of the world's work still happens in interfaces built for humans: browsers, desktop applications, spreadsheets, internal tools, and software with no API. HoloDesktop CLI gives agents the ability to see, understand, and act on these environments through the mouse and keyboard. From testing web applications through their GUI to navigating enterprise software, extracting information from documents, or interacting with internal tools, agents can now operate where work actually happens. We believe AI shouldn't just reason about work. It should be able to operate where work happens. #NVIDIA #ComputerUse #EnterpriseAI #DeveloperTools

This one is for builders who want an agent that can operate their computer. Today, we're releasing HoloDesktop CLI. Powered by Holo3 models, it brings H Agent directly into the agent harnesses you already use, including Claude R Perrin Code, Hermes Trismegistus, Cursor, and others. It runs locally on your device with low latency, full privacy, and no runtime cost. Most AI tooling assumes work happens through APIs. In reality, much of the world's work still happens in interfaces built for humans: browsers, desktop applications, spreadsheets, internal tools, and software with no API. HoloDesktop CLI gives agents the ability to see, understand, and act on these environments through the mouse and keyboard. From testing web applications through their GUI to navigating enterprise software, extracting information from documents, or interacting with internal tools, agents can now operate where work actually happens. We believe AI shouldn't just reason about work. It should be able to operate where work happens. #NVIDIA #ComputerUse #EnterpriseAI #DeveloperTools

H

29,953 views • 2 months ago

HERMES JUST FIXED THE BIGGEST PROBLEM WITH BROWSER AGENTS Most AI agents still click around websites like confused interns. This new setup gives them the map. Hermes + → Hermes now connects to Browserbase’s new Browse hub → Browse launched with 100+ browser skills → Each skill is a plain-text playbook for a specific website or task → Your agent can search, preview, and install skills inside Hermes Why This Matters: ✓ Less random clicking ✓ Fewer timeouts ✓ Better form filling ✓ More reliable website navigation ✓ Skills can be edited, reused, bundled, and shared The Real Stack: → Hermes runs the agent → Browserbase handles cloud browser infrastructure → gives the agent site-specific skills → Vision helps when pages get weird → Bundles let you load repeat workflows faster The killer detail: If a website breaks, you don’t wait for the model to magically improve. You update the skill. Now your agent gets better forever. That’s the difference between an AI tool and an actual agent system.

HERMES JUST FIXED THE BIGGEST PROBLEM WITH BROWSER AGENTS Most AI agents still click around websites like confused interns. This new setup gives them the map. Hermes + → Hermes now connects to Browserbase’s new Browse hub → Browse launched with 100+ browser skills → Each skill is a plain-text playbook for a specific website or task → Your agent can search, preview, and install skills inside Hermes Why This Matters: ✓ Less random clicking ✓ Fewer timeouts ✓ Better form filling ✓ More reliable website navigation ✓ Skills can be edited, reused, bundled, and shared The Real Stack: → Hermes runs the agent → Browserbase handles cloud browser infrastructure → gives the agent site-specific skills → Vision helps when pages get weird → Bundles let you load repeat workflows faster The killer detail: If a website breaks, you don’t wait for the model to magically improve. You update the skill. Now your agent gets better forever. That’s the difference between an AI tool and an actual agent system.

Julian Goldie SEO

43,472 views • 3 months ago

Today we're introducing Postman Fabric Gateway—an AI-native agent gateway. Here's why it matters: agents will consume APIs at several orders of magnitude beyond traditional apps. Traditional API Gateways are not designed for AI agents. While many legacy vendors have tried to support agentic workloads, Postman customers have been asking us about something that is AI-native and doesn’t have the baggage of pre-AI architecture. Postman Fabric has been designed from the ground-up for agents with support for MCPs, CLIs, as well as APIs - both AI and deterministic APIs. We are building this with a modular architecture as we are still in the early innings of AI and there are many new patterns that are likely to emerge. One thing is certain though, agents are driven by APIs, and you can’t have agent-ready APIs without a purpose-built gateway. Through Postman Fabric, customers will be able to: 1️⃣ Have a single plane to manage all MCP servers - public or private 2️⃣ Expose internal APIs as MCP tools 3️⃣ Centralize SaaS credentials 4️⃣ Govern agent traffic and run policies at a consumer, route, or service level 5️⃣ Deploy on cloud or in a hybrid model We are opening up early access soon - let me know if you are interested in the comments or a DM!

Today we're introducing Postman Fabric Gateway—an AI-native agent gateway. Here's why it matters: agents will consume APIs at several orders of magnitude beyond traditional apps. Traditional API Gateways are not designed for AI agents. While many legacy vendors have tried to support agentic workloads, Postman customers have been asking us about something that is AI-native and doesn’t have the baggage of pre-AI architecture. Postman Fabric has been designed from the ground-up for agents with support for MCPs, CLIs, as well as APIs - both AI and deterministic APIs. We are building this with a modular architecture as we are still in the early innings of AI and there are many new patterns that are likely to emerge. One thing is certain though, agents are driven by APIs, and you can’t have agent-ready APIs without a purpose-built gateway. Through Postman Fabric, customers will be able to: 1️⃣ Have a single plane to manage all MCP servers - public or private 2️⃣ Expose internal APIs as MCP tools 3️⃣ Centralize SaaS credentials 4️⃣ Govern agent traffic and run policies at a consumer, route, or service level 5️⃣ Deploy on cloud or in a hybrid model We are opening up early access soon - let me know if you are interested in the comments or a DM!

Abhinav Asthana

18,188 views • 4 months ago

Google’s new Agent2Agent protocol is a key step toward a future of AI Agent interoperability. No single tool has all the data a user or business needs for most workflows, so we need Agents to be able to talk to each other. Salesforce will have AI Agents that understand the inner workings of CRM, Workday will have AI Agents that understand HR workflows, Box has AI Agents that understand content and documents, and so on. It’s easy to conceive of a world where we have tens of thousands of these tool Agents and then billions or trillions of customized Agents that are extensions of those. So openness becomes key. Workflows need data from multiple systems to complete the task — like a sales report that needs documents and CRM data or an HR task that needs HR policies and employee details — and that’s where A2A comes in. This protocol gives AI Agent providers a way of talking to each other and simplifying the way that Agents communicate, understand each other’s capabilities, and so on. It’s great to see more interoperability, along with MCP, emerging in the AI space right now, as this will define the future of most IT stacks.

Google’s new Agent2Agent protocol is a key step toward a future of AI Agent interoperability. No single tool has all the data a user or business needs for most workflows, so we need Agents to be able to talk to each other. Salesforce will have AI Agents that understand the inner workings of CRM, Workday will have AI Agents that understand HR workflows, Box has AI Agents that understand content and documents, and so on. It’s easy to conceive of a world where we have tens of thousands of these tool Agents and then billions or trillions of customized Agents that are extensions of those. So openness becomes key. Workflows need data from multiple systems to complete the task — like a sales report that needs documents and CRM data or an HR task that needs HR policies and employee details — and that’s where A2A comes in. This protocol gives AI Agent providers a way of talking to each other and simplifying the way that Agents communicate, understand each other’s capabilities, and so on. It’s great to see more interoperability, along with MCP, emerging in the AI space right now, as this will define the future of most IT stacks.

Aaron Levie

139,591 views • 1 year ago

⚡️PREMIERE⚡️ Santiago: The Last Sail Today, July 21st marks Ernest Hemingway’s 127th birthday. On this day, we present a new vision of the ambitious project we released two years ago, when we set out to create an AI adaptation of Hemingway’s legendary novel, The Old Man and the Sea. Back in 2024, our goal was to build an ultra-realistic AI film inspired by one of the most iconic stories ever written. But at the time, despite ultra-rapid progress, many of the things were still beyond what AI could achieve. Today, through our creative partnership with Invideo - the story finally receives the visuals it deserves with the help of Agent One. We recreated the film entirely from scratch — every character, every scene, every location — rebuilding the project with Agent One into a comprehensive 10-minute short AI film. SANTIAGO: THE LAST SAIL Out now. *stay tuned for a detailed breakdown & tutorial on how we did this with Agent One*

⚡️PREMIERE⚡️ Santiago: The Last Sail Today, July 21st marks Ernest Hemingway’s 127th birthday. On this day, we present a new vision of the ambitious project we released two years ago, when we set out to create an AI adaptation of Hemingway’s legendary novel, The Old Man and the Sea. Back in 2024, our goal was to build an ultra-realistic AI film inspired by one of the most iconic stories ever written. But at the time, despite ultra-rapid progress, many of the things were still beyond what AI could achieve. Today, through our creative partnership with Invideo - the story finally receives the visuals it deserves with the help of Agent One. We recreated the film entirely from scratch — every character, every scene, every location — rebuilding the project with Agent One into a comprehensive 10-minute short AI film. SANTIAGO: THE LAST SAIL Out now. stay tuned for a detailed breakdown & tutorial on how we did this with Agent One

Digital da Vincis

28,130 views • 1 month ago

Today, we're introducing not one but TWO new models, striking the balance between efficiency and quality to enable you to build production AI agents. — Gemini 3.6 Flash: Addresses efficiency feedback we received from Gemini 3.5 Flash with upgrades in coding, knowledge work, and multimodal tasks faster, more accurately, and with substantially fewer tokens per task — Gemini 3.5 Flash-Lite: Our fastest, most cost-effective 3.5-class model yet built for agentic workflows, hitting ~350 output tokens/sec with improved coding and overall quality Start building with these today via the Gemini API in Google AI Studio or try them out in the Google Gemini

Today, we're introducing not one but TWO new models, striking the balance between efficiency and quality to enable you to build production AI agents. — Gemini 3.6 Flash: Addresses efficiency feedback we received from Gemini 3.5 Flash with upgrades in coding, knowledge work, and multimodal tasks faster, more accurately, and with substantially fewer tokens per task — Gemini 3.5 Flash-Lite: Our fastest, most cost-effective 3.5-class model yet built for agentic workflows, hitting ~350 output tokens/sec with improved coding and overall quality Start building with these today via the Gemini API in Google AI Studio or try them out in the Google Gemini

Google AI

438,466 views • 1 month ago

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,440 views • 2 years ago

a16z speedrun request for startups: GUIs for Agents we’re still in the MS-DOS era of agents today - CLI, terminal sessions, file directories deleted by openclaw etc. while a small slice of silicon valley are power users, we're SO early for the rest of the world at Speedrun, we’re looking for bold founders excited to bring the power of agents to normies everywhere. there's a whole slew of products to be built here - from agent builders to marketplaces to managed infrastructure one broad idea we’re excited about are visual abstraction layers for agents. if you don't know exactly what you want, a command line / chat interface is paralyzing - you need to see options 1 example - think of a GUI or visual command center inspired by strategy games (ex. Factorio) where agents and workflows are represented graphically. skills, tools, MCP connections, background processes, etc could all be configured and shown visually in a workspace on UX, strategy games have long perfected agent management. zoom to get a birds-eye view of your agents, batch and queue orders via shortcuts, assign agents in multiplayer etc. a well-designed agent command center would make multi-agent orchestration for normies feel easy & intuitive most folks today still haven't moved beyond ChatGPT. the potential is enormous - just as Windows unlocked mass-market use of personal computers, the right visual abstraction layer could unlock agentic work for everyone - from individuals to enterprise teams if you share our vision, we'd love to chat!

a16z speedrun request for startups: GUIs for Agents we’re still in the MS-DOS era of agents today - CLI, terminal sessions, file directories deleted by openclaw etc. while a small slice of silicon valley are power users, we're SO early for the rest of the world at Speedrun, we’re looking for bold founders excited to bring the power of agents to normies everywhere. there's a whole slew of products to be built here - from agent builders to marketplaces to managed infrastructure one broad idea we’re excited about are visual abstraction layers for agents. if you don't know exactly what you want, a command line / chat interface is paralyzing - you need to see options 1 example - think of a GUI or visual command center inspired by strategy games (ex. Factorio) where agents and workflows are represented graphically. skills, tools, MCP connections, background processes, etc could all be configured and shown visually in a workspace on UX, strategy games have long perfected agent management. zoom to get a birds-eye view of your agents, batch and queue orders via shortcuts, assign agents in multiplayer etc. a well-designed agent command center would make multi-agent orchestration for normies feel easy & intuitive most folks today still haven't moved beyond ChatGPT. the potential is enormous - just as Windows unlocked mass-market use of personal computers, the right visual abstraction layer could unlock agentic work for everyone - from individuals to enterprise teams if you share our vision, we'd love to chat!

Jon Lai

199,793 views • 4 months ago

MOST AI AGENTS RESET TO ZERO EVERY TIME YOU OPEN A NEW SESSION. Hermes Agent was built to do the opposite. Every task compounds. Every workflow becomes reusable. Every session teaches the next one. That is the core thesis behind Hermes, and it completely changes how AI agents work. Unlike most AI agents built by software tool companies, Hermes was created by the researchers training the Hermes model family itself. Meaning: The same system used internally to generate training data is now open source and available to everyone. One install gives you access to: Claude. GPT. Grok. Local models via Ollama. Telegram, Discord, Slack, iMessage integrations. Custom skills. Persistent memory. CLI control. But the real unlock is the compounding loop: You give Hermes a task. It records every tool call, decision, and result. Then it converts successful workflows into reusable skills automatically. A Hermes setup running for 3 months becomes fundamentally smarter than a fresh install. That’s the big idea. One of the coolest parts: You literally watch the agent evolve in real time from a blank install into a personalized AI operator. Instead of watching Netflix today, spend 28 mins with this. Follow Dami-Defi for more AI systems, agent workflows, and infrastructure alpha.

MOST AI AGENTS RESET TO ZERO EVERY TIME YOU OPEN A NEW SESSION. Hermes Agent was built to do the opposite. Every task compounds. Every workflow becomes reusable. Every session teaches the next one. That is the core thesis behind Hermes, and it completely changes how AI agents work. Unlike most AI agents built by software tool companies, Hermes was created by the researchers training the Hermes model family itself. Meaning: The same system used internally to generate training data is now open source and available to everyone. One install gives you access to: Claude. GPT. Grok. Local models via Ollama. Telegram, Discord, Slack, iMessage integrations. Custom skills. Persistent memory. CLI control. But the real unlock is the compounding loop: You give Hermes a task. It records every tool call, decision, and result. Then it converts successful workflows into reusable skills automatically. A Hermes setup running for 3 months becomes fundamentally smarter than a fresh install. That’s the big idea. One of the coolest parts: You literally watch the agent evolve in real time from a blank install into a personalized AI operator. Instead of watching Netflix today, spend 28 mins with this. Follow Dami-Defi for more AI systems, agent workflows, and infrastructure alpha.

Dami-Defi

11,685 views • 3 months ago

Most AI demos stop at text prompts. I wanted to see what Qwen3.7-Plus could do beyond chat, so I tested it through the API using an Airbnb interface screenshot. The model analyzed the UI, identified key components, explained the visual hierarchy, highlighted usability strengths, suggested UX improvements, and even generated a React + Tailwind structure to recreate the interface. A few things that stood out: 👁️ Multimodal understanding from a single screenshot 🧠 Strong reasoning around UX and product design decisions 💻 Able to bridge visual inputs and code generation ⚡ OpenAI-compatible API that was easy to integrate 💰 Impressive capability-to-cost ratio Qwen3.7-Plus isn't just a chatbot. It's a multimodal agent model designed for workflows that combine vision, reasoning, coding, and automation. Try Qwen3.7-Plus with limited time 20% off:

Most AI demos stop at text prompts. I wanted to see what Qwen3.7-Plus could do beyond chat, so I tested it through the API using an Airbnb interface screenshot. The model analyzed the UI, identified key components, explained the visual hierarchy, highlighted usability strengths, suggested UX improvements, and even generated a React + Tailwind structure to recreate the interface. A few things that stood out: 👁️ Multimodal understanding from a single screenshot 🧠 Strong reasoning around UX and product design decisions 💻 Able to bridge visual inputs and code generation ⚡ OpenAI-compatible API that was easy to integrate 💰 Impressive capability-to-cost ratio Qwen3.7-Plus isn't just a chatbot. It's a multimodal agent model designed for workflows that combine vision, reasoning, coding, and automation. Try Qwen3.7-Plus with limited time 20% off:

Kuria | AI

17,044 views • 3 months ago

We built an open-source research platform for agents utilizing Andrej Karpathy's autoresearch agent. agentipedia Crowd-sourced research will be the single biggest impact point for AI in the next 5 years and agentipedia will be a platform to drive it. Our vision: > There are potentially millions of niche use cases of research agents building strategies, better models, operating procedures and more. PHD level science is possible through agent collaboration. > Right now, very few entities control the vast majority of resources that can power this research; we believe in the future where any curious soul can harness the same energy. > Agentipedia was created to let curious souls (ML Engineers, Executives/CEOs, Founders, Builders, or literally anyone) think of a hypothesis for any application and be met with a swarm of agents experimenting to see if its true. Collaboration will yield magnitudes of impact on our society that we have not yet seen. The simulators for several use cases like (drug discovery, autonomous driving) and more already exist today.

We built an open-source research platform for agents utilizing Andrej Karpathy's autoresearch agent. agentipedia Crowd-sourced research will be the single biggest impact point for AI in the next 5 years and agentipedia will be a platform to drive it. Our vision: > There are potentially millions of niche use cases of research agents building strategies, better models, operating procedures and more. PHD level science is possible through agent collaboration. > Right now, very few entities control the vast majority of resources that can power this research; we believe in the future where any curious soul can harness the same energy. > Agentipedia was created to let curious souls (ML Engineers, Executives/CEOs, Founders, Builders, or literally anyone) think of a hypothesis for any application and be met with a swarm of agents experimenting to see if its true. Collaboration will yield magnitudes of impact on our society that we have not yet seen. The simulators for several use cases like (drug discovery, autonomous driving) and more already exist today.

invisible

30,345 views • 6 months ago