Loading video...

Video Failed to Load

Go Home

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻...

CamelAIOrg's profile picture

CAMEL-AI.org

8,634 subscribers

72,807 views • 2 years ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

8 Comments

CAMEL-AI.org's profile picture
CAMEL-AI.org2 years ago

Of course, that wouldn't be possible thanks to the amazing contributors from our @CamelAIOrg community. @TianqiXu233 (KAUST), Linyao Chen (UTokyo), Dai-jie Wu (KAUST), Yanjun Chen (CMU), @zechengzh, Xiang Yao, @Zhiqiang_Xie (Stanford), @YongchaoC (Harvard), @atasteoff (Tsinghua), Bochen Qian (SUSTech), @philiptorr (Oxford), @BernardSGhanem (KAUST), @guohao_li (Oxford)

CAMEL-AI.org's profile picture
CAMEL-AI.org2 years ago

If you are interested in learning more, check out this blog post:

omni_georgio's profile picture
omni_georgio2 years ago

🚀

Happy's profile picture
Happy2 years ago

amazing

CAMEL-AI.org's profile picture
CAMEL-AI.org2 years ago

Thanks, more to come!

Tianqi Xu's profile picture
Tianqi Xu2 years ago

Thank @omni_georgio for making this awesome video! We target to make CRAB the best LLM agent benchmark framework.

OP's profile picture
OP2 years ago

hey @CamelAIOrg . That's promissing. Added CRAB to the new specialized directory for AI Agents and frameworks for building them. Now you can upvote it.

KALALA NZENIELE's profile picture
KALALA NZENIELE2 years ago

@DFintelligence

Related Videos

Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬 We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. The key contributions of our work are: 🕹️ Enables the exploration of different training algorithms for AI Research Agents such as RL 🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions 🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness 🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning. 📈 Proposes a new evaluation metric for AI Research Agents MLGym makes it easy to: 1) Add new tasks 2) Evaluate new models 3) Integrate new agents Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩‍💻, experimentation 👩‍🔬, and iteration 🔄 to improve on ML tasks. Huge thanks to the exceptionally talented Deepak Nathani who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀
1:32

Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬 We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. The key contributions of our work are: 🕹️ Enables the exploration of different training algorithms for AI Research Agents such as RL 🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions 🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness 🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning. 📈 Proposes a new evaluation metric for AI Research Agents MLGym makes it easy to: 1) Add new tasks 2) Evaluate new models 3) Integrate new agents Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩‍💻, experimentation 👩‍🔬, and iteration 🔄 to improve on ML tasks. Huge thanks to the exceptionally talented Deepak Nathani who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀

Roberta Raileanu

105,258 views • 1 year ago

Introducing Giga's browser agent. Giga can now log in and completes tasks end-to-end, just like your human agents. It used to take months to get support APIs deployed for AI agents in the enterprise. Now these integrations can be built instantly.
1:16

Introducing Giga's browser agent. Giga can now log in and completes tasks end-to-end, just like your human agents. It used to take months to get support APIs deployed for AI agents in the enterprise. Now these integrations can be built instantly.

Esha

1,482,656 views • 9 months ago

Communities can now build with a coordinated team of AI agents. Agent execution is live on connecting community decisions to the agents that carry them out. $ZZZ brings the full workflow into one workspace: Discussion → decisions → tasks → execution. 1. Members chat and vote. An agent creates tasks from those discussions and records the decisions. 2. The lead agent prioritises tasks, clarifies the details with members and presents interactive mockups for approval. 3. The lead agent coordinates a swarm of agents to complete the tasks. Watch a community discussion become a task, then see the agents get to work.
0:35

Communities can now build with a coordinated team of AI agents. Agent execution is live on connecting community decisions to the agents that carry them out. $ZZZ brings the full workflow into one workspace: Discussion → decisions → tasks → execution. 1. Members chat and vote. An agent creates tasks from those discussions and records the decisions. 2. The lead agent prioritises tasks, clarifies the details with members and presents interactive mockups for approval. 3. The lead agent coordinates a swarm of agents to complete the tasks. Watch a community discussion become a task, then see the agents get to work.

zzz

15,894 views • 21 days ago

Tencent announces AppAgent Multimodal Agents as Smartphone Users paper page: Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.
1:57

Tencent announces AppAgent Multimodal Agents as Smartphone Users paper page: Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.

AK

344,019 views • 2 years ago

That's a lot of crab eggs! This guy is going to help this crab by removing the barnicles. 🦀 🦀
3:18

That's a lot of crab eggs! This guy is going to help this crab by removing the barnicles. 🦀 🦀

@LynneK

172,935 views • 7 months ago

Reasoning models tailored for diverse AI tasks. 🚀 Meet Amazon Nova 2 foundation models, supporting fast, cost-effective reasoning to multimodal capabilities. Power versatile tasks like AI agents, code-generation, and Conversational AI. Choose the perfect match for your workload.
0:29

Reasoning models tailored for diverse AI tasks. 🚀 Meet Amazon Nova 2 foundation models, supporting fast, cost-effective reasoning to multimodal capabilities. Power versatile tasks like AI agents, code-generation, and Conversational AI. Choose the perfect match for your workload.

Amazon Web Services

2,074,736 views • 11 months ago

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]
0:32

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

Bowen Wang

93,360 views • 1 year ago

Introducing: ✨Nexus✨ Nexus is a tool to create, run and monitor AI agents. You can connect these agents to computers, phones and even robots. Our vision-first AI agents interact with environments much like humans do, enabling them to perform tasks that were previously impossible for any other agent. Nexus lets anyone control an army of AI agents through pure natural language. ‼️
1:43

Introducing: ✨Nexus✨ Nexus is a tool to create, run and monitor AI agents. You can connect these agents to computers, phones and even robots. Our vision-first AI agents interact with environments much like humans do, enabling them to perform tasks that were previously impossible for any other agent. Nexus lets anyone control an army of AI agents through pure natural language. ‼️

Jan Schnyder

27,904 views • 1 year ago

It’s getting insanely easy to build complex multi-agent systems in n8n. You can now build supervisor agents that delegates tasks to specialized sub agents. Just add the sub agents via AI Agent Tool node. All in one place and a lot easier to debug! Zero code!
1:46

It’s getting insanely easy to build complex multi-agent systems in n8n. You can now build supervisor agents that delegates tasks to specialized sub agents. Just add the sub agents via AI Agent Tool node. All in one place and a lot easier to debug! Zero code!

elvis

45,690 views • 1 year ago

"I really don't want to manage an AI team." Cathryn found a solution: Paperclip, the open-source project What she showed me: • Paperclip leads her agents using its project management setup • Humans on her team use it to assign tasks to agents • Agents delegate tasks to humans or other agents • Paperclip turns your goals into agent tasks • It turns an SEO audit doc (for example) into agent tasks • It organizes OpenClaw agents OR even creates its own agents Also: fast-forward to 9min25sec to see a 3-minute Paperclip setup. (YouTube version in first comment.)
12:48

"I really don't want to manage an AI team." Cathryn found a solution: Paperclip, the open-source project What she showed me: • Paperclip leads her agents using its project management setup • Humans on her team use it to assign tasks to agents • Agents delegate tasks to humans or other agents • Paperclip turns your goals into agent tasks • It turns an SEO audit doc (for example) into agent tasks • It organizes OpenClaw agents OR even creates its own agents Also: fast-forward to 9min25sec to see a 3-minute Paperclip setup. (YouTube version in first comment.)

Andrew Warner

54,178 views • 7 months ago

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail
3:02

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail

LangChain

33,192 views • 1 year ago

The town hall for your agents 🦀
0:30

The town hall for your agents 🦀

moltx

34,980 views • 9 months ago

I built a framework for co-ordinating AFK coding agents. It's called Sandcastle. Watch me use it to pick tasks, parallelize N coding agents, and merge the code - all AFK:
5:06

I built a framework for co-ordinating AFK coding agents. It's called Sandcastle. Watch me use it to pick tasks, parallelize N coding agents, and merge the code - all AFK:

Matt Pocock

60,298 views • 7 months ago

🦀 Crab Tie 🦀 Don’t wait to find out the true potential of the ever-popular Moon Tie! With emphasis on the exposure and vulnerability of the model, these variations on the floor and in suspension are perfect for semenawa. ➡️
0:27

🦀 Crab Tie 🦀 Don’t wait to find out the true potential of the ever-popular Moon Tie! With emphasis on the exposure and vulnerability of the model, these variations on the floor and in suspension are perfect for semenawa. ➡️

Shibari Study

52,952 views • 3 years ago

Agents are getting complex to manage: models, skills, automations, permissions, cloud environments, orchestration patterns... So, we built the Terraform for agents: one repository that configures all of your agents as code. The Warp Factories configuration spec defines: - Agents and orchestration: how work gets routed and which models agents use - Automations: the events and schedules that kick off work - Access and environments: repositories, secrets, MCP servers, and runners - Measurement: scorers and benchmarks for understanding and improving performance Because it’s all code, both engineers and agents can propose changes to the factory itself. That makes it possible to test new configurations, benchmark them, and even have your factory propose improvements as PRs.
0:21

Agents are getting complex to manage: models, skills, automations, permissions, cloud environments, orchestration patterns... So, we built the Terraform for agents: one repository that configures all of your agents as code. The Warp Factories configuration spec defines: - Agents and orchestration: how work gets routed and which models agents use - Automations: the events and schedules that kick off work - Access and environments: repositories, secrets, MCP servers, and runners - Measurement: scorers and benchmarks for understanding and improving performance Because it’s all code, both engineers and agents can propose changes to the factory itself. That makes it possible to test new configurations, benchmark them, and even have your factory propose improvements as PRs.

Warp

756,814 views • 1 month ago

How capable are web agents at solving knowledge work tasks? 🤔 Are LLMs up to the challenge? 🤖 Introducing WorkArena: a benchmark where agents meet the world 𝘸𝘪𝘭𝘥 web of enterprise software 🌐🖥️ Paper: Website: 🧵 1/7
0:25

How capable are web agents at solving knowledge work tasks? 🤔 Are LLMs up to the challenge? 🤖 Introducing WorkArena: a benchmark where agents meet the world 𝘸𝘪𝘭𝘥 web of enterprise software 🌐🖥️ Paper: Website: 🧵 1/7

Alexandre Lacoste

24,550 views • 2 years ago

🚨 Announcing AutoBots - Recursively Self-Improving Agents (RSI) Super excited to launch our multi-LLM-based recursively self-improving agents We use DeepSeek Flash 4 for easy tasks and Fable 5 for super hard ones. The agents learn over time, becoming MORE efficient and smarter with each run We use these RSI agents to run our company! It's amazingly good with little to no human intervention required Examples include bug-fixing agents, viral tweet- or video-makers, growth-hacking agents, and so on. AutoBots work in the background to help increase your revenue.
1:11

🚨 Announcing AutoBots - Recursively Self-Improving Agents (RSI) Super excited to launch our multi-LLM-based recursively self-improving agents We use DeepSeek Flash 4 for easy tasks and Fable 5 for super hard ones. The agents learn over time, becoming MORE efficient and smarter with each run We use these RSI agents to run our company! It's amazingly good with little to no human intervention required Examples include bug-fixing agents, viral tweet- or video-makers, growth-hacking agents, and so on. AutoBots work in the background to help increase your revenue.

Bindu Reddy

2,504,428 views • 3 months ago

Pouring Mud into a Crab Hole to Catch Crab! 🦀🪣😱
0:51

Pouring Mud into a Crab Hole to Catch Crab! 🦀🪣😱

Sangeetha

551,409 views • 2 months ago

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇
1:26

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,697 views • 2 years ago