正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic... show more

Junxian He

6,464 subscribers

44,403 次观看 • 10 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

14 条评论

Harsh Trivedi 的头像

Harsh Trivedi10 个月前

Nice work, @junxian_he, @syz0x1, @gneubig & many others! Thanks for comparing with AppWorld! A couple of notes on it (your Table 1): The AppWorld row for "Avg # Turns" is "--" 👉 AppWorld tasks are very long-range in terms of # turns & # tool calls. See the histogram of tasks% requiring N tool calls is in our Fig. 5 (snapshots attached). They often go over 50. These solutions were written by us (to verify solvability), with near-perfect env apis knowledge; Sonnet typically takes more. Can share comparable Sonnet # turn stats/traj. soon offline. The AppWorld row for "Real States & Init" is X 👉 Each AppWorld task comes with its own initial state that is carefully & uniquely programmed (involving often 100+ LOCs) to ensure the task is hard to cheat, challenging, and realistic. We've thought quite carefully about designing each task's initial state, tailoring it to its unique requirements. Tab. 1, 8, 9 (snapshots attached). 👉 IMO, ensuring these good properties is what one needs. Sourcing data from real software can help with realism, but by itself, it is not enough, and there are other ways of achieving the desirable properties, as we did in AppWorld.

Junxian He 的头像

Junxian He10 个月前

Hi Harsh, thanks for the clarification! For the "# Turns" sorry we missed the specific number in the paper, are the # turns and # tool calls the same in AppWorld? Shall we refer to Table 2 in App World that is 42.5 turns? For the "Real States & Init" column yeah we marked it as X because we noticed the initial state was programmed, but we agree it is a bit confusing there and we'll make necessary revision soon about this information.

Harsh Trivedi 的头像

Harsh Trivedi10 个月前

Thank you, @junxian_he! Greatly appreciate your openness to revise that AppWorld information! 🙏 >> Re: Average # Turns That 42.5 number refers to the average tool calls in solutions that we wrote. It shows tasks are indeed quite long-horizon & complex. But this number isn't head-to-head comparable with the average turns in solutions that Claude wrote. Claude typically makes more tool calls than us, but then one turn can have multiple (parallel) tool calls. So I'm not sure which way the comparable number will go, and by how much. Note that we do have many task difficulty estimators (Figure 9), but none based on the Claude-based metric you have. I'm doing Claude and other model runs soon, and we will also release trajectories. I'll share the comparable stat from them with you then; it'd be wonderful if you could update the table in your next version with that. >> Re: we noticed the initial state was programmed Yes—the initialization function is indeed a program we manually wrote for each task. But this is quite similar to what you also do with your state-initialization scripts: you initialize your real apps with task-specific data, whereas we initialize our high-fidelity simulated apps. Concretely, we first generate a large-scale, realistic data simulating the digital activity of 100+ people over 2 years, resulting in ~350K rows & 100+ tables. Then, for each task, we manually write initialization functions (setup: 3.1) that adapt this base data as per task-specific requirements. Each task has its own unique requirements, not only to make it sensible/well-defined, but also hard to cheat, and challenging. Having just reread your paper again, I do really think it fits your description of realistic state inits. Sharing some snapshots from the talks I gave earlier and a link to the slides for more context (has speaker notes for details).

Junxian He 的头像

Junxian He10 个月前

Performance of different models. Claude-4.5-Sonnet leads the race with 38.6% pass@1 accuracy, GPT-5 and Claude-4-Sonnet are following with similar performance. Toolathlon reveals a clear gap between open-source and proprietary models. DeepSeek-V3.2 is the best-performing open-source model with 20% pass@1 at a remarkably low cost, spending $5 in total

Junxian He 的头像

Junxian He10 个月前

An example task

Junxian He 的头像

Junxian He10 个月前

Each task in Toolathlon is based on real-world environments with dedicated state init (such as a prefilled email inbox and real-world financial google sheets), and is verifiable to ensure reliable eval. Most tasks require interleaving multiple applications to complete. The tasks take Claude-4-Sonnet 26.8 turns on average.

Junxian He 的头像

Junxian He10 个月前

What fails the SOTA LLMs? We first analyze the tool call errors, some models just hallucinate non-existing tool names, for example Gemini2.5

Junxian He 的头像

Junxian He10 个月前

Long context is a clear challenge for language agents. When tool returns are overlong, the success rates are generally lower than the tasks without overlong tool returns.

Junxian He 的头像

Junxian He10 个月前

The cost-effectiveness of models is also very important. Claude is generally more expensive, whereas GPT-5 offers better cost efficiency among proprietary models. Among open-source models, DeepSeekv3.2 is the most affordable while achieving the highest performance.

Junxian He 的头像

Junxian He10 个月前

Finally, check out our eval toolkit! We support parallel, containerized evaluation and running full evaluation of Claude 4.5 Sonnet takes 70 minutes. Our website ( also allows to explore all the tasks and trajectories!

Junxian He 的头像

Junxian He10 个月前

This project is co-led by @lockonlvange Wenshuo Zhao, Jian Zhao, @AndrewZeng17 @HaozeWu7 , and a wonderful collaboration from all the collaborators from HKUST, @OpenHandsDev , CMU, and Duke: Xiaochen Wang, Rui Ge, @SArt_Maxine @yuzhenh17 @WeiLiu99 @junteng88716710 @SuZhaochen0110 Yiyang Guo, @FaZhou_998, Lueyang Zhang, Juan Michelini, @xingyaow_ @xiangyue96 @syz0x1 @gneubig !

Mykhailo Sorochuk 的头像

Mykhailo Sorochuk10 个月前

Interesting to see how real-world scenarios expose the gaps in SOTA LLMs. Tool interleaving sounds tough for them.

Allison Zhan 的头像

Allison Zhan10 个月前

nice work！

Min Chon Chi 的头像

Min Chon Chi10 个月前

600+ tools sounds comprehensive for evaluating language agent capabilities.

相关视频

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 次观看 • 4 个月前

Excited to present the LLM-Council skill. Initial idea by Karpathy. I just packaged it as a skill. You can easily spin up a council of LLMs or agents via Fireworks. Watch how the new GLM-5 model "deliberates" on other LLMs' thoughts on the big question, "Can LLMs reason?" Things worth paying attention to: New open models like GLM-5 have surprisingly improved on complex reasoning and long-running agentic tasks. The AskUserQuestion tool in Claude Code came in handy to select the council and chairperson. As Andrej Karpathy puts it, it's a really interesting way to get different perspectives from LLMs, which can lead to better decision-making on whatever task you are working on. You can use it for other agentic coding use cases, like evaluation, tool building, designing, and research.

Excited to present the LLM-Council skill. Initial idea by Karpathy. I just packaged it as a skill. You can easily spin up a council of LLMs or agents via Fireworks. Watch how the new GLM-5 model "deliberates" on other LLMs' thoughts on the big question, "Can LLMs reason?" Things worth paying attention to: New open models like GLM-5 have surprisingly improved on complex reasoning and long-running agentic tasks. The AskUserQuestion tool in Claude Code came in handy to select the council and chairperson. As Andrej Karpathy puts it, it's a really interesting way to get different perspectives from LLMs, which can lead to better decision-making on whatever task you are working on. You can use it for other agentic coding use cases, like evaluation, tool building, designing, and research.

elvis

39,593 次观看 • 7 个月前

Today we are open-sourcing boundary bench, a new paper, benchmark, and GitHub repo that enterprises can use to find out the REAL performance of their agents. Boundary-Bench was developed by researchers from Accomplish and NYU, where we tested 12 frontier agents across roughly 10,000 runs, with realistic enterprise policies, simulating environments with EDR, SASE, and DLP security tools enforcing those policies. We did this because generic leaderboard scores are being generated under conditions no security team would ever allow, which means orgs are making deployment and risk decisions based on numbers that don't hold up. The results are surprising >>

Today we are open-sourcing boundary bench, a new paper, benchmark, and GitHub repo that enterprises can use to find out the REAL performance of their agents. Boundary-Bench was developed by researchers from Accomplish and NYU, where we tested 12 frontier agents across roughly 10,000 runs, with realistic enterprise policies, simulating environments with EDR, SASE, and DLP security tools enforcing those policies. We did this because generic leaderboard scores are being generated under conditions no security team would ever allow, which means orgs are making deployment and risk decisions based on numbers that don't hold up. The results are surprising >>

Or Hiltch

75,573 次观看 • 1 个月前

🌍Are LLMs aware of cultural and legal safety in today’s geo-diverse world? 🚀Introducing SafeWorld, our #NeurIPS2024 paper and benchmark assessing LLMs’ understanding of geo-diverse safety, based on cultural norms and policies across 50 countries and 493 regions/races. ⚖️We also propose a multi-dimensional framework for evaluating contextual appropriateness, accuracy, and comprehensiveness, revealing major gaps in current LLMs. 🧨To address this, we train SafeWorldLM using DPO, achieving SOTA performance and a 20% higher global human evaluator rating in helpfulness and harmfulness over competing models, including GPT-4o. 🔗Paper: 💻 GitHub: 🫶🏻This is a joint leading effort with Da Yin. Also many thanks to the amazing team Kung-Hsiang Steeve Huang Kai-Wei Chang, and Violet Peng for their hard work. Check out more details and results we conclude from our paper in the thread below. 🧵

🌍Are LLMs aware of cultural and legal safety in today’s geo-diverse world? 🚀Introducing SafeWorld, our #NeurIPS2024 paper and benchmark assessing LLMs’ understanding of geo-diverse safety, based on cultural norms and policies across 50 countries and 493 regions/races. ⚖️We also propose a multi-dimensional framework for evaluating contextual appropriateness, accuracy, and comprehensiveness, revealing major gaps in current LLMs. 🧨To address this, we train SafeWorldLM using DPO, achieving SOTA performance and a 20% higher global human evaluator rating in helpfulness and harmfulness over competing models, including GPT-4o. 🔗Paper: 💻 GitHub: 🫶🏻This is a joint leading effort with Da Yin. Also many thanks to the amazing team Kung-Hsiang Steeve Huang Kai-Wei Chang, and Violet Peng for their hard work. Check out more details and results we conclude from our paper in the thread below. 🧵

Haoyi Qiu

16,344 次观看 • 1 年前

🚀 Introducing AgentCPM-Explore: The First Open-Source 4B-Agent Model to Conquer GAIA & Complex Real-World Tasks! 🤗 Hugging Face: 🔗 GitHub: ✨ Key Highlights: ✅ SOTA Agentic Performance: Sets a new benchmark for 4B-scale agent models—outperforming all peers, surpassing 8B models, and rivaling select 30B+ and closed-source LLMs. 🧠 Deep Research Capability: Excels at long-horizon reasoning, supports 100+ turns of autonomous interaction with multi-source cross-validation, human-like self-correction, and dynamic tool use + strategy adaptation—just like a real researcher! 🔓 Full-Stack Open Source: We’re open-sourcing the entire end-to-end agent stack—not just the model! Empower your own innovations with - AgentRL: Asynchronous reinforcement learning framework - AgentDock: Secure, extensible tool sandbox - AgentToLeaP: An one-click evaluation platform for agent tool-learning capabilitie - Full training data pipeline & reproducible workflows #AgentCPM #OpenSourceAI #AgenticAI #AI #GAIA #LLM #OpenBMB #AIAgents #HuggingFace

🚀 Introducing AgentCPM-Explore: The First Open-Source 4B-Agent Model to Conquer GAIA & Complex Real-World Tasks! 🤗 Hugging Face: 🔗 GitHub: ✨ Key Highlights: ✅ SOTA Agentic Performance: Sets a new benchmark for 4B-scale agent models—outperforming all peers, surpassing 8B models, and rivaling select 30B+ and closed-source LLMs. 🧠 Deep Research Capability: Excels at long-horizon reasoning, supports 100+ turns of autonomous interaction with multi-source cross-validation, human-like self-correction, and dynamic tool use + strategy adaptation—just like a real researcher! 🔓 Full-Stack Open Source: We’re open-sourcing the entire end-to-end agent stack—not just the model! Empower your own innovations with - AgentRL: Asynchronous reinforcement learning framework - AgentDock: Secure, extensible tool sandbox - AgentToLeaP: An one-click evaluation platform for agent tool-learning capabilitie - Full training data pipeline & reproducible workflows #AgentCPM #OpenSourceAI #AgenticAI #AI #GAIA #LLM #OpenBMB #AIAgents #HuggingFace

OpenBMB

13,996 次观看 • 8 个月前

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

Bowen Wang

93,200 次观看 • 1 年前

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

Jing Yu Koh

124,894 次观看 • 2 年前

LARP: Language-Agent Role Play for Open-World Games paper page: Language agents have shown impressive problem-solving skills within defined settings and brief timelines. Yet, with the ever-evolving complexities of open-world simulations, there's a pressing need for agents that can flexibly adapt to complex environments and consistently maintain a long-term memory to ensure coherent actions. To bridge the gap between language agents and open-world games, we introduce Language Agent for Role-Playing (LARP), which includes a cognitive architecture that encompasses memory processing and a decision-making assistant, an environment interaction module with a feedback-driven learnable action space, and a postprocessing method that promotes the alignment of various personalities. The LARP framework refines interactions between users and agents, predefined with unique backgrounds and personalities, ultimately enhancing the gaming experience in open-world contexts. Furthermore, it highlights the diverse uses of language models in a range of areas such as entertainment, education, and various simulation scenarios.

LARP: Language-Agent Role Play for Open-World Games paper page: Language agents have shown impressive problem-solving skills within defined settings and brief timelines. Yet, with the ever-evolving complexities of open-world simulations, there's a pressing need for agents that can flexibly adapt to complex environments and consistently maintain a long-term memory to ensure coherent actions. To bridge the gap between language agents and open-world games, we introduce Language Agent for Role-Playing (LARP), which includes a cognitive architecture that encompasses memory processing and a decision-making assistant, an environment interaction module with a feedback-driven learnable action space, and a postprocessing method that promotes the alignment of various personalities. The LARP framework refines interactions between users and agents, predefined with unique backgrounds and personalities, ultimately enhancing the gaming experience in open-world contexts. Furthermore, it highlights the diverse uses of language models in a range of areas such as entertainment, education, and various simulation scenarios.

AK

143,974 次观看 • 2 年前

Tencent announces AppAgent Multimodal Agents as Smartphone Users paper page: Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.

Tencent announces AppAgent Multimodal Agents as Smartphone Users paper page: Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.

AK

343,919 次观看 • 2 年前

The MiniMax M2 model is mind-blowing! It's open-source. It outperforms Gemini 2.5, Claude 4.1, and Qwen3 across coding and tool-use benchmarks. Right now, it's one of the world's top 5 models in intelligence! And here is the best part: Claude is one of the best models you can use today, and MiniMax M2 costs only 8% of that! It's smaller, faster, and cheaper. Extremely efficient at using tokens. Minimax M2's biggest strength: High agentic capabilities. The model can plan and execute complex multi-tool workflows. It's reliable and very robust at executing long-horizon tool chains. In summary: • Low latency • Very cheap • Excels at agentic tasks • Open-source The model currently powers the MiniMax Agent and is available for a free global trial. You can access MiniMax M2's API here: To access the agent: And here is the MiniMax website: Thanks to the MiniMax team for showing me the ropes and partnering with me on this post.

The MiniMax M2 model is mind-blowing! It's open-source. It outperforms Gemini 2.5, Claude 4.1, and Qwen3 across coding and tool-use benchmarks. Right now, it's one of the world's top 5 models in intelligence! And here is the best part: Claude is one of the best models you can use today, and MiniMax M2 costs only 8% of that! It's smaller, faster, and cheaper. Extremely efficient at using tokens. Minimax M2's biggest strength: High agentic capabilities. The model can plan and execute complex multi-tool workflows. It's reliable and very robust at executing long-horizon tool chains. In summary: • Low latency • Very cheap • Excels at agentic tasks • Open-source The model currently powers the MiniMax Agent and is available for a free global trial. You can access MiniMax M2's API here: To access the agent: And here is the MiniMax website: Thanks to the MiniMax team for showing me the ropes and partnering with me on this post.

Santiago

91,240 次观看 • 10 个月前

Excited to share our new project 🦉OWL - an open-source alternative to Manus AI with 6K+ stars and climbing… 🦉OWL: Everyone’s talking about Manus AI (an agent that can research, browse, code, and automate tasks.) But there’s a catch: → All the above agents cost $20 to $200 a month → Manus AI is available via an invite code only OWL by CAMEL-AI is an opensource project that’s breaking records: ⭐ 6K+ GitHub stars in 2 days 🏆 Ranks #1 on the GAIA Benchmark among open-source projects with a 58.18 avg score! 📌 Specialized Multi-Agent Collaboration ↳ AI user agents break down complex tasks ↳ Assistants agents create detailed execution strategies and interact with tools and tool agents ↳ Tools / Tool agents interface with external services and APIs 📌 Real-World Capabilities ↳ Autonomous research, coding, and web browsing ↳ Runs locally on your machine for privacy ↳ Supports top LLMs (easily integrated with GPT-4O, Claude 3.7, Gemini, Mistral, DeepSeek , Qwen, Ollama, Groq more) ↳ Automates complex workflows, from research to execution I 📌 Simple Setup ↳ Fast installs via conda or uv ↳ Docker support for easy deployment ↳ Straightforward config with TOML files No restrictions. No invites. No paywalls. Just powerful multi-agent collaboration. AI autonomy shouldn’t be locked behind paywalls.

Excited to share our new project 🦉OWL - an open-source alternative to Manus AI with 6K+ stars and climbing… 🦉OWL: Everyone’s talking about Manus AI (an agent that can research, browse, code, and automate tasks.) But there’s a catch: → All the above agents cost $20 to $200 a month → Manus AI is available via an invite code only OWL by CAMEL-AI is an opensource project that’s breaking records: ⭐ 6K+ GitHub stars in 2 days 🏆 Ranks #1 on the GAIA Benchmark among open-source projects with a 58.18 avg score! 📌 Specialized Multi-Agent Collaboration ↳ AI user agents break down complex tasks ↳ Assistants agents create detailed execution strategies and interact with tools and tool agents ↳ Tools / Tool agents interface with external services and APIs 📌 Real-World Capabilities ↳ Autonomous research, coding, and web browsing ↳ Runs locally on your machine for privacy ↳ Supports top LLMs (easily integrated with GPT-4O, Claude 3.7, Gemini, Mistral, DeepSeek , Qwen, Ollama, Groq more) ↳ Automates complex workflows, from research to execution I 📌 Simple Setup ↳ Fast installs via conda or uv ↳ Docker support for easy deployment ↳ Straightforward config with TOML files No restrictions. No invites. No paywalls. Just powerful multi-agent collaboration. AI autonomy shouldn’t be locked behind paywalls.

Guohao Li 🐫

223,313 次观看 • 1 年前

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Deyao Zhu

362,795 次观看 • 2 个月前

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more 📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use. 🏠 Homepage: 📄 Paper: 💻 Code: 🤗 Dataset: 🧵 [1/8]

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more 📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use. 🏠 Homepage: 📄 Paper: 💻 Code: 🤗 Dataset: 🧵 [1/8]

XLANG NLP Lab

169,416 次观看 • 2 个月前

I’ve built an order execution tool specifically for IB users. The open-source code is available on GitHub, but since most traders probably prefer something simple and ready to use, I also made an executable (.exe) version and uploaded it to Google Drive. This order execution tool can automatically retrieve your account balance and calculate position size based on your specified risk %. You can choose to set the stop loss at the LOD (low of the day) or enter it manually. The tool automatically places stop orders according to your stop level, one stop or 3 stops Why three stop orders? You can refer to Jeff Sun, CFTe Jeff Sun’s article for the reasoning behind this risk management approach. It also comes with customizable hotkeys for quick and efficient order placement. In the demo video, I used a virtual keyboard to show you how it works in real time. Lastly, this tool is completely free to use. If you find it helpful, you’re welcome to buy me a coffee — your support is what keeps me motivated to keep improving and adding new features. I’ve recorded a demo video showing how to set it up and use it — it’s already posted on X for reference. Github: Googledrive:

I’ve built an order execution tool specifically for IB users. The open-source code is available on GitHub, but since most traders probably prefer something simple and ready to use, I also made an executable (.exe) version and uploaded it to Google Drive. This order execution tool can automatically retrieve your account balance and calculate position size based on your specified risk %. You can choose to set the stop loss at the LOD (low of the day) or enter it manually. The tool automatically places stop orders according to your stop level, one stop or 3 stops Why three stop orders? You can refer to Jeff Sun, CFTe Jeff Sun’s article for the reasoning behind this risk management approach. It also comes with customizable hotkeys for quick and efficient order placement. In the demo video, I used a virtual keyboard to show you how it works in real time. Lastly, this tool is completely free to use. If you find it helpful, you’re welcome to buy me a coffee — your support is what keeps me motivated to keep improving and adding new features. I’ve recorded a demo video showing how to set it up and use it — it’s already posted on X for reference. Github: Googledrive:

Will Hu

29,715 次观看 • 10 个月前

Arena reached a $100M annual revenue run rate just 8 months after launching our evaluation product. We started as a research project at UC Berkeley with a simple mission: measure AI progress through real-world use. As AI shifts from chatbots to agents taking on longer, higher-stakes work, the problem matters more than ever. Today, Arena measures real-world AI utility with a community of tens of millions. With Agent Arena, we’re evaluating long-running agents on complex, real-world tasks - how they use tools, adapt to feedback, recover from errors, and accomplish goals set by humans. We are excited to keep deepening our work in agentic evaluations. Here’s Anastasios Nikolas Angelopoulos on what this milestone means and where we go from here:

Arena reached a $100M annual revenue run rate just 8 months after launching our evaluation product. We started as a research project at UC Berkeley with a simple mission: measure AI progress through real-world use. As AI shifts from chatbots to agents taking on longer, higher-stakes work, the problem matters more than ever. Today, Arena measures real-world AI utility with a community of tens of millions. With Agent Arena, we’re evaluating long-running agents on complex, real-world tasks - how they use tools, adapt to feedback, recover from errors, and accomplish goals set by humans. We are excited to keep deepening our work in agentic evaluations. Here’s Anastasios Nikolas Angelopoulos on what this milestone means and where we go from here:

Arena.ai

211,962 次观看 • 2 个月前

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

📢 Announcing one of the most exciting works from us this year on scalable robot policy evaluation through real-to-sim transfer, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is orders of magnitude slower than the development of language agents. We propose a new framework where simulation performance strongly correlates with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing state-of-the-art correlation between simulation and reality for deformable object manipulation. It provides a scalable and reproducible evaluation engine for robot learning. 🌐

Yunzhu Li

39,960 次观看 • 10 个月前

🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖 Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶 Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds. SimWorld features 3 key designs: 1⃣Open-ended realistic world simulation - built on Unreal Engine 5, with accurate physical social dynamics - 100+ built-in environments (city, island, wilderness ...) - language-controllable procedural generation - text-to-3D asset generation 2⃣Native interface for LLM/VLM agents - Gym-like agent-environment interaction APIs - plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...) - rich multi-modal perception - open-vocabulary natural-language action outputs 3⃣Diverse physical and social reasoning scenarios - long-horizon embodied reasoning - multi-agent collaboration / competition - easily customizable for any reasoning tasks SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc. Project website + more details in the thread👇 ...1/

🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖 Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶 Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds. SimWorld features 3 key designs: 1⃣Open-ended realistic world simulation - built on Unreal Engine 5, with accurate physical social dynamics - 100+ built-in environments (city, island, wilderness ...) - language-controllable procedural generation - text-to-3D asset generation 2⃣Native interface for LLM/VLM agents - Gym-like agent-environment interaction APIs - plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...) - rich multi-modal perception - open-vocabulary natural-language action outputs 3⃣Diverse physical and social reasoning scenarios - long-horizon embodied reasoning - multi-agent collaboration / competition - easily customizable for any reasoning tasks SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc. Project website + more details in the thread👇 ...1/

Lianhui Qin

65,156 次观看 • 9 个月前

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 次观看 • 3 年前

The 5 biggest AI leaks are on GitHub. Free. This wasn't supposed to end up online. 1. instructkr/claw-code, 195k ⭐️ stars A clean rewrite of the leaked Claude Code CLI. Got 100k stars in one day. Nobody has ever done that on GitHub. 2. x1xhlol/system-prompts-and-models-of-ai-tools, 143k ⭐️ stars 28 coding tools, with the JSON tool schemas. Not just the prompt text, the actual list of what each agent can touch. 3. asgeirtj/system_prompts_leaks, 63k ⭐️ stars The real prompts behind Claude Opus 5, GPT-5.6, Gemini 3.5, Grok and Cursor. Claude's is 3,677 lines long. The Washington Post ran a story off this repo. 4. elder-plinius/CL4R1T4S, 47k ⭐️ stars Same thing, every big lab in one place. Updated when new models ship. 5. Piebald-AI/claude-code-system-prompts, 12.4k ⭐️ stars Claude Code's prompt pulled from compiled source. 27 tool descriptions, subagent prompts, security review logic. The best prompt engineering course ever made is 5 free repos written by people who never meant to publish them. Saved it. Most of it is publicly available, but few people know about it 👇

The 5 biggest AI leaks are on GitHub. Free. This wasn't supposed to end up online. 1. instructkr/claw-code, 195k ⭐️ stars A clean rewrite of the leaked Claude Code CLI. Got 100k stars in one day. Nobody has ever done that on GitHub. 2. x1xhlol/system-prompts-and-models-of-ai-tools, 143k ⭐️ stars 28 coding tools, with the JSON tool schemas. Not just the prompt text, the actual list of what each agent can touch. 3. asgeirtj/system_prompts_leaks, 63k ⭐️ stars The real prompts behind Claude Opus 5, GPT-5.6, Gemini 3.5, Grok and Cursor. Claude's is 3,677 lines long. The Washington Post ran a story off this repo. 4. elder-plinius/CL4R1T4S, 47k ⭐️ stars Same thing, every big lab in one place. Updated when new models ship. 5. Piebald-AI/claude-code-system-prompts, 12.4k ⭐️ stars Claude Code's prompt pulled from compiled source. 27 tool descriptions, subagent prompts, security review logic. The best prompt engineering course ever made is 5 free repos written by people who never meant to publish them. Saved it. Most of it is publicly available, but few people know about it 👇

unicode

12,437 次观看 • 23 天前

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Andrew Ng

170,541 次观看 • 1 年前