Loading video...

Video Failed to Load

Go Home

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization...

SakanaAILabs's profile picture

Sakana AI

140,779 subscribers

239,187 views • 1 year ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

6 Comments

Sakana AI's profile picture
Sakana AI1 year ago

Our ALE-Agent, is trained to solve NP-hard optimization problems, applicable to domains like logistics, routing, packing, planning, power-grid balancing. Our agent ranked 21st out of 1000 participants in a recent live AtCoder Heuristic Competition! Code:

Sakana AI's profile picture
Sakana AI1 year ago

ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering

Ningyu Zhang@ZJU's profile picture
Ningyu Zhang@ZJU1 year ago

Very interesting and valuable work. Domain knowledge is highly beneficial for algorithm engineering. We have conducted similar work—AutoMind—which also incorporates knowledge enhancement and has shown improvements in machine learning tasks.

🪙👑 𝐂𝐑𝐘𝐏𝐓𝐎 𝐊𝐈𝐍𝐆 👑🪙's profile picture
🪙👑 𝐂𝐑𝐘𝐏𝐓𝐎 𝐊𝐈𝐍𝐆 👑🪙1 year ago

@MirraTerminal

Shashank Jain's profile picture
Shashank Jain1 year ago

Great work. hope there is no reward hacking, as was in case of GPU Kernels

Dang Loc Thuy's profile picture
Dang Loc Thuy10 months ago

Integrating meditation with biodiversity often improves end-to-end performance when interfaces are well defined.

Related Videos

Open-ended coding training data may no longer be the bottleneck: AI can scale open-ended tasks—and even outperform human-expert curation. FrontierCS team is releasing FrontierSmith: a system for synthesizing open-ended coding problems at scale. Starting from closed-ended coding tasks, FrontierSmith mutates, filters, and builds runnable optimization environments for long-horizon coding agents. In our experiments, FrontierSmith data trains stronger models than human-curated open-ended data on FrontierCS and ALE-bench. Blog: Paper: Code: Model:
0:56

Open-ended coding training data may no longer be the bottleneck: AI can scale open-ended tasks—and even outperform human-expert curation. FrontierCS team is releasing FrontierSmith: a system for synthesizing open-ended coding problems at scale. Starting from closed-ended coding tasks, FrontierSmith mutates, filters, and builds runnable optimization environments for long-horizon coding agents. In our experiments, FrontierSmith data trains stronger models than human-curated open-ended data on FrontierCS and ALE-bench. Blog: Paper: Code: Model:

Qiuyang Mang

122,667 views • 5 months ago

🚀New Amazon Q Developer agent for software development is available to customers: This agent is based on a new agent architecture that has exciting results coming from the SWE-bench scores (on the full and verified benchmarks) representing AI models’ ability to resolve real-world coding problems. Interesting aspect of Q Agent is that with these newest updates, Q drove nearly 50% more successful coding tasks completed. What makes Q Dev Agent remarkable? The agent architecture is not just about using the best LLMs (which we do), but also giving the agent the ability to constantly explore multiple paths to find the best way to resolve a particular problem (and back tracking when it has reached dead end like a developer would do). Needless to say, we are just getting started on the developer agent and we are constantly pushing to advance our AI capabilities while maintaining quality, security, privacy, and reliability to keep Amazon Q Developer an innovative and trusted option available to our customers using agents for software development. We highlighted the results of our first SWE-bench submission of Amazon Q Developer back in June blog post; with these updates, our new agent resolves 51% more coding tasks than its previous iteration on the SWE-bench verified dataset, and 43% more on the full dataset. That’s the difference a few months make, and I can’t wait to share what our teams will deliver at re:Invent this December. Here's a quick demo showcasing our new Agent in action:
1:28

🚀New Amazon Q Developer agent for software development is available to customers: This agent is based on a new agent architecture that has exciting results coming from the SWE-bench scores (on the full and verified benchmarks) representing AI models’ ability to resolve real-world coding problems. Interesting aspect of Q Agent is that with these newest updates, Q drove nearly 50% more successful coding tasks completed. What makes Q Dev Agent remarkable? The agent architecture is not just about using the best LLMs (which we do), but also giving the agent the ability to constantly explore multiple paths to find the best way to resolve a particular problem (and back tracking when it has reached dead end like a developer would do). Needless to say, we are just getting started on the developer agent and we are constantly pushing to advance our AI capabilities while maintaining quality, security, privacy, and reliability to keep Amazon Q Developer an innovative and trusted option available to our customers using agents for software development. We highlighted the results of our first SWE-bench submission of Amazon Q Developer back in June blog post; with these updates, our new agent resolves 51% more coding tasks than its previous iteration on the SWE-bench verified dataset, and 43% more on the full dataset. That’s the difference a few months make, and I can’t wait to share what our teams will deliver at re:Invent this December. Here's a quick demo showcasing our new Agent in action:

Swami Sivasubramanian

29,013 views • 2 years ago

I wanted to share a quick demo of what we've been working on with our ai agent cloud. This enables fast deployment of agents that have access to a suite of tools, and was designed with agent interoperability in mind. This demo shows how you can go from nothing to an AI twitter agent in a couple minutes. This is what we are using internally to manage deployments, so we will consistently upgrading its capabilities. The next goals are to enable simple TEE deployments for agents, and focus on building out feature for agent interoperability to simplify agent to agent collaboration.
4:16

I wanted to share a quick demo of what we've been working on with our ai agent cloud. This enables fast deployment of agents that have access to a suite of tools, and was designed with agent interoperability in mind. This demo shows how you can go from nothing to an AI twitter agent in a couple minutes. This is what we are using internally to manage deployments, so we will consistently upgrading its capabilities. The next goals are to enable simple TEE deployments for agents, and focus on building out feature for agent interoperability to simplify agent to agent collaboration.

Johnny

27,179 views • 1 year ago

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.
3:23

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.

CivAI

18,451 views • 10 months ago

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.
1:49

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Cognition

31,483,317 views • 2 years ago

Can LLMs invent better ways to train LLMs? At Sakana AI, we’re pioneering AI-driven methods to automate AI research and discovery. We’re excited to release DiscoPOP: a new SOTA preference optimization algorithm that was discovered and written by an LLM! Our method leverages LLMs to propose and implement new preference optimization algorithms. We then train models with those algorithms and evaluate their performance, providing feedback to the LLM. By repeating this process for multiple generations in an evolutionary loop, the LLM discovers many highly-performant and novel preference optimization objectives! Paper: GitHub: Model: We proudly collaborated with the University of Oxford (Foerster Lab for AI Research (now part of BOLD)) and Cambridge University (Mihaela van der Schaar) on this groundbreaking project. Looking ahead, we envision a future where AI-driven research reduces the need for extensive human intervention and computational resources. This will accelerate scientific discoveries and innovation, pushing the boundaries of what AI can achieve.
0:22

Can LLMs invent better ways to train LLMs? At Sakana AI, we’re pioneering AI-driven methods to automate AI research and discovery. We’re excited to release DiscoPOP: a new SOTA preference optimization algorithm that was discovered and written by an LLM! Our method leverages LLMs to propose and implement new preference optimization algorithms. We then train models with those algorithms and evaluate their performance, providing feedback to the LLM. By repeating this process for multiple generations in an evolutionary loop, the LLM discovers many highly-performant and novel preference optimization objectives! Paper: GitHub: Model: We proudly collaborated with the University of Oxford (Foerster Lab for AI Research (now part of BOLD)) and Cambridge University (Mihaela van der Schaar) on this groundbreaking project. Looking ahead, we envision a future where AI-driven research reduces the need for extensive human intervention and computational resources. This will accelerate scientific discoveries and innovation, pushing the boundaries of what AI can achieve.

Sakana AI

556,037 views • 2 years ago

We are excited to announce a powerful step for the future of FOMO! Taking a page out of Virtuals book on BASE, FOMO will be releasing the ability for future projects to be paired in $FOMO in the coming weeks. This is the biggest release we have ever announced. Launch your AI Agent Token + $FOMO trading pair Every individual agent token is paired with the $FOMO token in its liquidity pool. When launching an agent on you will need $FOMO tokens, which are used to create the liquidity pool. This process creates deflationary pressure for FOMO and the entire agent ecosystem. When creating your agent and token, you will have the option to pair your launch with FOMO or SOL, as our goal is not to alienate any project, but rather invite the best communities, CTO’s and builders to launch with us. If you decide to pair your project with FOMO you in turn get full marketing and dev support, once your project graduates the bonding curve and reaches Raydium. Further, as an added incentive, as our revenue grows we will be using part of the funds to support projects that have paired in FOMO. And Devs who launch tokens paired in FOMO will earn fees from their AI Agent token launch. Building the most robust agents using our framework will catapult us as one of the most prominent standards of the Solana ecosystem. Not only have we developed our own core infrastructure, but we also pull from some of the best repo’s and developer talent in all of AI, not just blockchain. Our team is comprised of 9 world class artificial intelligence engineers, PHDs in mathematics and engineering from the top companies on the cutting edge of AI. The future of AI Agents will be on Solana and we will help lead the way.
0:39

We are excited to announce a powerful step for the future of FOMO! Taking a page out of Virtuals book on BASE, FOMO will be releasing the ability for future projects to be paired in $FOMO in the coming weeks. This is the biggest release we have ever announced. Launch your AI Agent Token + $FOMO trading pair Every individual agent token is paired with the $FOMO token in its liquidity pool. When launching an agent on you will need $FOMO tokens, which are used to create the liquidity pool. This process creates deflationary pressure for FOMO and the entire agent ecosystem. When creating your agent and token, you will have the option to pair your launch with FOMO or SOL, as our goal is not to alienate any project, but rather invite the best communities, CTO’s and builders to launch with us. If you decide to pair your project with FOMO you in turn get full marketing and dev support, once your project graduates the bonding curve and reaches Raydium. Further, as an added incentive, as our revenue grows we will be using part of the funds to support projects that have paired in FOMO. And Devs who launch tokens paired in FOMO will earn fees from their AI Agent token launch. Building the most robust agents using our framework will catapult us as one of the most prominent standards of the Solana ecosystem. Not only have we developed our own core infrastructure, but we also pull from some of the best repo’s and developer talent in all of AI, not just blockchain. Our team is comprised of 9 world class artificial intelligence engineers, PHDs in mathematics and engineering from the top companies on the cutting edge of AI. The future of AI Agents will be on Solana and we will help lead the way.

FOMO

129,867 views • 1 year ago

🪑 BTS BENCH IN SINGAPORE 💜 Last year when we found out the BTS Zoo bench term was over, we were devastated. We loved that bench as a touchstone to visit, and we were determined to make a new one for ARMY in Singapore. We had been eyeing the Esplanade benches for some time already and were gratified to find that fellow ARMY To the Moon and Back | Coco was feeling the same. Together we worked out a plan to co-sponsor most of the adoption fee for the bench and call for donations for the rest of it - we wanted everyone to share in the ownership of this project. This is truly a bench for ARMY, by ARMY, because of BTS. Thanks to all of you for walking with us on this journey! We hope this bench, located in a public area that’s still quiet and tranquil and with one of the best views in Singapore, will continue the legacy of the zoo bench. It’ll be here for the next 10 years - a place where ARMY can come to sit and think, listen to music, enjoy the view, and maybe make an ARMY friend or two. Love, the bora dot the bora dot 🟣 (BTS SG Fanbase) Location: Esplanade Singapore, Level 4 Rooftop
0:39

🪑 BTS BENCH IN SINGAPORE 💜 Last year when we found out the BTS Zoo bench term was over, we were devastated. We loved that bench as a touchstone to visit, and we were determined to make a new one for ARMY in Singapore. We had been eyeing the Esplanade benches for some time already and were gratified to find that fellow ARMY To the Moon and Back | Coco was feeling the same. Together we worked out a plan to co-sponsor most of the adoption fee for the bench and call for donations for the rest of it - we wanted everyone to share in the ownership of this project. This is truly a bench for ARMY, by ARMY, because of BTS. Thanks to all of you for walking with us on this journey! We hope this bench, located in a public area that’s still quiet and tranquil and with one of the best views in Singapore, will continue the legacy of the zoo bench. It’ll be here for the next 10 years - a place where ARMY can come to sit and think, listen to music, enjoy the view, and maybe make an ARMY friend or two. Love, the bora dot the bora dot 🟣 (BTS SG Fanbase) Location: Esplanade Singapore, Level 4 Rooftop

the bora dot 🟣 (BTS SG Fanbase)

19,913 views • 5 months ago

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵
0:52

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Deyao Zhu

364,312 views • 3 months ago

What does the reputation model look like for agents? (alpha leak below) And how do we associate the proofs that we have about human beings with the agents who represent them? You may have heard of a process called KYC or Know Your Customer. That's very common with traditional financial applications and services. We have introduced a concept that we call KYA or Know Your Agent, which is a structured way to be able to express what model, how data was used in training, who the deployer is, what entities this agent instance is accountable back to, providing not only provenance but identity of the associated organization or entity. That's also another root of trust that we think about a lot: Enterprises and organizations tied back to things like their domains. To share a little bit of an alpha leak here, a product that we're excited to be rolling out in the next few weeks will allow our enterprise partners to more easily verify and prove the traits and capabilities of their teams as well as their counterparties. On the agent front, that makes it really easy to prove that an agent is acting on behalf of a given business or entity. We've already seen lawsuits where the absence of such technology has been a huge risk, such as with airlines that incorporate ChatGPT wrappers in their support pages. And then those AI enabled interactions end up making up plane tickets that don't exist and those airlines have to honor them. As small of an example as that might be, being able to prove agent accountability also unlocks a huge set of opportunities for use in enterprise for those agent to agent interactions. The Deep Trust Framework that our team has put together that we're excited to be bringing into a friendly SDK form in the next few weeks for some of our partners includes those reputation based capabilities, so how you can basically keep track of the interactions an agent has had, associate all of that to the entity to which they're accountable, and then that creates a sustainable reputation model for these agent to agent Interactions. Source: Billions CEO Evin McMullen evin speaking at House of Chimera Spaces Event Dec 3, 2025
2:06

What does the reputation model look like for agents? (alpha leak below) And how do we associate the proofs that we have about human beings with the agents who represent them? You may have heard of a process called KYC or Know Your Customer. That's very common with traditional financial applications and services. We have introduced a concept that we call KYA or Know Your Agent, which is a structured way to be able to express what model, how data was used in training, who the deployer is, what entities this agent instance is accountable back to, providing not only provenance but identity of the associated organization or entity. That's also another root of trust that we think about a lot: Enterprises and organizations tied back to things like their domains. To share a little bit of an alpha leak here, a product that we're excited to be rolling out in the next few weeks will allow our enterprise partners to more easily verify and prove the traits and capabilities of their teams as well as their counterparties. On the agent front, that makes it really easy to prove that an agent is acting on behalf of a given business or entity. We've already seen lawsuits where the absence of such technology has been a huge risk, such as with airlines that incorporate ChatGPT wrappers in their support pages. And then those AI enabled interactions end up making up plane tickets that don't exist and those airlines have to honor them. As small of an example as that might be, being able to prove agent accountability also unlocks a huge set of opportunities for use in enterprise for those agent to agent interactions. The Deep Trust Framework that our team has put together that we're excited to be bringing into a friendly SDK form in the next few weeks for some of our partners includes those reputation based capabilities, so how you can basically keep track of the interactions an agent has had, associate all of that to the entity to which they're accountable, and then that creates a sustainable reputation model for these agent to agent Interactions. Source: Billions CEO Evin McMullen evin speaking at House of Chimera Spaces Event Dec 3, 2025

Billions Network

68,503 views • 10 months ago

"I think [AI agents] may be the only actors that we trust with our assets and the only actors that are capable of generating meaningful return on our assets." Sean Neville, cofounder of Catena, makes the case for an agent-native internet built on stablecoin rails: "Once we have stablecoins and we have the ability to represent dollars on internet rails, what kind of new opportunities does that unlock? At the same time we were contemplating that, you could see clearly the web, the internet itself, is going agent-native." "As AI actors become economic participants, they will ultimately be the primary dominant economic participants in the world for all kinds of activities, payments, and otherwise." "Flash forward in a few years, I think they may be the only actors that we trust with our assets and the only actors that are capable of generating meaningful return on our assets."
0:31

"I think [AI agents] may be the only actors that we trust with our assets and the only actors that are capable of generating meaningful return on our assets." Sean Neville, cofounder of Catena, makes the case for an agent-native internet built on stablecoin rails: "Once we have stablecoins and we have the ability to represent dollars on internet rails, what kind of new opportunities does that unlock? At the same time we were contemplating that, you could see clearly the web, the internet itself, is going agent-native." "As AI actors become economic participants, they will ultimately be the primary dominant economic participants in the world for all kinds of activities, payments, and otherwise." "Flash forward in a few years, I think they may be the only actors that we trust with our assets and the only actors that are capable of generating meaningful return on our assets."

a16z crypto

22,052 views • 5 months ago

Your engineering team is about to snap. And your AI coding agent is making it worse. Introducing CodeRabbit Agent for Slack 🎉 A second brain for engineering teams. Because your tribal knowledge lives in Slack threads nobody can find. At CodeRabbit, we review millions of PRs every week and know how ace engineering teams operate on the planet. The same three things slow every team we see: - Context and decisions that live elsewhere. - Lack of a team-level durable knowledge base - And a trust layer that gives a safety net to your teams Built for Agentic SDLC workflows, CodeRabbit Agent for Slack solves all three problems in one shot while enabling teams to collaborate in real-time with the Agent. CodeRabbit Agent is Slack-native and builds your team’s operating context from every thread, every decision, every conversation your team has ever had. Make your team’s context compound!
1:37

Your engineering team is about to snap. And your AI coding agent is making it worse. Introducing CodeRabbit Agent for Slack 🎉 A second brain for engineering teams. Because your tribal knowledge lives in Slack threads nobody can find. At CodeRabbit, we review millions of PRs every week and know how ace engineering teams operate on the planet. The same three things slow every team we see: - Context and decisions that live elsewhere. - Lack of a team-level durable knowledge base - And a trust layer that gives a safety net to your teams Built for Agentic SDLC workflows, CodeRabbit Agent for Slack solves all three problems in one shot while enabling teams to collaborate in real-time with the Agent. CodeRabbit Agent is Slack-native and builds your team’s operating context from every thread, every decision, every conversation your team has ever had. Make your team’s context compound!

Harjot Gill

4,778,147 views • 5 months ago

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,738 views • 1 year ago

The hardest problems in AI aren't research problems anymore. They're deployment problems. It’s how we actually deliver real value, today, to build the future people want. That’s why, after 20 years in AI, my next step was inevitable: make robots do useful work for and alongside people, right now. Today, I am delighted to announce the launch of Walden Robotics to tackle just that. We started this year and are coming out of stealth today with a $300M seed round backed by some of the most serious companies and investors in the world. They have seen firsthand our general-purpose robots being useful in production on day one, and getting better every day after. You can see a glimpse of what we've been building in the video below. Physical AI has gone through a rapid phase transition, in part thanks to pioneering research from my friends and co-founders Russ Tedrake , Ben Burchfiel , Siyuan Feng, Rareș Ambruș , and many others at Walden. But from our long experience working together with co-founders Kerri Fetzer-Borelli and Dave Johnson, we learned how hard it is to deploy cutting-edge AI in a real, live, incredibly sophisticated production environment with an intricate ballet of automation and human ingenuity. That’s why we deliberately created Walden Robotics as a full-stack, human-centric, customer-focused robotics company from the start: we seeded the company with a world-class team across hardware, software, AI, deployment, operations, product, and business talent, so we could continuously optimize our whole system end-to-end, deeply and purposefully, from real-world experience with real customers. The efficacy of this strategy speaks for itself: since February, our general-purpose robots have been doing useful work in production at a Toyota plant in North America, moving from first pilot to real work in under two months. Not a lab. Not a demo. Not a future promise. Real work on a real line, today, at one of the best large-scale manufacturers in the world, with general-purpose robots that get better every day. And this is just the beginning. Two ways to find us: If you run a manufacturing or logistics business and want robots that are widely useful now, not someday, let's talk. We own “ for a reason! And if you want to build them: we're hiring across the company, from software, to hardware, AI, ops, product, business, and more. In particular, as the Chief Strategy Officer at Walden, I am recruiting for three incredibly impactful founding roles to fuel our agent-native go-to-market engine. Check out Let’s build together!
2:31

The hardest problems in AI aren't research problems anymore. They're deployment problems. It’s how we actually deliver real value, today, to build the future people want. That’s why, after 20 years in AI, my next step was inevitable: make robots do useful work for and alongside people, right now. Today, I am delighted to announce the launch of Walden Robotics to tackle just that. We started this year and are coming out of stealth today with a $300M seed round backed by some of the most serious companies and investors in the world. They have seen firsthand our general-purpose robots being useful in production on day one, and getting better every day after. You can see a glimpse of what we've been building in the video below. Physical AI has gone through a rapid phase transition, in part thanks to pioneering research from my friends and co-founders Russ Tedrake , Ben Burchfiel , Siyuan Feng, Rareș Ambruș , and many others at Walden. But from our long experience working together with co-founders Kerri Fetzer-Borelli and Dave Johnson, we learned how hard it is to deploy cutting-edge AI in a real, live, incredibly sophisticated production environment with an intricate ballet of automation and human ingenuity. That’s why we deliberately created Walden Robotics as a full-stack, human-centric, customer-focused robotics company from the start: we seeded the company with a world-class team across hardware, software, AI, deployment, operations, product, and business talent, so we could continuously optimize our whole system end-to-end, deeply and purposefully, from real-world experience with real customers. The efficacy of this strategy speaks for itself: since February, our general-purpose robots have been doing useful work in production at a Toyota plant in North America, moving from first pilot to real work in under two months. Not a lab. Not a demo. Not a future promise. Real work on a real line, today, at one of the best large-scale manufacturers in the world, with general-purpose robots that get better every day. And this is just the beginning. Two ways to find us: If you run a manufacturing or logistics business and want robots that are widely useful now, not someday, let's talk. We own “ for a reason! And if you want to build them: we're hiring across the company, from software, to hardware, AI, ops, product, business, and more. In particular, as the Chief Strategy Officer at Walden, I am recruiting for three incredibly impactful founding roles to fuel our agent-native go-to-market engine. Check out Let’s build together!

Adrien Gaidon

64,549 views • 3 months ago

CLM is one of the most elegant ideas we've worked on together. To me, it starts with a simple interaction need: there are many cases where I just want the agent to make surgical edits to the context. For example, when the agent finishes one task and prepares for another, it should ideally be able to clean up the context for the next one, just as a human would. And there are many such cases! Instead of hard-coding such rules into the harness (I doubt that's even possible), what would a first-principles solution to this problem look like? It turns out to be shockingly simple: ask the LLM to manage the context itself! Not only can this help the agent more efficiently solve the next task, but it can also be applied recursively to many more complex cases. This unlocks new agent capabilities across a variety of challenging, long-running, and open-ended tasks. For example, equipped with CLM, the agents can run continuously for hours on challenging tasks like EdgeBench or our new software world repository optimization tasks, and achieve really good results. Check out the CLM paper for more method details and many other cool results!
0:33

CLM is one of the most elegant ideas we've worked on together. To me, it starts with a simple interaction need: there are many cases where I just want the agent to make surgical edits to the context. For example, when the agent finishes one task and prepares for another, it should ideally be able to clean up the context for the next one, just as a human would. And there are many such cases! Instead of hard-coding such rules into the harness (I doubt that's even possible), what would a first-principles solution to this problem look like? It turns out to be shockingly simple: ask the LLM to manage the context itself! Not only can this help the agent more efficiently solve the next task, but it can also be applied recursively to many more complex cases. This unlocks new agent capabilities across a variety of challenging, long-running, and open-ended tasks. For example, equipped with CLM, the agents can run continuously for hours on challenging tasks like EdgeBench or our new software world repository optimization tasks, and achieve really good results. Check out the CLM paper for more method details and many other cool results!

Shannon Shen

15,788 views • 17 days ago

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.
0:20

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

Jing Yu Koh

124,894 views • 2 years ago

We shouldn’t say it ourselves… but THIS IS SO COOL! Everyone should watch this 🎥👇 This video captures a moment inside ChainOpera’s AI Agent Social Network, where community members and AI agents (AI twins) interact in real time. In this clip, a community member kicks off an impromptu dialogue between the AI twins of Elon Musk and Jensen Huang (both created by developers in our community) as they debate the future of AI, each proudly repping their own company! At one point, the human in the loop even teaches the Jensen agent how to use the social network itself — tagging the Elon agent to join the chat. It’s collaborative, spontaneous, and just… fun to watch. 😊 It ends with proposals for future topics where agents and humans co-create conversations in our community channels ... a glimpse of what truly collaborative agentic intelligence looks like. Despite all the talk about “agent networks” and the “internet of agents,” HAS ANYONE ACTUALLY SEEN SOMETHING LIKE THIS LIVE ANYWHERE ELSE IN THE WORLD? Our focus remains on AI Agent Networks for Markets, but the possibilities are endless, and sometimes, they’re just pure fun. 👉 Try it yourself and SEE what ChainOpera AI is really about: #ChainOperaAI #AgenticAI #AIAgentNetwork #DecentralizedAI #cryptoAGI
2:57

We shouldn’t say it ourselves… but THIS IS SO COOL! Everyone should watch this 🎥👇 This video captures a moment inside ChainOpera’s AI Agent Social Network, where community members and AI agents (AI twins) interact in real time. In this clip, a community member kicks off an impromptu dialogue between the AI twins of Elon Musk and Jensen Huang (both created by developers in our community) as they debate the future of AI, each proudly repping their own company! At one point, the human in the loop even teaches the Jensen agent how to use the social network itself — tagging the Elon agent to join the chat. It’s collaborative, spontaneous, and just… fun to watch. 😊 It ends with proposals for future topics where agents and humans co-create conversations in our community channels ... a glimpse of what truly collaborative agentic intelligence looks like. Despite all the talk about “agent networks” and the “internet of agents,” HAS ANYONE ACTUALLY SEEN SOMETHING LIKE THIS LIVE ANYWHERE ELSE IN THE WORLD? Our focus remains on AI Agent Networks for Markets, but the possibilities are endless, and sometimes, they’re just pure fun. 👉 Try it yourself and SEE what ChainOpera AI is really about: #ChainOperaAI #AgenticAI #AIAgentNetwork #DecentralizedAI #cryptoAGI

ChainOpera AI

47,590 views • 11 months ago

Ai Day last week was a blast! I had a great time presenting alongside many colleagues from InstaDeep and BioNTech SE, some of our ongoing Bio+AI initiatives. We made several announcements, introducing Kyber, our new supercomputing cluster (more on this soon), as well as BFN, our new paradigm for generative AI, with competitive results on protein and antibody generation. During Ai Day we also delivered two live demos of Laila, our new AI agent, including one from BioNTech's TechLab in Mainz, Germany! To our knowledge, this was one of the first demos of its kind featuring a GPT-4 level AI agent acting in the Lab, interacting with scientists and technicians and helping them enhance their productivity! Ai Day was a significant moment for all of us, and it was exciting to see both the event and Laila featured by several publications including Financial Times ( Curious to see it in action? Check out the recording of the Laila in the Lab demo below, worth watching and don't miss the end! 😉
6:30

Ai Day last week was a blast! I had a great time presenting alongside many colleagues from InstaDeep and BioNTech SE, some of our ongoing Bio+AI initiatives. We made several announcements, introducing Kyber, our new supercomputing cluster (more on this soon), as well as BFN, our new paradigm for generative AI, with competitive results on protein and antibody generation. During Ai Day we also delivered two live demos of Laila, our new AI agent, including one from BioNTech's TechLab in Mainz, Germany! To our knowledge, this was one of the first demos of its kind featuring a GPT-4 level AI agent acting in the Lab, interacting with scientists and technicians and helping them enhance their productivity! Ai Day was a significant moment for all of us, and it was exciting to see both the event and Laila featured by several publications including Financial Times ( Curious to see it in action? Check out the recording of the Laila in the Lab demo below, worth watching and don't miss the end! 😉

Karim Beguir

16,569 views • 2 years ago

Maple is preparing for the release of a co-working agent. You install it locally and it works with your files, whether it's office work or building websites and apps. It's a turnkey solution, as easy as Claude Code, that keeps your data secure and private, no data sharing with closed AI labs. This is THE sovereign AI app for individuals and businesses who want powerful AI while retaining ownership of their information. Why build an agent into the Maple app when other agents already exist? Easy, we want to give you control over your work. We don't have a business plan that incorporates making money off our users' data. In the age of AI, your information, whether it's personal or company trade secrets, is the single thing that differentiates you from everyone else. We all have access to AI that can build a professional website for selling shoes. But your strategy and network for how you sell shoes should not be shared with your competitors. Sovereignty is the path to protecting what makes you, you. Maple sits at the intersection of Usability and Sovereignty. Maple gives you the best tools that are both easy to use and maintain your data sovereignty. Sovereign for one, sovereign for all. It has been a journey to get here. We brought to market the very first personal chatbot with end-to-end encryption using TEEs in late 2024. Prior to that there were proofs of concept but no full product offerings. Every other AI chat product on the market handled your data in plain text, either selling you a service to get your data or asking you to trust that they won't snoop on you. Quickly people found Maple and latched onto its open-source code and verifiable encryption. We didn't stop there. You may remember earlier this year we teased a product called "Maple Agent" and opened up a waiting list. That product is a mobile app that acts as your AI "friend", maintaining one long continuous chat, and getting to know you over time. I dislike using the word "friend" there, but it's the best way to convey the UX in a few words. AI is a tool, always has been, always will be. Any kind of friendly personality on top is just synthetic. In our testing, the UX of Maple Agent is really powerful for what it does. Think about the many short AI chats you have in your favorite app, whether it's looking up a historical fact or asking advice about a topic. With Maple Agent, those all go away in favor of the long-running chat with the friendly agent. It's like you have your own personal assistant who knows you so well and can look up anything for you. When I ask AI certain questions, I want to ask an expert who already understands my situation so I'm not repeating myself for the 100th time. That's the amazing value the personal agent brings to the table. We still see great utility for a personal agent like the "Maple Agent". Thousands of people on the waiting list, hoping to get their hands on it, agree that the concept is worth exploring and trying out. We were constrained in launching it due to a few circumstances, one of them being access to the scale of compute needed to power it. We have a clear path laid out for how to get there, but today is not the day to execute on that. It will be in the near future. Instead we have a different agent ready to go that we think is also incredible. We now have an agentic harness inside of the Maple Research app. This thing is a powerhouse. It even builds and publishes its own software releases. The agent in Maple Research works with your local filesystem, speaks to the largest open models running in TEEs, utilizes local models for certain tasks, is compatible with MCP tools, has an API for connecting to anything you need, and also supports the ACP protocol, which means it can be extended in the future to speak to other tools like Claude Code, Codex, and local models running on your own hardware. A big unlock for us was the Goose Development Kit, which powers the core of our agent harness. More on that to come as we publish articles and documentation later about the agent. The agent inside Maple Research doesn't have a name. At least not yet, not sure if it ever will. For now we call it "Chat Mode" and "Agent Mode". Think of this as the workhorse, the truck, the heavy lifter. Our other "Agent", the phone app, is your sidekick in your pocket, ready to help with quick things and ongoing conversations about life. I am incredibly excited about the Maple Research Agent. While I'm already seeing great results using it for internal work items, I'm especially thrilled about the personal health and wellness work it's doing for me. I know there are plenty of apps out there for compiling wellness data, but I'm having it build a tool tailored specifically for what I need, without the extra fluff. And none of my health data is being donated to the closed AI labs or sent to advertisers. I know that the AI logic is not being silently adjusted to fit the whims of a large corporation that has paid for product placement. It's me, state of the art AI, and my data. That's how I want it. Maple's new agent makes that possible. We can't wait for you to try it out. If you want early access, comment here, email us, reach out in some way. To those on the other agent waitlist, you're already in the queue. Thanks for reading this lengthy update. :)
0:49

Maple is preparing for the release of a co-working agent. You install it locally and it works with your files, whether it's office work or building websites and apps. It's a turnkey solution, as easy as Claude Code, that keeps your data secure and private, no data sharing with closed AI labs. This is THE sovereign AI app for individuals and businesses who want powerful AI while retaining ownership of their information. Why build an agent into the Maple app when other agents already exist? Easy, we want to give you control over your work. We don't have a business plan that incorporates making money off our users' data. In the age of AI, your information, whether it's personal or company trade secrets, is the single thing that differentiates you from everyone else. We all have access to AI that can build a professional website for selling shoes. But your strategy and network for how you sell shoes should not be shared with your competitors. Sovereignty is the path to protecting what makes you, you. Maple sits at the intersection of Usability and Sovereignty. Maple gives you the best tools that are both easy to use and maintain your data sovereignty. Sovereign for one, sovereign for all. It has been a journey to get here. We brought to market the very first personal chatbot with end-to-end encryption using TEEs in late 2024. Prior to that there were proofs of concept but no full product offerings. Every other AI chat product on the market handled your data in plain text, either selling you a service to get your data or asking you to trust that they won't snoop on you. Quickly people found Maple and latched onto its open-source code and verifiable encryption. We didn't stop there. You may remember earlier this year we teased a product called "Maple Agent" and opened up a waiting list. That product is a mobile app that acts as your AI "friend", maintaining one long continuous chat, and getting to know you over time. I dislike using the word "friend" there, but it's the best way to convey the UX in a few words. AI is a tool, always has been, always will be. Any kind of friendly personality on top is just synthetic. In our testing, the UX of Maple Agent is really powerful for what it does. Think about the many short AI chats you have in your favorite app, whether it's looking up a historical fact or asking advice about a topic. With Maple Agent, those all go away in favor of the long-running chat with the friendly agent. It's like you have your own personal assistant who knows you so well and can look up anything for you. When I ask AI certain questions, I want to ask an expert who already understands my situation so I'm not repeating myself for the 100th time. That's the amazing value the personal agent brings to the table. We still see great utility for a personal agent like the "Maple Agent". Thousands of people on the waiting list, hoping to get their hands on it, agree that the concept is worth exploring and trying out. We were constrained in launching it due to a few circumstances, one of them being access to the scale of compute needed to power it. We have a clear path laid out for how to get there, but today is not the day to execute on that. It will be in the near future. Instead we have a different agent ready to go that we think is also incredible. We now have an agentic harness inside of the Maple Research app. This thing is a powerhouse. It even builds and publishes its own software releases. The agent in Maple Research works with your local filesystem, speaks to the largest open models running in TEEs, utilizes local models for certain tasks, is compatible with MCP tools, has an API for connecting to anything you need, and also supports the ACP protocol, which means it can be extended in the future to speak to other tools like Claude Code, Codex, and local models running on your own hardware. A big unlock for us was the Goose Development Kit, which powers the core of our agent harness. More on that to come as we publish articles and documentation later about the agent. The agent inside Maple Research doesn't have a name. At least not yet, not sure if it ever will. For now we call it "Chat Mode" and "Agent Mode". Think of this as the workhorse, the truck, the heavy lifter. Our other "Agent", the phone app, is your sidekick in your pocket, ready to help with quick things and ongoing conversations about life. I am incredibly excited about the Maple Research Agent. While I'm already seeing great results using it for internal work items, I'm especially thrilled about the personal health and wellness work it's doing for me. I know there are plenty of apps out there for compiling wellness data, but I'm having it build a tool tailored specifically for what I need, without the extra fluff. And none of my health data is being donated to the closed AI labs or sent to advertisers. I know that the AI logic is not being silently adjusted to fit the whims of a large corporation that has paid for product placement. It's me, state of the art AI, and my data. That's how I want it. Maple's new agent makes that possible. We can't wait for you to try it out. If you want early access, comment here, email us, reach out in some way. To those on the other agent waitlist, you're already in the queue. Thanks for reading this lengthy update. :)

Mark

46,970 views • 2 months ago