Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Built something fun this week: behavior-judge Long-horizon agents are hard to evaluate, but observing the process agents take to get to outcomes can help `behavior-judge` helps eval long-horizon agents by compiling natural-language specs into deterministic checks for behaviors

maanav's profile picture

Maanav Khaitan

6,977 subscribers

14,387 Aufrufe • vor 13 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Learn how to build a Gemini 3 Deep Agents with this video from LangChain. These agents can reason through complex, long horizon tasks by: > Breaking goals into actionable steps > Delegating specific work for specialized models > Leveraging file systems and code execution
7:49

Learn how to build a Gemini 3 Deep Agents with this video from LangChain. These agents can reason through complex, long horizon tasks by: > Breaking goals into actionable steps > Delegating specific work for specialized models > Leveraging file systems and code execution

Google AI Developers

19,484 Aufrufe • vor 9 Monaten

asked claude to host a virtual stablecoin hackathon between agents using nothing but solana/helius APIs results weren't too awful for only taking 5 minutes in theory, you can get it to do this with hundreds of agents and judge recursively until something useful is built
1:11

asked claude to host a virtual stablecoin hackathon between agents using nothing but solana/helius APIs results weren't too awful for only taking 5 minutes in theory, you can get it to do this with hundreds of agents and judge recursively until something useful is built

mert

24,879 Aufrufe • vor 5 Monaten

How to build long-horizon AI agents: behavior specs, ontologies, process supervision - my conversation with Mitchell Troyanovsky, co-founder of Basis 01:09 Why Everyone at Basis Was Whispering to AI when Stephanie Palazzolo walked in 04:12 Accounting as "an Intelligence Over the Economy" 06:11 What Makes an Agent Truly Long-Horizon 08:24 Inside an Autonomous, Multi-Day Tax Return 10:19 Agents That Hand Off Like Senior Engineers 11:17 A Brief History of Agents: From ReAct to Today 12:33 Why LLMs Have No Long-Term Memory 14:13 Why AutoGPT Didn't Live Up to Its Promise 15:51 The Three Breakthroughs: Opus 3, o1, o3 17:07 Why Reasoning Models Unlocked Agents 18:23 "Let's Verify Step by Step": The Road Not Taken 20:32 Pushing Back on the METR Chart 22:09 Why Coding Agents Won First 25:14 Why Real-World Agents Are Harder 26:55 How Accountants Verify Non-Deterministic Work 29:18 You Can't Scale Tax Returns Like Math 33:16 100 Evals Pass - So What? 35:53 Right Answer, Wrong Process 36:37 Behavior Specs, Explained 39:58 How Specific Should Behaviors Be? 42:18 Context Is Runtime Training Data 44:21 Who Judges the Judge? 46:45 The Move 37 Objection 50:02 The Magic Box Mental Model 52:41 "Nothing Has Changed Since o3" 54:56 Open-Sourcing Behavior Specs with Ankur Goyal Braintrust 59:45 Ontologies: A World for Agents to Live In 01:04:20 Documentation as Codebase 01:06:33 Why the Founding Fathers Were Context Engineers 01:09:05 Onboarding 300 Brilliant Alien Employees 01:11:10 Self-Improving Agent Systems 01:12:50 The Context Mistake Agent Builders Make 01:14:29 RL on Behavior Adherence 01:17:01 Will the Bitter Lesson Swallow the Harness 01:18:46 "Technical Moats Are Not Real Moats" 01:21:03 Advice for AI Builders
1:22:51

How to build long-horizon AI agents: behavior specs, ontologies, process supervision - my conversation with Mitchell Troyanovsky, co-founder of Basis 01:09 Why Everyone at Basis Was Whispering to AI when Stephanie Palazzolo walked in 04:12 Accounting as "an Intelligence Over the Economy" 06:11 What Makes an Agent Truly Long-Horizon 08:24 Inside an Autonomous, Multi-Day Tax Return 10:19 Agents That Hand Off Like Senior Engineers 11:17 A Brief History of Agents: From ReAct to Today 12:33 Why LLMs Have No Long-Term Memory 14:13 Why AutoGPT Didn't Live Up to Its Promise 15:51 The Three Breakthroughs: Opus 3, o1, o3 17:07 Why Reasoning Models Unlocked Agents 18:23 "Let's Verify Step by Step": The Road Not Taken 20:32 Pushing Back on the METR Chart 22:09 Why Coding Agents Won First 25:14 Why Real-World Agents Are Harder 26:55 How Accountants Verify Non-Deterministic Work 29:18 You Can't Scale Tax Returns Like Math 33:16 100 Evals Pass - So What? 35:53 Right Answer, Wrong Process 36:37 Behavior Specs, Explained 39:58 How Specific Should Behaviors Be? 42:18 Context Is Runtime Training Data 44:21 Who Judges the Judge? 46:45 The Move 37 Objection 50:02 The Magic Box Mental Model 52:41 "Nothing Has Changed Since o3" 54:56 Open-Sourcing Behavior Specs with Ankur Goyal Braintrust 59:45 Ontologies: A World for Agents to Live In 01:04:20 Documentation as Codebase 01:06:33 Why the Founding Fathers Were Context Engineers 01:09:05 Onboarding 300 Brilliant Alien Employees 01:11:10 Self-Improving Agent Systems 01:12:50 The Context Mistake Agent Builders Make 01:14:29 RL on Behavior Adherence 01:17:01 Will the Bitter Lesson Swallow the Harness 01:18:46 "Technical Moats Are Not Real Moats" 01:21:03 Advice for AI Builders

Matt Turck

20,898 Aufrufe • vor 20 Tagen

If your agents are hitting roadblocks on long-running tasks, MPP can help unblock ambitious prompts by enabling agents to easily pay for services and get back to the task at hand. Hear more from Georgios Konstantopoulos at MPP IRL:
0:41

If your agents are hitting roadblocks on long-running tasks, MPP can help unblock ambitious prompts by enabling agents to easily pay for services and get back to the task at hand. Hear more from Georgios Konstantopoulos at MPP IRL:

Tempo

11,593 Aufrufe • vor 19 Tagen

🚀 New LangChain Academy Course: Building Reliable Agents 🚀 Shipping agents to production is hard. Traditional software is deterministic – when something breaks, you check the logs and fix the code. But agents rely on non-deterministic models. Add multi-step reasoning, tool use, and real user traffic, and building reliable agents becomes far more complex than traditional system design. The goal of this course is to teach you how to take an agent from first run to production-ready system through iterative cycles of improvement. You’ll learn how to do this with LangSmith, our agent engineering platform for observing, evaluating, and deploying agents. Enroll for free ➡️
2:02

🚀 New LangChain Academy Course: Building Reliable Agents 🚀 Shipping agents to production is hard. Traditional software is deterministic – when something breaks, you check the logs and fix the code. But agents rely on non-deterministic models. Add multi-step reasoning, tool use, and real user traffic, and building reliable agents becomes far more complex than traditional system design. The goal of this course is to teach you how to take an agent from first run to production-ready system through iterative cycles of improvement. You’ll learn how to do this with LangSmith, our agent engineering platform for observing, evaluating, and deploying agents. Enroll for free ➡️

LangChain

30,744 Aufrufe • vor 5 Monaten

i think coding agents need an overhaul... long horizon tasks are the norm now, but agents only show you what they are doing ~right now~ i built a "mission control" concept with: - progress log - human-friendly artifacts - roadmap - eta please steal this idea, i'm begging ya
2:15

i think coding agents need an overhaul... long horizon tasks are the norm now, but agents only show you what they are doing ~right now~ i built a "mission control" concept with: - progress log - human-friendly artifacts - roadmap - eta please steal this idea, i'm begging ya

Nathan Flurry 🔩

14,027 Aufrufe • vor 1 Monat

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:
1:53

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

Divya Saini

22,323 Aufrufe • vor 5 Monaten

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:
1:53

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

Sri Laasya Nutheti

18,524 Aufrufe • vor 5 Monaten

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail
3:02

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail

LangChain

33,192 Aufrufe • vor 1 Jahr

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.
3:35

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

Kleiner Perkins

17,255 Aufrufe • vor 2 Monaten

Temporal CEO Samar Birwadker says GitHub was designed for humans, but is now at the mercy of agents. "Agents are starting to get more and more long-lived, more asynchronous, and more mission-critical." "For example, if you pick coding agents alone, is something like GitHub was designed for humans collaborate and create [pull requests] right now." "Now a lot of code is being generated by these agents, and they're the ones creating these PRs essentially, and even code review process is being automated by other agents." "So now this entire system backend, which is being built for human consumption, is now at the mercy of these agents kind of using it for their own coordination and collaboration."
3:30

Temporal CEO Samar Birwadker says GitHub was designed for humans, but is now at the mercy of agents. "Agents are starting to get more and more long-lived, more asynchronous, and more mission-critical." "For example, if you pick coding agents alone, is something like GitHub was designed for humans collaborate and create [pull requests] right now." "Now a lot of code is being generated by these agents, and they're the ones creating these PRs essentially, and even code review process is being automated by other agents." "So now this entire system backend, which is being built for human consumption, is now at the mercy of these agents kind of using it for their own coordination and collaboration."

MTS

26,902 Aufrufe • vor 3 Monaten

Most AI agents are built to replace creatives, we built ours to stay by your side For the first time, what Agents learn is shared across your team. You direct, watch the process in real time, and edit it live Agents + MCP + Flows, just launched at Upscale Conf SF
0:38

Most AI agents are built to replace creatives, we built ours to stay by your side For the first time, what Agents learn is shared across your team. You direct, watch the process in real time, and edit it live Agents + MCP + Flows, just launched at Upscale Conf SF

Magnific

827,395 Aufrufe • vor 2 Monaten

As token costs keep climbing 📈, where else can we find efficiency gains? Partner Spencer Farrar dug into how Sail Research is answering this question with more efficient inference and long-horizon agents. We're proud to back their Series A.
1:51

As token costs keep climbing 📈, where else can we find efficiency gains? Partner Spencer Farrar dug into how Sail Research is answering this question with more efficient inference and long-horizon agents. We're proud to back their Series A.

Theory Ventures

14,032 Aufrufe • vor 1 Monat

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%
2:00

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Brendan (can/do)

145,326 Aufrufe • vor 7 Monaten

Finally spent enough local compute to process the partial recording of partial solar eclipse last week. This eclipse is too close to horizon, so you get all the clouds you (don't) want. But you can still pull things out! Full + specs:
0:26

Finally spent enough local compute to process the partial recording of partial solar eclipse last week. This eclipse is too close to horizon, so you get all the clouds you (don't) want. But you can still pull things out! Full + specs:

Aleksey Shipilëv

108,386 Aufrufe • vor 10 Tagen

Can we get robots to improve at long-horizon tasks without supervision? Our latest work tackles this problem by planning to practice! Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)
1:32

Can we get robots to improve at long-horizon tasks without supervision? Our latest work tackles this problem by planning to practice! Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)

Nishanth Kumar

18,722 Aufrufe • vor 2 Jahren

Dev spent the weekend building Liquidity Agents... The Hive's Liquidity Agents simplify the process of earning on your idle assets. You can get a preview by asking The Hive to see pools on your favorite coins. One-click deposits will be available to everyone later this week!
0:25

Dev spent the weekend building Liquidity Agents... The Hive's Liquidity Agents simplify the process of earning on your idle assets. You can get a preview by asking The Hive to see pools on your favorite coins. One-click deposits will be available to everyone later this week!

The Hive

44,414 Aufrufe • vor 1 Jahr

Introducing Giga's browser agent. Giga can now log in and completes tasks end-to-end, just like your human agents. It used to take months to get support APIs deployed for AI agents in the enterprise. Now these integrations can be built instantly.
1:16

Introducing Giga's browser agent. Giga can now log in and completes tasks end-to-end, just like your human agents. It used to take months to get support APIs deployed for AI agents in the enterprise. Now these integrations can be built instantly.

Esha

1,481,739 Aufrufe • vor 7 Monaten

📊 Learn how to observe & evaluate agents on LangChain Academy 📊 Testing applications is essential to the development lifecycle, but LLM systems are non-deterministic – you can’t always predict how they will behave. Add multi-turn interactions and tool-calling agents, and testing agents becomes even more complex than traditional software testing 😵‍💫 To address this challenge, LangSmith provides a comprehensive platform for agent engineering that helps teams use live production data for continuous testing and improvement 🚀 In our quickstart course, LangSmith Essentials, you’ll learn how to observe, evaluate, and deploy your agent in less than 30 minutes. Enroll for free ➡️
1:44

📊 Learn how to observe & evaluate agents on LangChain Academy 📊 Testing applications is essential to the development lifecycle, but LLM systems are non-deterministic – you can’t always predict how they will behave. Add multi-turn interactions and tool-calling agents, and testing agents becomes even more complex than traditional software testing 😵‍💫 To address this challenge, LangSmith provides a comprehensive platform for agent engineering that helps teams use live production data for continuous testing and improvement 🚀 In our quickstart course, LangSmith Essentials, you’ll learn how to observe, evaluate, and deploy your agent in less than 30 minutes. Enroll for free ➡️

LangChain

13,518 Aufrufe • vor 7 Monaten

1/4Introducing Qwen3-Coder-Next: Our latest open-weights model designed specifically to power the next generation of autonomous Coding Agents. Built on Qwen3-Next, this model is engineered to handle complex, long-horizon programming tasks with unprecedented efficiency. High-performance agentic intelligence is now in your hands.
2:01

1/4Introducing Qwen3-Coder-Next: Our latest open-weights model designed specifically to power the next generation of autonomous Coding Agents. Built on Qwen3-Next, this model is engineered to handle complex, long-horizon programming tasks with unprecedented efficiency. High-performance agentic intelligence is now in your hands.

Tongyi Lab

212,645 Aufrufe • vor 6 Monaten