Загрузка видео...

Не удалось загрузить видео

На главную

Your team deserves more from your agent observability platform - a single, connected place for instrumentation, investigation, and measurement, enhanced with intelligence. In Braintrust, you can start with an open-ended prompt about agent behavior and carry the investigation into code, evals, automation, and monitoring. New tools like Patterns and...

43,782 просмотров • 1 месяц назад •via X (Twitter)

Комментарии: 17

Фото профиля Braintrust
Braintrust1 месяц назад

Patterns looks through your production traces and identifies recurring failures or opportunities for review. Choose a Slack channel for regular updates, review the findings, and mark each as either useful or noise. Patterns preserves findings over time, and feedback helps future investigations.

Фото профиля Braintrust
Braintrust1 месяц назад

Loop lets you interact with anything in Braintrust, including findings from Patterns, in natural language. Ask Loop to investigate across traces, logs, experiments, and datasets in a Braintrust project. Then create Loop automations that repeat open-ended investigations on a schedule. Compare behavior before and after a release, identify accounts adopting a new workflow, or produce a recurring quality review for a product team.

Фото профиля Braintrust
Braintrust1 месяц назад

For cost-effective, broad analysis of your traces, use Topics. Topics classifies traces into tasks, issues, sentiment, and custom facets, and turns findings into structured, SQL-queryable context.

Фото профиля Braintrust
Braintrust1 месяц назад

Once an issue has been identified, Debugger helps explain why it occurred in a representative trace and provides a diagnosis that can be tested with evals. Move from a symptom to the part of an agent trajectory that needs attention.

Фото профиля Braintrust
Braintrust1 месяц назад

All of this works together, with your preferred coding agent, across the same traces, datasets, evaluators, experiments, dashboards, and monitors in your Braintrust environment. Open a Braintrust project, enable Patterns, and review the findings with your team. Use Loop to dig in further, create an evaluator or dataset, and configure monitoring. Get started with the docs →

Фото профиля Ryan Hoover
Ryan Hoover1 месяц назад

I like how refreshingly unflashy this video is

Фото профиля Anna
Anna1 месяц назад

🧠🔥

Фото профиля Elijah Kurien
Elijah Kurien1 месяц назад

Very cool

Фото профиля Olmo Maldonado
Olmo Maldonado1 месяц назад

This is how we scale intelligence

Фото профиля ornella
ornella1 месяц назад

lfg

Фото профиля Andy Berman
Andy Berman1 месяц назад

🔥

Фото профиля Oleks
Oleks1 месяц назад

investigation is the real gap, most teams have traces and still cannot answer why the agent did that

Фото профиля catman
catman1 месяц назад

The context following a prompt from investigation into code and evals is the interesting part. What exactly gets carried over: evidence, hypotheses, or just a summary?

Фото профиля SOP | AI Agent Builder
SOP | AI Agent Builder1 месяц назад

Trace every tool call and intermediate decision, not just final output; otherwise root-cause analysis misses recovery failures.

Фото профиля Izzy Hurley
Izzy Hurley1 месяц назад

gimme the insights!!

Фото профиля Izzy Hurley
Izzy Hurley1 месяц назад

yay!!

Фото профиля Anra
Anra1 месяц назад

Connected observability is the right frame — with 20+ agents, the debugging unit is the trajectory, not the span. The platforms that stitch evals, traces and permissions into one surface will own the ops layer of the fleet era.

Похожие видео

New short course: Long-Term Agentic Memory with LangGraph. Learn to build an agent with long-term memory in this course developed in collaboration with taught by its Co-Founder and CEO, Harrison Chase! Personal assistance and productivity tasks have become important use cases for agents. An important feature of an AI assistant, such as a coding or calendar assistant, is its ability to keep improving over time from its experience. Agent memory is the key capability that enables this. To add memory to an agent, you must first figure out what to store and what to retrieve when it is time to use the information. Additionally, you’ll have to decide when to update the stored information. For example, you might update in each iteration loop of the agent or perform updates in the background, with a helper agent. In this course, you will learn a mental framework to build agents with long-term memory. You'll create a useful email assistant that can respond, ignore, and notify using writing, scheduling, and memory-management tools. You’ll develop your agent's memory by adding facts to its memory store, provide examples to learn the user's preferences, and optimize system prompts to evolve instructions based on previous responses. In detail, you’ll: - Learn how the three types of memory--semantic, episodic, and procedural–and the two update mechanisms–via hot path and in the background–apply to your agents. - Build an email agent with writing, scheduling, and availability tools, along with a router that triages incoming email and handles it accordingly by ignoring, responding, or notifying the user. - Add tools to your email agent that allow it to operate on semantic memory by learning facts about the user, storing them in a long-term memory store, and searching over them in future interactions. - Incorporate episodic memory, in the form of few-shot examples, in the triage step of your agents to help them learn and update user preferences. - Add procedural memory as system prompts, optimized with feedback to improve the instructions the agent follows. Learn how to approach memory in agents, and start building agents with long-term memory with LangGraph! Please sign up here:

Andrew Ng

132,273 просмотров • 1 год назад

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,763 просмотров • 1 год назад