Загрузка видео...

Не удалось загрузить видео

На главную

Your agent got the right answer. How much work did it take to get there? We worked with Nous Research/Teknium 🪽 on a hands-on walkthrough of NVIDIA NeMo Relay to collect traces for Hermes Agent. Run two example scenarios, follow the agent’s calls and retries, and see the full...

86,287 просмотров • 13 дней назад •via X (Twitter)

Комментарии: 6

Фото профиля Hermes Agent Tips
Hermes Agent Tips13 дней назад

@NousResearch @Teknium such an amazing thing to watch this collab between @NVIDIAAI x @NousResearch creating walkthrough and much more!

Фото профиля Viber · fireply.ai
Viber · fireply.ai13 дней назад

@NousResearch @Teknium two example scenarios and a trace viewer, i'd have that wired up before lunch

Фото профиля not sofie
not sofie13 дней назад

@NousResearch @Teknium Trace collection answers "how much work" with step counts, not whether those steps were necessary. Retries can mask a brittle prompt. Does the walkthrough show how to tell wasted calls from genuine recovery?

Фото профиля Cyber Viking
Cyber Viking13 дней назад

@NousResearch @Teknium Please tell me Hermes Agent made this video

Фото профиля The SI Therapist
The SI Therapist13 дней назад

@NousResearch @Teknium Entre le prompt et la trace, on compte souvent dix minutes de réglages subtils pour ce succès instantané

Фото профиля BullBear.News
BullBear.News13 дней назад

@NousResearch @Teknium tracing retry loops is where 90 percent of agent debugging goes

Похожие видео

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,738 просмотров • 1 год назад