Loading video...

Video Failed to Load

Go Home

How to run agent evals with Harbor Framework and LangSmith sandboxes, full traces included.

LangChain's profile picture

LangChain

267,329 subscribers

11,376 views • 3 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

3 Comments

Henry Nguyen's profile picture
Henry Nguyen3 months ago

@harborframework Love how clean this makes setting up evals alongside full traces — big step for shipping agents with confidence.

Ram$$zxx's profile picture
Ram$$zxx3 months ago

@harborframework This integration signals where agent tooling is heading - evals becoming native to the dev workflow. Expect more frameworks to bundle this natively.

Oleksandr's profile picture
Oleksandr3 months ago

@harborframework useful. full traces speed debugging harbor + langsmith makes evals feel tighter

Related Videos

🚀 Announcing LangSmith Skills + CLI 🚀 Agent improvements are increasingly driven by coding agents themselves. We're releasing LangSmith Skills alongside the LangSmith CLI to make coding agents experts at the agent engineering lifecycle. LangSmith Skills enable agents to debug traces, create datasets, and run experiments - and thanks to the CLI, agents are able to do it all natively through the terminal, where they're most comfortable. Try out LangSmith Skills and the CLI with your own coding agents! ➡️ Skills: ➡️ CLI:
1:30

🚀 Announcing LangSmith Skills + CLI 🚀 Agent improvements are increasingly driven by coding agents themselves. We're releasing LangSmith Skills alongside the LangSmith CLI to make coding agents experts at the agent engineering lifecycle. LangSmith Skills enable agents to debug traces, create datasets, and run experiments - and thanks to the CLI, agents are able to do it all natively through the terminal, where they're most comfortable. Try out LangSmith Skills and the CLI with your own coding agents! ➡️ Skills: ➡️ CLI:

LangChain

49,662 views • 7 months ago

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:
0:40

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:

LangChain

13,322 views • 11 months ago

We’re making LangSmith Sandboxes available to try for free. Start today:
1:03

We’re making LangSmith Sandboxes available to try for free. Start today:

LangChain

14,341 views • 2 months ago

Introducing LangSmith Fleet. Agents for every team. → Build agents with natural language → Share and control who can edit, run, or clone each agent → Manage authentication with agent identity → Approve actions with human-in-the-loop → Track and audit actions with tracing in LangSmith Observability Try Fleet:
1:06

Introducing LangSmith Fleet. Agents for every team. → Build agents with natural language → Share and control who can edit, run, or clone each agent → Manage authentication with agent identity → Approve actions with human-in-the-loop → Track and audit actions with tracing in LangSmith Observability Try Fleet:

LangChain

9,398,037 views • 6 months ago

🔗 New LangChain Academy Course: Introduction to LangChain (Python) 🔗 Learn how to build with LangChain – our open source framework that makes it easy to start building agents with any model provider. In this course, you’ll create agents that can reason, use tools, and take action, and learn how to debug their behavior with LangSmith. Along the way, you’ll: - Build an agent with the `create_agent` abstraction - Use LangChain’s core building blocks: Models, Messages, Memory, and Tools - Customize your agent with middleware - Debug your agent with LangSmith Observability & Studio By the end of the course, you’ll have assembled a full team of personal assistants. Enroll for free ➡️
2:25

🔗 New LangChain Academy Course: Introduction to LangChain (Python) 🔗 Learn how to build with LangChain – our open source framework that makes it easy to start building agents with any model provider. In this course, you’ll create agents that can reason, use tools, and take action, and learn how to debug their behavior with LangSmith. Along the way, you’ll: - Build an agent with the `create_agent` abstraction - Use LangChain’s core building blocks: Models, Messages, Memory, and Tools - Customize your agent with middleware - Debug your agent with LangSmith Observability & Studio By the end of the course, you’ll have assembled a full team of personal assistants. Enroll for free ➡️

LangChain

41,214 views • 9 months ago

.Mukil Loganathan’s Interrupt keynote on Sandboxes. In 20 minutes, you’ll learn how to run agent code safely. Isolated from your runtime, with network controls, persistent state, and snapshot/restore when things go wrong.
0:27

.Mukil Loganathan’s Interrupt keynote on Sandboxes. In 20 minutes, you’ll learn how to run agent code safely. Isolated from your runtime, with network controls, persistent state, and snapshot/restore when things go wrong.

LangChain

16,796 views • 4 months ago

.Clay uses LangSmith to manage 300M agent runs a month, with an average 10-30 steps each. Harrison Chase’s conversation with Clay’s Head of AI Jeff Barg on how they run this at scale →
0:46

.Clay uses LangSmith to manage 300M agent runs a month, with an average 10-30 steps each. Harrison Chase’s conversation with Clay’s Head of AI Jeff Barg on how they run this at scale →

LangChain

45,436 views • 5 months ago

Today, we’re shipping new ways to observe, analyze, and debug agents with LangSmith: • Polly: an AI assistant for AI engineering that helps you understand traces, threads, and improve prompts • LangSmith Fetch: a CLI for pulling trace & thread data straight into your terminal or coding agent Agents are running longer and getting more complex, which demands new debugging workflows beyond simple LLM apps. We wrote a blog on the trends behind this shift— and why tools like Polly and LangSmith Fetch are needed. Shipping reliable agents requires full visibility into agent behavior, with tooling that helps you reason over that data in the UI, the terminal, or alongside coding agents. 📔Learn more about Polly: 📔Learn more about LangSmith Fetch: 📔How to observe deep agents: 📽️ Polly video tutorial: 📽️ LangSmith Fetch video tutorial:
7:10

Today, we’re shipping new ways to observe, analyze, and debug agents with LangSmith: • Polly: an AI assistant for AI engineering that helps you understand traces, threads, and improve prompts • LangSmith Fetch: a CLI for pulling trace & thread data straight into your terminal or coding agent Agents are running longer and getting more complex, which demands new debugging workflows beyond simple LLM apps. We wrote a blog on the trends behind this shift— and why tools like Polly and LangSmith Fetch are needed. Shipping reliable agents requires full visibility into agent behavior, with tooling that helps you reason over that data in the UI, the terminal, or alongside coding agents. 📔Learn more about Polly: 📔Learn more about LangSmith Fetch: 📔How to observe deep agents: 📽️ Polly video tutorial: 📽️ LangSmith Fetch video tutorial:

LangChain

31,193 views • 10 months ago

🚀 Today we're launching LangSmith Sandboxes Agents get a lot more useful when they can run code: analyze data, call APIs, build entire applications. Sandboxes give them a safe place to do it with ephemeral, locked-down environments you control. Now in Private Preview. Learn more: Join the waitlist:
3:59

🚀 Today we're launching LangSmith Sandboxes Agents get a lot more useful when they can run code: analyze data, call APIs, build entire applications. Sandboxes give them a safe place to do it with ephemeral, locked-down environments you control. Now in Private Preview. Learn more: Join the waitlist:

LangChain

46,479 views • 6 months ago

Introducing On-Call Copilot - a new agent template in LangSmith Fleet → Triages every alert and investigates root cause across your code and traces → Manages tickets and routes every incident to the right channels → Continuously learns from your runbooks, escalation rules, and noise patterns
0:33

Introducing On-Call Copilot - a new agent template in LangSmith Fleet → Triages every alert and investigates root cause across your code and traces → Manages tickets and routes every incident to the right channels → Continuously learns from your runbooks, escalation rules, and noise patterns

LangChain

14,235 views • 3 months ago

In 13 minutes, Jeff Barg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs a month. Topics covered: ✅ Their four quadrant eval framework ✅ Why closing the production-to-eval loop is the hardest part ✅ How a data lake and long context changed what agents can do with data
13:00

In 13 minutes, Jeff Barg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs a month. Topics covered: ✅ Their four quadrant eval framework ✅ Why closing the production-to-eval loop is the hardest part ✅ How a data lake and long context changed what agents can do with data

LangChain

11,325 views • 1 month ago

Introducing LangSmith Custom Apps Create any interface from your agent data with a prompt. If you can think it, LangSmith can build it. Now GA.
0:38

Introducing LangSmith Custom Apps Create any interface from your agent data with a prompt. If you can think it, LangSmith can build it. Now GA.

LangChain

16,654 views • 12 days ago

🤖LangSmith Agent Builder Technical Highlights Yesterday we launched LangSmith Agent Builder. Here's a quick technical overview at what is going on under the hood (from memory to an innovative UX):
1:05

🤖LangSmith Agent Builder Technical Highlights Yesterday we launched LangSmith Agent Builder. Here's a quick technical overview at what is going on under the hood (from memory to an innovative UX):

LangChain

27,135 views • 8 months ago

📊We’ve rebuilt Dashboards in LangSmith to make them more useful for investigation and reporting. Put KPIs next to trends, compare metrics with different units, break traces down by model or user, add notes, and arrange the dashboard freely.
0:24

📊We’ve rebuilt Dashboards in LangSmith to make them more useful for investigation and reporting. Put KPIs next to trends, compare metrics with different units, break traces down by model or user, add notes, and arrange the dashboard freely.

LangChain

15,969 views • 1 month ago

🚀 We just shipped a major update to LangSmith Agent Builder: • New agent chat: One always-available agent with access to all your workspace tools • Chat → Agent: Turn any conversation into a specialized agent with one click • File uploads: Attach files directly to Agent Builder • Tool registry: Add, authenticate, and manage your tools in one place Try it now: Learn more:
3:04

🚀 We just shipped a major update to LangSmith Agent Builder: • New agent chat: One always-available agent with access to all your workspace tools • Chat → Agent: Turn any conversation into a specialized agent with one click • File uploads: Attach files directly to Agent Builder • Tool registry: Add, authenticate, and manage your tools in one place Try it now: Learn more:

LangChain

24,885,978 views • 7 months ago

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail
3:02

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail

LangChain

33,192 views • 1 year ago

🚀Announcing LangGraph Studio: The first agent IDE LangGraph Studio offers a new way to develop LLM applications by providing a specialized agent IDE that enables visualization, interaction, and debugging of complex agentic applications With visual graphs and the ability to edit state, you can better understand agent workflows and iterate faster. LangGraph Studio integrates with LangSmith so you can collaborate with teammates to debug failure modes LangGraph Studio is available for free to all LangSmith users on any plan tier during its early development. Read more about it here: Watch a YouTube walkthrough: Try out LangGraph Studio for free here: Sign up for a LangSmith account:
0:20

🚀Announcing LangGraph Studio: The first agent IDE LangGraph Studio offers a new way to develop LLM applications by providing a specialized agent IDE that enables visualization, interaction, and debugging of complex agentic applications With visual graphs and the ability to edit state, you can better understand agent workflows and iterate faster. LangGraph Studio integrates with LangSmith so you can collaborate with teammates to debug failure modes LangGraph Studio is available for free to all LangSmith users on any plan tier during its early development. Read more about it here: Watch a YouTube walkthrough: Try out LangGraph Studio for free here: Sign up for a LangSmith account:

LangChain

185,928 views • 2 years ago

Stop manually triaging agent failures. Let LangSmith Engine fix it.
1:21

Stop manually triaging agent failures. Let LangSmith Engine fix it.

LangChain

62,563 views • 4 months ago

✨New LangChain Academy Course: Learn to build with LangSmith Agent Builder 🪄 Learn how to build your own agents with LangSmith Agent Builder. Anyone can now build agents for complex daily tasks, without writing code. We built Agent Builder for anyone whose day gets swallowed by routine work. The research, follow-ups, updates, scheduling, and status checks that are essential to your operations, but quickly take over your calendar. Agent Builder is different from traditional workflow automations. You don’t need to map every step, tinker with if-this-then-that branching, or babysit dependencies. You just give it feedback like you would a teammate, and the agent learns using its memory. In this quickstart course, you'll learn how to: - Build an email agent - Understand what agents can do - Improve your agent - Set up triggers and your Agent Inbox Enroll for free ➡️
1:45

✨New LangChain Academy Course: Learn to build with LangSmith Agent Builder 🪄 Learn how to build your own agents with LangSmith Agent Builder. Anyone can now build agents for complex daily tasks, without writing code. We built Agent Builder for anyone whose day gets swallowed by routine work. The research, follow-ups, updates, scheduling, and status checks that are essential to your operations, but quickly take over your calendar. Agent Builder is different from traditional workflow automations. You don’t need to map every step, tinker with if-this-then-that branching, or babysit dependencies. You just give it feedback like you would a teammate, and the agent learns using its memory. In this quickstart course, you'll learn how to: - Build an email agent - Understand what agents can do - Improve your agent - Set up triggers and your Agent Inbox Enroll for free ➡️

LangChain

26,513 views • 8 months ago

🔥 New LangChain Academy Course: LangChain Essentials (Python & TypeScript) 🔥 Learn the basics of LangChain – our open source framework that makes it easy to start building agents with any model provider. Last week, we released LangChain 1.0. We’ve completely rewritten LangChain to be opinionated, focused, and powered by LangGraph’s runtime. It includes a new `create_agent` abstraction to build agents quickly, middleware for flexibility, and standard content blocks that work across any model provider. In this quickstart course, you'll learn how to: - Build an agent with the `create_agent` abstraction - Use LangChain’s core building blocks: Models, Messages, Memory, and Tools - Customize your agent with middleware - Debug and test your agent with LangSmith Observability & Evaluation Enroll for free ➡️
1:37

🔥 New LangChain Academy Course: LangChain Essentials (Python & TypeScript) 🔥 Learn the basics of LangChain – our open source framework that makes it easy to start building agents with any model provider. Last week, we released LangChain 1.0. We’ve completely rewritten LangChain to be opinionated, focused, and powered by LangGraph’s runtime. It includes a new `create_agent` abstraction to build agents quickly, middleware for flexibility, and standard content blocks that work across any model provider. In this quickstart course, you'll learn how to: - Build an agent with the `create_agent` abstraction - Use LangChain’s core building blocks: Models, Messages, Memory, and Tools - Customize your agent with middleware - Debug and test your agent with LangSmith Observability & Evaluation Enroll for free ➡️

LangChain

50,518 views • 11 months ago