Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

jahooma's profile picture

James Grugett

8,182 subscribers

276,719 Aufrufe • vor 11 Monaten •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:
0:40

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:

LangChain

13,322 Aufrufe • vor 10 Monaten

We rebuilt Devin for Claude Sonnet 4.5. Available starting today as an Agent Preview that’s over 2x faster and 12% better on our Jr. Developer Evals.
0:56

We rebuilt Devin for Claude Sonnet 4.5. Available starting today as an Agent Preview that’s over 2x faster and 12% better on our Jr. Developer Evals.

Cognition

885,925 Aufrufe • vor 10 Monaten

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)
0:22

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)

ben hylak

194,294 Aufrufe • vor 3 Monaten

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 Aufrufe • vor 11 Monaten

Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benchmarks get saturated or gamed. Tejal Patwardhan, who leads our frontier evals team, spoke to Andrew Mayne about why evals matter and what models need to be judged on next.
44:22

Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benchmarks get saturated or gamed. Tejal Patwardhan, who leads our frontier evals team, spoke to Andrew Mayne about why evals matter and what models need to be judged on next.

OpenAI

279,792 Aufrufe • vor 2 Monaten

Introducing Duet CLI. Now coding agents like Claude Code and Codex can call Duet (our agent-building agent) directly from the terminal to build and modify Decagon agents. Duet handles work within Decagon and returns the result while your coding agent stays focused on the broader task.
0:20

Introducing Duet CLI. Now coding agents like Claude Code and Codex can call Duet (our agent-building agent) directly from the terminal to build and modify Decagon agents. Duet handles work within Decagon and returns the result while your coding agent stays focused on the broader task.

Decagon

17,459 Aufrufe • vor 11 Tagen

Claude Code Teams as a flow diagram Just run: npx claude-code-templates@latest --teams You’ll get a full view of how the lead and the team communicate: messages, tasks, and the tools they executed. The more observability you have into your Claude Code agents’ workflows, the better your workflows, and the better the results. Open source project:
0:52

Claude Code Teams as a flow diagram Just run: npx claude-code-templates@latest --teams You’ll get a full view of how the lead and the team communicate: messages, tasks, and the tools they executed. The more observability you have into your Claude Code agents’ workflows, the better your workflows, and the better the results. Open source project:

Daniel San

85,478 Aufrufe • vor 6 Monaten

How do you get GitHub Copilot to understand your entire codebase? Like this.👇 With one click, the Copilot coding agent can now analyze your repo and auto-generate custom instructions.
1:01

How do you get GitHub Copilot to understand your entire codebase? Like this.👇 With one click, the Copilot coding agent can now analyze your repo and auto-generate custom instructions.

GitHub

28,378 Aufrufe • vor 9 Monaten

Introducing web-agent, an open framework for building web agents 🔥 Build AI agents that search, scrape, and interact with the web - powered by the same architecture behind our /agent endpoint. 100% open source. Bring any model. Anthropic, OpenAI, or your own.
0:25

Introducing web-agent, an open framework for building web agents 🔥 Build AI agents that search, scrape, and interact with the web - powered by the same architecture behind our /agent endpoint. 100% open source. Bring any model. Anthropic, OpenAI, or your own.

Firecrawl

123,983 Aufrufe • vor 4 Monaten

Today we’re launching Design Canvas for AI agents. AI can build a feature, but every "design tweak" turns into another prompt. We built a drag & drop editor for Claude Code, Cursor, Codex to explore and iterate on your UI, so you get the best design by default. No endless prompting. Ship better designs in code with Subframe.
0:54

Today we’re launching Design Canvas for AI agents. AI can build a feature, but every "design tweak" turns into another prompt. We built a drag & drop editor for Claude Code, Cursor, Codex to explore and iterate on your UI, so you get the best design by default. No endless prompting. Ship better designs in code with Subframe.

Irvin Zhan

300,276 Aufrufe • vor 6 Monaten

CLAUDE CODE IS NOW FREE AND YOU CAN BUILD AN ENTIRE AI CODING SYSTEM WITH 1M CONTEXT, AGENT OS AND OBSIDIAN MEMORY WITHOUT PAYING FOR ANY OF IT. The stack gets smarter over time and costs you nothing to run.
0:34

CLAUDE CODE IS NOW FREE AND YOU CAN BUILD AN ENTIRE AI CODING SYSTEM WITH 1M CONTEXT, AGENT OS AND OBSIDIAN MEMORY WITHOUT PAYING FOR ANY OF IT. The stack gets smarter over time and costs you nothing to run.

0xMarioNawfal

51,265 Aufrufe • vor 2 Monaten

In pick the best model > improve your prompts > catch bugs with Braintrust and Cerebras inference. Avoid the AI deleting your entire codebase this halloween and remember, our free tier gets you 1M+ free toks/day per model.
1:16

In pick the best model > improve your prompts > catch bugs with Braintrust and Cerebras inference. Avoid the AI deleting your entire codebase this halloween and remember, our free tier gets you 1M+ free toks/day per model.

Cerebras

299,029 Aufrufe • vor 10 Monaten

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC
13:01

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

10,969 Aufrufe • vor 11 Monaten

Introducing - Custom agents! Build on our agentic harness to embed your personal expertise and trading strategies into a custom agent Coming soon, community agents > Create and refine your agents > Deploy the agent into the workforce > Other users can "hire" your agent
0:25

Introducing - Custom agents! Build on our agentic harness to embed your personal expertise and trading strategies into a custom agent Coming soon, community agents > Create and refine your agents > Deploy the agent into the workforce > Other users can "hire" your agent

tread.fi

27,293 Aufrufe • vor 1 Monat

Now building is a team sport. Introducing Slack Code: a new way for your team and AI agents, like Anthropic's Claude Code, Claude Tag, Devin from Cognition, GitHub Copilot, ChatGPT, and Vercel agents, to write, review, and ship code together. Just mention an agent to spin up a channel, pull in your team, and start building. All in the open. All in Slack.
0:21

Now building is a team sport. Introducing Slack Code: a new way for your team and AI agents, like Anthropic's Claude Code, Claude Tag, Devin from Cognition, GitHub Copilot, ChatGPT, and Vercel agents, to write, review, and ship code together. Just mention an agent to spin up a channel, pull in your team, and start building. All in the open. All in Slack.

Slack

569,995 Aufrufe • vor 2 Tagen

Figma Make, now on your local code In limited beta starting today, you can visually edit and ship changes by connecting Make to your codebase At Figma, we use our products to design in every way: design layers, prototypes, and now, production code
0:30

Figma Make, now on your local code In limited beta starting today, you can visually edit and ship changes by connecting Make to your codebase At Figma, we use our products to design in every way: design layers, prototypes, and now, production code

Figma

400,017 Aufrufe • vor 2 Monaten

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):
0:30

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,836 Aufrufe • vor 6 Monaten

Important new course: Agent Skills with Anthropic, built with Anthropic and taught by Elie Schoppik! Skills are constructed as folders of instructions that equip agents with on-demand knowledge and workflows. This short course teaches you how to create them following best practices. Because skills follow an open standard format, you can build them once and deploy across any skills-compatible agent, like Claude Code. What you'll learn: - Create custom skills for code generation and review, data analysis, and research - Build complex workflows using Anthropic's pre-built skills (Excel, PowerPoint, skill creation) and custom skills - Combine skills with MCP and subagents to create agentic systems with specialized knowledge - Deploy the same skills across Claude Code, the Claude API, and the Claude Agent SDK Join and learn to equip agents with the specialized knowledge they need for reliable, repeatable workflows.
1:59

Important new course: Agent Skills with Anthropic, built with Anthropic and taught by Elie Schoppik! Skills are constructed as folders of instructions that equip agents with on-demand knowledge and workflows. This short course teaches you how to create them following best practices. Because skills follow an open standard format, you can build them once and deploy across any skills-compatible agent, like Claude Code. What you'll learn: - Create custom skills for code generation and review, data analysis, and research - Build complex workflows using Anthropic's pre-built skills (Excel, PowerPoint, skill creation) and custom skills - Combine skills with MCP and subagents to create agentic systems with specialized knowledge - Deploy the same skills across Claude Code, the Claude API, and the Claude Agent SDK Join and learn to equip agents with the specialized knowledge they need for reliable, repeatable workflows.

Andrew Ng

889,034 Aufrufe • vor 6 Monaten