Загрузка видео...

Не удалось загрузить видео

На главную

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

jahooma's profile picture

James Grugett

8,182 subscribers

276,719 просмотров • 11 месяцев назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:
0:40

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:

LangChain

13,322 просмотров • 9 месяцев назад

We rebuilt Devin for Claude Sonnet 4.5. Available starting today as an Agent Preview that’s over 2x faster and 12% better on our Jr. Developer Evals.
0:56

We rebuilt Devin for Claude Sonnet 4.5. Available starting today as an Agent Preview that’s over 2x faster and 12% better on our Jr. Developer Evals.

Cognition

885,925 просмотров • 10 месяцев назад

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)
0:22

we built the first sane way to debug your agent locally. you can see your traces. codex/claude code can too. this lets them write evals and test your agents automatically. best part: it's completely free and open source. install with 1 line. (github below)

ben hylak

194,294 просмотров • 2 месяцев назад

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 просмотров • 10 месяцев назад

Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benchmarks get saturated or gamed. Tejal Patwardhan, who leads our frontier evals team, spoke to Andrew Mayne about why evals matter and what models need to be judged on next.
44:22

Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benchmarks get saturated or gamed. Tejal Patwardhan, who leads our frontier evals team, spoke to Andrew Mayne about why evals matter and what models need to be judged on next.

OpenAI

278,506 просмотров • 1 месяц назад

How do you get GitHub Copilot to understand your entire codebase? Like this.👇 With one click, the Copilot coding agent can now analyze your repo and auto-generate custom instructions.
1:01

How do you get GitHub Copilot to understand your entire codebase? Like this.👇 With one click, the Copilot coding agent can now analyze your repo and auto-generate custom instructions.

GitHub

28,378 просмотров • 9 месяцев назад

Claude Code Teams as a flow diagram Just run: npx claude-code-templates@latest --teams You’ll get a full view of how the lead and the team communicate: messages, tasks, and the tools they executed. The more observability you have into your Claude Code agents’ workflows, the better your workflows, and the better the results. Open source project:
0:52

Claude Code Teams as a flow diagram Just run: npx claude-code-templates@latest --teams You’ll get a full view of how the lead and the team communicate: messages, tasks, and the tools they executed. The more observability you have into your Claude Code agents’ workflows, the better your workflows, and the better the results. Open source project:

Daniel San

85,478 просмотров • 6 месяцев назад

Introducing web-agent, an open framework for building web agents 🔥 Build AI agents that search, scrape, and interact with the web - powered by the same architecture behind our /agent endpoint. 100% open source. Bring any model. Anthropic, OpenAI, or your own.
0:25

Introducing web-agent, an open framework for building web agents 🔥 Build AI agents that search, scrape, and interact with the web - powered by the same architecture behind our /agent endpoint. 100% open source. Bring any model. Anthropic, OpenAI, or your own.

Firecrawl

123,675 просмотров • 3 месяцев назад

Today we’re launching Design Canvas for AI agents. AI can build a feature, but every "design tweak" turns into another prompt. We built a drag & drop editor for Claude Code, Cursor, Codex to explore and iterate on your UI, so you get the best design by default. No endless prompting. Ship better designs in code with Subframe.
0:54

Today we’re launching Design Canvas for AI agents. AI can build a feature, but every "design tweak" turns into another prompt. We built a drag & drop editor for Claude Code, Cursor, Codex to explore and iterate on your UI, so you get the best design by default. No endless prompting. Ship better designs in code with Subframe.

Irvin Zhan

300,276 просмотров • 6 месяцев назад

Think of it as the Shopify of Claude Code, but for agents, commands, and MCPs 🚀 Pick what you need, “add to cart,” and… it’s 100% FREE. Just copy the command, paste it in your terminal, and instantly get the perfect Claude Code stack for your project. Built as an open-source project so the community can spin up, share, and install Claude Code stacks in seconds, no setup headaches, no paywalls. Here’s the video showing how I built this post… and yes, if you share the command, anyone can clone the exact same stack you’re using. Enjoy, share, and help others build faster with Claude Code ✨
0:59

Think of it as the Shopify of Claude Code, but for agents, commands, and MCPs 🚀 Pick what you need, “add to cart,” and… it’s 100% FREE. Just copy the command, paste it in your terminal, and instantly get the perfect Claude Code stack for your project. Built as an open-source project so the community can spin up, share, and install Claude Code stacks in seconds, no setup headaches, no paywalls. Here’s the video showing how I built this post… and yes, if you share the command, anyone can clone the exact same stack you’re using. Enjoy, share, and help others build faster with Claude Code ✨

Daniel San

124,913 просмотров • 1 год назад

CLAUDE CODE IS NOW FREE AND YOU CAN BUILD AN ENTIRE AI CODING SYSTEM WITH 1M CONTEXT, AGENT OS AND OBSIDIAN MEMORY WITHOUT PAYING FOR ANY OF IT. The stack gets smarter over time and costs you nothing to run.
0:34

CLAUDE CODE IS NOW FREE AND YOU CAN BUILD AN ENTIRE AI CODING SYSTEM WITH 1M CONTEXT, AGENT OS AND OBSIDIAN MEMORY WITHOUT PAYING FOR ANY OF IT. The stack gets smarter over time and costs you nothing to run.

0xMarioNawfal

51,201 просмотров • 2 месяцев назад

In pick the best model > improve your prompts > catch bugs with Braintrust and Cerebras inference. Avoid the AI deleting your entire codebase this halloween and remember, our free tier gets you 1M+ free toks/day per model.
1:16

In pick the best model > improve your prompts > catch bugs with Braintrust and Cerebras inference. Avoid the AI deleting your entire codebase this halloween and remember, our free tier gets you 1M+ free toks/day per model.

Cerebras

299,029 просмотров • 9 месяцев назад

Introducing - Custom agents! Build on our agentic harness to embed your personal expertise and trading strategies into a custom agent Coming soon, community agents > Create and refine your agents > Deploy the agent into the workforce > Other users can "hire" your agent
0:25

Introducing - Custom agents! Build on our agentic harness to embed your personal expertise and trading strategies into a custom agent Coming soon, community agents > Create and refine your agents > Deploy the agent into the workforce > Other users can "hire" your agent

tread.fi

27,114 просмотров • 25 дней назад

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC
13:01

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

10,969 просмотров • 10 месяцев назад

Figma Make, now on your local code In limited beta starting today, you can visually edit and ship changes by connecting Make to your codebase At Figma, we use our products to design in every way: design layers, prototypes, and now, production code
0:30

Figma Make, now on your local code In limited beta starting today, you can visually edit and ship changes by connecting Make to your codebase At Figma, we use our products to design in every way: design layers, prototypes, and now, production code

Figma

399,404 просмотров • 2 месяцев назад

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):
0:30

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,836 просмотров • 6 месяцев назад

Important new course: Agent Skills with Anthropic, built with Anthropic and taught by Elie Schoppik! Skills are constructed as folders of instructions that equip agents with on-demand knowledge and workflows. This short course teaches you how to create them following best practices. Because skills follow an open standard format, you can build them once and deploy across any skills-compatible agent, like Claude Code. What you'll learn: - Create custom skills for code generation and review, data analysis, and research - Build complex workflows using Anthropic's pre-built skills (Excel, PowerPoint, skill creation) and custom skills - Combine skills with MCP and subagents to create agentic systems with specialized knowledge - Deploy the same skills across Claude Code, the Claude API, and the Claude Agent SDK Join and learn to equip agents with the specialized knowledge they need for reliable, repeatable workflows.
1:59

Important new course: Agent Skills with Anthropic, built with Anthropic and taught by Elie Schoppik! Skills are constructed as folders of instructions that equip agents with on-demand knowledge and workflows. This short course teaches you how to create them following best practices. Because skills follow an open standard format, you can build them once and deploy across any skills-compatible agent, like Claude Code. What you'll learn: - Create custom skills for code generation and review, data analysis, and research - Build complex workflows using Anthropic's pre-built skills (Excel, PowerPoint, skill creation) and custom skills - Combine skills with MCP and subagents to create agentic systems with specialized knowledge - Deploy the same skills across Claude Code, the Claude API, and the Claude Agent SDK Join and learn to equip agents with the specialized knowledge they need for reliable, repeatable workflows.

Andrew Ng

888,862 просмотров • 6 месяцев назад

Agents live in terminals and tabs, people juggle between them. We built . Figma for agents: manage and run agent teams with no code. One workspace where humans and agents actually work together. Comment what you’re working on and we’ll build an agent team to help you.
1:08

Agents live in terminals and tabs, people juggle between them. We built . Figma for agents: manage and run agent teams with no code. One workspace where humans and agents actually work together. Comment what you’re working on and we’ll build an agent team to help you.

Stefano

81,834 просмотров • 4 месяцев назад

Introducing Claude Managed Agents: everything you need to build and deploy agents at scale. It pairs an agent harness tuned for performance with production infrastructure, so you can go from prototype to launch in days. Now in public beta on the Claude Platform.
0:59

Introducing Claude Managed Agents: everything you need to build and deploy agents at scale. It pairs an agent harness tuned for performance with production infrastructure, so you can go from prototype to launch in days. Now in public beta on the Claude Platform.

Claude

21,754,860 просмотров • 4 месяцев назад