Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Evals now supports tool use. 🛠️ You can now use tools and Structured Outputs when completing eval runs, and evaluate tool calls based on the arguments passed and responses returned. This supports tools that are OpenAI-hosted, MCP, and non-hosted. Read more in our guides below.

OpenAI Developers

363,875 subscribers

65,080 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

11 Comments

OpenAI Developers1 year ago

Web Search evaluation:

OpenAI Developers1 year ago

Tools evaluation:

OpenAI Developers1 year ago

Structured Outputs evaluation:

OpenAI Developers1 year ago

MCP evaluation:

Matt Figdore2 years ago

This is the biggest productivity cheat code right now. Kiss reading documents goodbye. You can get an instant summary of any document with this tool.

Opulent Byte1 year ago

When o3 pro? 🥲🫠

Brij Singh1 year ago

this is good!

Cliffinkent 🇬🇧1 year ago

You keep building and shipping!

Prashant1 year ago

This just makes running evals so much easier. Thank you

Vishal1 year ago

Tool support in Evals makes testing AI tools easier and improves accuracy.

Henkjan de Krijger1 year ago

Does relying on eval tools hinder critical thinking in AI systems?

Related Videos

We’re excited to share the beta release of our tool use API! Now developers can easily equip any LLM with hosted open-source tools for web search, web crawling, and maps data (+ more coming soon). Under the hood, it's powered by MCP — but that's just an implementation detail ↓

We’re excited to share the beta release of our tool use API! Now developers can easily equip any LLM with hosted open-source tools for web search, web crawling, and maps data (+ more coming soon). Under the hood, it's powered by MCP — but that's just an implementation detail ↓

OpenTools

125,965 views • 1 year ago

Gemini 3.5 Flash now supports native computer use. This built-in tool lets developers build custom agents that can see and take action across browser, mobile, and desktop interfaces. Find out more →

Gemini 3.5 Flash now supports native computer use. This built-in tool lets developers build custom agents that can see and take action across browser, mobile, and desktop interfaces. Find out more →

Google DeepMind

111,914 views • 1 month ago

Here’s Nikunj Handa and Steven Heidel to show you how MCP, reasoning, and multiple Responses API tools can work together. Read more in our blog:

Here’s Nikunj Handa and Steven Heidel to show you how MCP, reasoning, and multiple Responses API tools can work together. Read more in our blog:

OpenAI Developers

77,216 views • 1 year ago

Supabase MCP server now supports creating and deploying Edge Functions! Refresh your MCP server and make sure you have the `deploy_edge_function` tool, and you are good to go! Just ask your AI tool to create an edge function or a backend, and it will create it for you!

Supabase MCP server now supports creating and deploying Edge Functions! Refresh your MCP server and make sure you have the `deploy_edge_function` tool, and you are good to go! Just ask your AI tool to create an edge function or a backend, and it will create it for you!

Tyler Shukert

28,997 views • 1 year ago

PSA: Groq Inc now supports remote MCP server integration (BETA). ⚡️ LLMs are only as good as the context you give them and MCP enables you to connect them directly to live APIs, databases, and tools. Now your models can pull fresh data, perform actions, AND respond in seconds.

PSA: Groq Inc now supports remote MCP server integration (BETA). ⚡️ LLMs are only as good as the context you give them and MCP enables you to connect them directly to live APIs, databases, and tools. Now your models can pull fresh data, perform actions, AND respond in seconds.

Hatice Ozen

13,829 views • 10 months ago

Replit now supports 450+ integrations. No matter what you're building, Replit Agent can wire it up to the tools you already use: payments, messaging, data, CRMs, design tools, analytics, and hundreds more. Just describe what you want. We'll handle the connection.

Replit now supports 450+ integrations. No matter what you're building, Replit Agent can wire it up to the tools you already use: payments, messaging, data, CRMs, design tools, analytics, and hundreds more. Just describe what you want. We'll handle the connection.

Replit ⠕

40,147 views • 1 month ago

Cline can now create and add tools to himself using MCP! Try asking him to “add a tool that pulls the latest npm docs” - Cline handles everything from creating the MCP server to installing it *into himself*, ready for future tasks. Servers are saved in ~/Documents/Cline/MCP so you can easily share tools with friends!

Cline can now create and add tools to himself using MCP! Try asking him to “add a tool that pulls the latest npm docs” - Cline handles everything from creating the MCP server to installing it into himself, ready for future tasks. Servers are saved in ~/Documents/Cline/MCP so you can easily share tools with friends!

Saoud Rizwan

61,730 views • 1 year ago

Streamlining the UX in Gemini CLI ✨ We got a lot of feedback on how our UI was way too cluttered with noise and text... so we made changes! 🛠️ Compact tool calls: Tool calls for reading files, folders, searching text, etc are now a single line (no more tool boxes around everything!) 💭 Topics: The agent outputs a one line overview of the rationale and direction it is going. A topic can span several tool calls and makes it easy to see what the agent is working on at a glance and why.

Streamlining the UX in Gemini CLI ✨ We got a lot of feedback on how our UI was way too cluttered with noise and text... so we made changes! 🛠️ Compact tool calls: Tool calls for reading files, folders, searching text, etc are now a single line (no more tool boxes around everything!) 💭 Topics: The agent outputs a one line overview of the rationale and direction it is going. A topic can span several tool calls and makes it easy to see what the agent is working on at a glance and why.

Jack Wotherspoon

15,960 views • 2 months ago

MCP connectors are now available in Basedash. Basedash already reads from your databases and SaaS tools. Now it can act on them too. Connect any MCP server (Linear, HubSpot, Slack, Resend, Notion, GitHub, your own internal one) and the Basedash agent gets new tools it can use right inside chat. Try it today.

MCP connectors are now available in Basedash. Basedash already reads from your databases and SaaS tools. Now it can act on them too. Connect any MCP server (Linear, HubSpot, Slack, Resend, Notion, GitHub, your own internal one) and the Basedash agent gets new tools it can use right inside chat. Try it today.

Basedash

11,138 views • 2 months ago

With Google Flow Tools, you can now use natural language to create bespoke tools and workflows. In this video, you'll see how you can: ➡️ Explore and try the various Tools within Google Flow ➡️ Remix a tool to make it even more relevant for your unique workflow ➡️Create your own tool from scratch with a natural language prompt ➡️ Share your tools with a friend for them to try out Whether you’re looking for an image editor, video resizer or custom shaders, you can now create them in Google Flow, no coding experience required.

With Google Flow Tools, you can now use natural language to create bespoke tools and workflows. In this video, you'll see how you can: ➡️ Explore and try the various Tools within Google Flow ➡️ Remix a tool to make it even more relevant for your unique workflow ➡️Create your own tool from scratch with a natural language prompt ➡️ Share your tools with a friend for them to try out Whether you’re looking for an image editor, video resizer or custom shaders, you can now create them in Google Flow, no coding experience required.

Google Flow

19,473 views • 1 month ago

🚀 LangSmith Agent Builder is now available in Public Beta Now anyone can create production ready agents without writing code, just chat. Agent Builder guides you from initial idea to deployed agent, creating detailed prompts, selecting required tools, and even creating subagents. Our Beta release also includes: 🧰 Bring your own tools via MCP server 🧩 Browse, copy, and customize agents across a shared workspace 🧠 Use your preferred model Try Agent Builder free: Read more on the launch:

🚀 LangSmith Agent Builder is now available in Public Beta Now anyone can create production ready agents without writing code, just chat. Agent Builder guides you from initial idea to deployed agent, creating detailed prompts, selecting required tools, and even creating subagents. Our Beta release also includes: 🧰 Bring your own tools via MCP server 🧩 Browse, copy, and customize agents across a shared workspace 🧠 Use your preferred model Try Agent Builder free: Read more on the launch:

LangChain

37,802 views • 7 months ago

130+ skills are now built into your Superagent. Some are ready to use, and some can be created based on what you need. Add a skill once, and your Superagent can use it as part of your workflows. Stack skills, connect tools, and build flows that run end-to-end.

130+ skills are now built into your Superagent. Some are ready to use, and some can be created based on what you need. Add a skill once, and your Superagent can use it as part of your workflows. Stack skills, connect tools, and build flows that run end-to-end.

Base44

1,858,657 views • 4 months ago

ChatGPT can now read responses to you. On iOS or Android, tap and hold the message and then tap “Read Aloud”. We’ve also started rolling on web - click the "Read Aloud" button below the message.

ChatGPT can now read responses to you. On iOS or Android, tap and hold the message and then tap “Read Aloud”. We’ve also started rolling on web - click the "Read Aloud" button below the message.

OpenAI

1,026,908 views • 2 years ago

Not just AI. A tool people actually use. The people in this video are real SMEs who already use MWX tools and see real benefits: faster content creation, steady ideas, lower costs, and more consistent results. AI that works for real businesses, not just demos.

Not just AI. A tool people actually use. The people in this video are real SMEs who already use MWX tools and see real benefits: faster content creation, steady ideas, lower costs, and more consistent results. AI that works for real businesses, not just demos.

MWX | Real-Time Alpha AI

40,596 views • 7 months ago

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

Junxian He

43,743 views • 9 months ago

Codex for (almost) everything. It can now use apps on your Mac, connect to more of your tools, create images, learn from previous actions, remember how you like to work, and take on ongoing and repeatable tasks.

Codex for (almost) everything. It can now use apps on your Mac, connect to more of your tools, create images, learn from previous actions, remember how you like to work, and take on ongoing and repeatable tasks.

OpenAI

3,407,061 views • 3 months ago

Hosted skills are now supported in TanStack AI. 🧩 Pass { skills } to your code-execution tool, TanStack AI lifts them into container skills and auto-wires the beta headers. Anthropic + OpenAI. 🚀

Hosted skills are now supported in TanStack AI. 🧩 Pass { skills } to your code-execution tool, TanStack AI lifts them into container skills and auto-wires the beta headers. Anthropic + OpenAI. 🚀

TANSTACK

14,549 views • 1 month ago

You are brain-dead if you think the CLI killed MCP. Anthropic just reported 300 million downloads for the MCP SDK (up from 100M at the beginning of the year). The complaint I keep hearing: MCP pollutes your context. That's a 1-year-old argument. Context pollution is not a protocol problem, but a client one. None of the agentic tools you use have that issue: Claude Code, Codex, Cursor, etc. All of them implement progressive disclosure and load MCP tools on demand. MCPs and CLIs complement each other: • MCP offers OAuth-based authentication, standardized transport, remote access for cloud-hosted agents, and a public registry with thousands of servers. • CLIs offer Unix-style composition (pipes, stdin/stdout), zero upfront schema cost, and massive training distribution (models have seen far more shell than any structured tool format). You don't (and shouldn't) pick a side here. You should be using both where they fit.

You are brain-dead if you think the CLI killed MCP. Anthropic just reported 300 million downloads for the MCP SDK (up from 100M at the beginning of the year). The complaint I keep hearing: MCP pollutes your context. That's a 1-year-old argument. Context pollution is not a protocol problem, but a client one. None of the agentic tools you use have that issue: Claude Code, Codex, Cursor, etc. All of them implement progressive disclosure and load MCP tools on demand. MCPs and CLIs complement each other: • MCP offers OAuth-based authentication, standardized transport, remote access for cloud-hosted agents, and a public registry with thousands of servers. • CLIs offer Unix-style composition (pipes, stdin/stdout), zero upfront schema cost, and massive training distribution (models have seen far more shell than any structured tool format). You don't (and shouldn't) pick a side here. You should be using both where they fit.

Santiago

68,584 views • 2 months ago