Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Evals now supports tool use. 🛠️ You can now use tools and Structured Outputs when completing eval runs, and evaluate tool calls based on the arguments passed and responses returned. This supports tools that are OpenAI-hosted, MCP, and non-hosted. Read more in our guides below.

OpenAIDevs's profile picture

OpenAI Developers

363,875 subscribers

65,080 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

11 Kommentare

Profilbild von OpenAI Developers
OpenAI Developersvor 1 Jahr

Web Search evaluation:

Profilbild von OpenAI Developers
OpenAI Developersvor 1 Jahr

Tools evaluation:

Profilbild von OpenAI Developers
OpenAI Developersvor 1 Jahr

Structured Outputs evaluation:

Profilbild von OpenAI Developers
OpenAI Developersvor 1 Jahr

MCP evaluation:

Profilbild von Matt Figdore
Matt Figdorevor 2 Jahren

This is the biggest productivity cheat code right now. Kiss reading documents goodbye. You can get an instant summary of any document with this tool.

Profilbild von Opulent Byte
Opulent Bytevor 1 Jahr

When o3 pro? 🥲🫠

Profilbild von Brij Singh
Brij Singhvor 1 Jahr

this is good!

Profilbild von Cliffinkent 🇬🇧
Cliffinkent 🇬🇧vor 1 Jahr

You keep building and shipping!

Profilbild von Prashant
Prashantvor 1 Jahr

This just makes running evals so much easier. Thank you

Profilbild von Vishal
Vishalvor 1 Jahr

Tool support in Evals makes testing AI tools easier and improves accuracy.

Profilbild von Henkjan de Krijger
Henkjan de Krijgervor 1 Jahr

Does relying on eval tools hinder critical thinking in AI systems?

Ähnliche Videos

We’re excited to share the beta release of our tool use API! Now developers can easily equip any LLM with hosted open-source tools for web search, web crawling, and maps data (+ more coming soon). Under the hood, it's powered by MCP — but that's just an implementation detail ↓
3:19

We’re excited to share the beta release of our tool use API! Now developers can easily equip any LLM with hosted open-source tools for web search, web crawling, and maps data (+ more coming soon). Under the hood, it's powered by MCP — but that's just an implementation detail ↓

OpenTools

126,577 Aufrufe • vor 1 Jahr

Gemini 3.5 Flash now supports native computer use. This built-in tool lets developers build custom agents that can see and take action across browser, mobile, and desktop interfaces. Find out more →
1:21

Gemini 3.5 Flash now supports native computer use. This built-in tool lets developers build custom agents that can see and take action across browser, mobile, and desktop interfaces. Find out more →

Google DeepMind

117,756 Aufrufe • vor 3 Monaten

Here’s Nikunj Handa and Steven Heidel to show you how MCP, reasoning, and multiple Responses API tools can work together. Read more in our blog:
14:33

Here’s Nikunj Handa and Steven Heidel to show you how MCP, reasoning, and multiple Responses API tools can work together. Read more in our blog:

OpenAI Developers

77,251 Aufrufe • vor 1 Jahr

New stateful server tool: Shell! Any model on OpenRouter can now run commands in a hosted Linux container and use the new Files API to move files in and out. Available today in beta: `openrouter:shell`
0:23

New stateful server tool: Shell! Any model on OpenRouter can now run commands in a hosted Linux container and use the new Files API to move files in and out. Available today in beta: `openrouter:shell`

OpenRouter

23,952 Aufrufe • vor 16 Tagen

PSA: Groq Inc now supports remote MCP server integration (BETA). ⚡️ LLMs are only as good as the context you give them and MCP enables you to connect them directly to live APIs, databases, and tools. Now your models can pull fresh data, perform actions, AND respond in seconds.
7:30

PSA: Groq Inc now supports remote MCP server integration (BETA). ⚡️ LLMs are only as good as the context you give them and MCP enables you to connect them directly to live APIs, databases, and tools. Now your models can pull fresh data, perform actions, AND respond in seconds.

Hatice Ozen

13,829 Aufrufe • vor 1 Jahr

Replit now supports 450+ integrations. No matter what you're building, Replit Agent can wire it up to the tools you already use: payments, messaging, data, CRMs, design tools, analytics, and hundreds more. Just describe what you want. We'll handle the connection.
0:26

Replit now supports 450+ integrations. No matter what you're building, Replit Agent can wire it up to the tools you already use: payments, messaging, data, CRMs, design tools, analytics, and hundreds more. Just describe what you want. We'll handle the connection.

Replit ⠕

40,147 Aufrufe • vor 3 Monaten

Cline can now create and add tools to himself using MCP! Try asking him to “add a tool that pulls the latest npm docs” - Cline handles everything from creating the MCP server to installing it *into himself*, ready for future tasks. Servers are saved in ~/Documents/Cline/MCP so you can easily share tools with friends!
3:24

Cline can now create and add tools to himself using MCP! Try asking him to “add a tool that pulls the latest npm docs” - Cline handles everything from creating the MCP server to installing it *into himself*, ready for future tasks. Servers are saved in ~/Documents/Cline/MCP so you can easily share tools with friends!

Saoud Rizwan

62,122 Aufrufe • vor 1 Jahr

Streamlining the UX in Gemini CLI ✨ We got a lot of feedback on how our UI was way too cluttered with noise and text... so we made changes! 🛠️ Compact tool calls: Tool calls for reading files, folders, searching text, etc are now a single line (no more tool boxes around everything!) 💭 Topics: The agent outputs a one line overview of the rationale and direction it is going. A topic can span several tool calls and makes it easy to see what the agent is working on at a glance and why.
0:33

Streamlining the UX in Gemini CLI ✨ We got a lot of feedback on how our UI was way too cluttered with noise and text... so we made changes! 🛠️ Compact tool calls: Tool calls for reading files, folders, searching text, etc are now a single line (no more tool boxes around everything!) 💭 Topics: The agent outputs a one line overview of the rationale and direction it is going. A topic can span several tool calls and makes it easy to see what the agent is working on at a glance and why.

Jack Wotherspoon

15,960 Aufrufe • vor 4 Monaten

MCP connectors are now available in Basedash. Basedash already reads from your databases and SaaS tools. Now it can act on them too. Connect any MCP server (Linear, HubSpot, Slack, Resend, Notion, GitHub, your own internal one) and the Basedash agent gets new tools it can use right inside chat. Try it today.
0:29

MCP connectors are now available in Basedash. Basedash already reads from your databases and SaaS tools. Now it can act on them too. Connect any MCP server (Linear, HubSpot, Slack, Resend, Notion, GitHub, your own internal one) and the Basedash agent gets new tools it can use right inside chat. Try it today.

Basedash

11,138 Aufrufe • vor 4 Monaten

10 agent evals every AI engineer should know 1) golden set a frozen set of cases you run after every prompt model or tool change use it as your baseline to see whether the system improved or quietly broke OpenAI Evals helps you build repeatable benchmark sets and compare model changes → 2) llm as judge a second model scores open ended answers against a written rubric use it when there is no exact output to compare against OpenEvals provides ready made evaluators for LLM applications → 3) rubric scoring score correctness tone safety and cost separately one quality number hides the problem DeepEval helps you create custom metrics and score every dimension independently → 4) trajectory eval grade the path the agent took not only the final answer AgentEvals checks agent actions decisions and tool calls across the full trajectory → 5) tool unit tests test every tool with fixed inputs and outputs no model in the loop MCP Inspector helps you inspect and test MCP servers tools and responses separately → 6) regression suite replay previous runs against every new prompt model or toolset then compare the results Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI → 7) a b in prod split real traffic between two versions and compare actual outcomes GrowthBook provides feature flags controlled experiments and product analytics → 8) human review sample real runs and let a person grade them honestly use it to calibrate your automated judge Argilla helps teams collect human feedback review outputs and build better datasets → 9) shadow run let the candidate run on real traffic while its output is shown to nobody use it before a risky rollout Langfuse helps you trace production runs compare candidates and monitor eval results → 10) red team attack the system before somebody else does jailbreaks prompt injection data leaks and tool abuse Garak scans LLM systems for vulnerabilities and unsafe behavior → offline evals tell you it works online evals tell you it still works you probably do not need all ten today start with the two that would have caught your last outage bookmark this
0:21

10 agent evals every AI engineer should know 1) golden set a frozen set of cases you run after every prompt model or tool change use it as your baseline to see whether the system improved or quietly broke OpenAI Evals helps you build repeatable benchmark sets and compare model changes → 2) llm as judge a second model scores open ended answers against a written rubric use it when there is no exact output to compare against OpenEvals provides ready made evaluators for LLM applications → 3) rubric scoring score correctness tone safety and cost separately one quality number hides the problem DeepEval helps you create custom metrics and score every dimension independently → 4) trajectory eval grade the path the agent took not only the final answer AgentEvals checks agent actions decisions and tool calls across the full trajectory → 5) tool unit tests test every tool with fixed inputs and outputs no model in the loop MCP Inspector helps you inspect and test MCP servers tools and responses separately → 6) regression suite replay previous runs against every new prompt model or toolset then compare the results Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI → 7) a b in prod split real traffic between two versions and compare actual outcomes GrowthBook provides feature flags controlled experiments and product analytics → 8) human review sample real runs and let a person grade them honestly use it to calibrate your automated judge Argilla helps teams collect human feedback review outputs and build better datasets → 9) shadow run let the candidate run on real traffic while its output is shown to nobody use it before a risky rollout Langfuse helps you trace production runs compare candidates and monitor eval results → 10) red team attack the system before somebody else does jailbreaks prompt injection data leaks and tool abuse Garak scans LLM systems for vulnerabilities and unsafe behavior → offline evals tell you it works online evals tell you it still works you probably do not need all ten today start with the two that would have caught your last outage bookmark this

elune

155,026 Aufrufe • vor 2 Monaten

With Google Flow Tools, you can now use natural language to create bespoke tools and workflows. In this video, you'll see how you can: ➡️ Explore and try the various Tools within Google Flow ➡️ Remix a tool to make it even more relevant for your unique workflow ➡️Create your own tool from scratch with a natural language prompt ➡️ Share your tools with a friend for them to try out Whether you’re looking for an image editor, video resizer or custom shaders, you can now create them in Google Flow, no coding experience required.
2:30

With Google Flow Tools, you can now use natural language to create bespoke tools and workflows. In this video, you'll see how you can: ➡️ Explore and try the various Tools within Google Flow ➡️ Remix a tool to make it even more relevant for your unique workflow ➡️Create your own tool from scratch with a natural language prompt ➡️ Share your tools with a friend for them to try out Whether you’re looking for an image editor, video resizer or custom shaders, you can now create them in Google Flow, no coding experience required.

Google Flow

20,037 Aufrufe • vor 3 Monaten

🚀 LangSmith Agent Builder is now available in Public Beta Now anyone can create production ready agents without writing code, just chat. Agent Builder guides you from initial idea to deployed agent, creating detailed prompts, selecting required tools, and even creating subagents. Our Beta release also includes: 🧰 Bring your own tools via MCP server 🧩 Browse, copy, and customize agents across a shared workspace 🧠 Use your preferred model Try Agent Builder free: Read more on the launch:
0:56

🚀 LangSmith Agent Builder is now available in Public Beta Now anyone can create production ready agents without writing code, just chat. Agent Builder guides you from initial idea to deployed agent, creating detailed prompts, selecting required tools, and even creating subagents. Our Beta release also includes: 🧰 Bring your own tools via MCP server 🧩 Browse, copy, and customize agents across a shared workspace 🧠 Use your preferred model Try Agent Builder free: Read more on the launch:

LangChain

37,866 Aufrufe • vor 9 Monaten

130+ skills are now built into your Superagent. Some are ready to use, and some can be created based on what you need. Add a skill once, and your Superagent can use it as part of your workflows. Stack skills, connect tools, and build flows that run end-to-end.
0:32

130+ skills are now built into your Superagent. Some are ready to use, and some can be created based on what you need. Add a skill once, and your Superagent can use it as part of your workflows. Stack skills, connect tools, and build flows that run end-to-end.

Base44

1,861,311 Aufrufe • vor 6 Monaten

ChatGPT can now read responses to you. On iOS or Android, tap and hold the message and then tap “Read Aloud”. We’ve also started rolling on web - click the "Read Aloud" button below the message.
0:28

ChatGPT can now read responses to you. On iOS or Android, tap and hold the message and then tap “Read Aloud”. We’ve also started rolling on web - click the "Read Aloud" button below the message.

OpenAI

1,027,117 Aufrufe • vor 2 Jahren

Not just AI. A tool people actually use. The people in this video are real SMEs who already use MWX tools and see real benefits: faster content creation, steady ideas, lower costs, and more consistent results. AI that works for real businesses, not just demos.
1:05

Not just AI. A tool people actually use. The people in this video are real SMEs who already use MWX tools and see real benefits: faster content creation, steady ideas, lower costs, and more consistent results. AI that works for real businesses, not just demos.

MWX | Real-Time Alpha AI

40,596 Aufrufe • vor 9 Monaten

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️
1:06

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

Junxian He

44,425 Aufrufe • vor 11 Monaten

Support for Agent Plugins is live in Cursor (and I made a plugin for searching X!) Agent Plugins combine Skills and MCP servers into a single package Think about them like an Ikea furniture kit: your table comes with instructions, parts, and tools. You use the instructions to understand how things work (which tools on which parts), then use the tools to assemble your furniture Agent Plugins are bundles of tools & resources that allow agents to accomplish a goal: - MCP servers: fixed context and tools - Skills: dynamic instructions, loaded progressively Plugins bundle tools and instructions so your agent is able to use tokens efficiently Many of us already use MCP servers and skills together - Agent Plugins formalize this pattern for more powerful workflows I'm excited to see where the standard takes us!
0:24

Support for Agent Plugins is live in Cursor (and I made a plugin for searching X!) Agent Plugins combine Skills and MCP servers into a single package Think about them like an Ikea furniture kit: your table comes with instructions, parts, and tools. You use the instructions to understand how things work (which tools on which parts), then use the tools to assemble your furniture Agent Plugins are bundles of tools & resources that allow agents to accomplish a goal: - MCP servers: fixed context and tools - Skills: dynamic instructions, loaded progressively Plugins bundle tools and instructions so your agent is able to use tokens efficiently Many of us already use MCP servers and skills together - Agent Plugins formalize this pattern for more powerful workflows I'm excited to see where the standard takes us!

matt palmer

40,657 Aufrufe • vor 1 Monat