Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Run evals—directly from the OpenAI dashboard. Use your test data to compare model performance, iterate on prompts, and improve outputs. Here's a quick walkthrough:

OpenAIDevs's profile picture

OpenAI Developers

363,875 subscribers

86,738 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von alex fazio
alex faziovor 1 Jahr

just a chill guy k-lling thousands of startups with a single post

Profilbild von Markus Odenthal
Markus Odenthalvor 1 Jahr

This is pretty cool! Most people miss the evaluation steps. But it's actually crucial.

Profilbild von Quintus 🏛️
Quintus 🏛️vor 1 Jahr

Can you let us export the eval to a .csv?

Profilbild von Prompt Perfect
Prompt Perfectvor 1 Jahr

this is a healthy post.

Profilbild von william
williamvor 1 Jahr

eval api wen?

Profilbild von Diego | AI 🚀 - e/acc
Diego | AI 🚀 - e/accvor 1 Jahr

🧪 What Are OpenAI Evals? OpenAI Evals is a tool to test and measure how well AI models perform specific tasks. Think of it like a report card for AI—researchers and developers use it to ensure the model gives accurate and helpful answers before it’s used in the real world. 📊🤖

Profilbild von TestingCatalog News 🗞
TestingCatalog News 🗞vor 1 Jahr

I think now we will need a way to run these on CI 😋

Profilbild von Jim Hull
Jim Hullvor 1 Jahr

This is same as it has been for two months right? Took me a bit to wrap my head around it, but now i want to eval EVERYTHING! Great feature.

Profilbild von Chase Brower
Chase Browervor 1 Jahr

The factuality passing grades seems to be mixed up somewhere in the pipeline, it gives a result (A) matching what I selected, but then it says fail. I found if I check the 'Response disagrees with the ground truth' (and only that) then it says success.

Profilbild von Soumyajit
Soumyajitvor 1 Jahr

This is a great feature.

Ähnliche Videos

here's a quick look at an internal tool we made to build agents faster. we iterate on our agent's internal structure, run through dynamically generated prompts, and test new samples of outputs more in-distribution of how our agents act. let us know below if you'd like access
0:30

here's a quick look at an internal tool we made to build agents faster. we iterate on our agent's internal structure, run through dynamically generated prompts, and test new samples of outputs more in-distribution of how our agents act. let us know below if you'd like access

nico

15,692 Aufrufe • vor 2 Jahren

The Chat Playground is now the Prompts Playground—we've redesigned it to help you better test, compare, and iterate on prompts, including with tools like web search and file search. 🛝
0:28

The Chat Playground is now the Prompts Playground—we've redesigned it to help you better test, compare, and iterate on prompts, including with tools like web search and file search. 🛝

OpenAI Developers

94,721 Aufrufe • vor 1 Jahr

Wow OpenAI just launched ChatGPT Tasks and it's basically AI Agents for everyone Now AI Agents can do work for you 24/7 Here's your full walkthrough, how to use ChatGPT Tasks best, and 6 prompts that will IMMEDIATELY improve your life (trust me you want to bookmark this one)
7:49

Wow OpenAI just launched ChatGPT Tasks and it's basically AI Agents for everyone Now AI Agents can do work for you 24/7 Here's your full walkthrough, how to use ChatGPT Tasks best, and 6 prompts that will IMMEDIATELY improve your life (trust me you want to bookmark this one)

Alex Finn

566,331 Aufrufe • vor 1 Jahr

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,680 Aufrufe • vor 6 Monaten

how you can use openAI codex & gpt 5.5 completely FREE (the full guide) 100% legit. no subscription, zero API cost. up to 1M+ token/day. you need just an openAI account and here's how to set it up in 5mins. openAI has a program that gives eligible developers free API usage every day in exchange for sharing API data that helps improve future models. it's not a one-time credit, your allowance refreshes daily. depending on your usage tier, you can get access to hundreds of thousands, or even millions, of free tokens every single day on supported models. here's how to activate it: 1️⃣open your API dashboard: 2️⃣go to settings → data controls 3️⃣enable data sharing for your organization or project 4️⃣make sure your account has a positive API balance 5️⃣save the settings if your account is eligible, you'll see a message confirming access to complimentary daily usage. before you turn it on, know the tradeoff: • prompts and outputs from shared projects can be used to improve openai's models • don't use it for confidential information, client work, or sensitive data • eligibility depends on your account type and settings for everyone else, it's an incredible deal. use it to: • learn AI development • build side projects • experiment with codex • test agents and automations • prototype ideas without worrying about API costs most developers burn money testing ideas. this lets you experiment at scale while spending little to nothing.
1:27

how you can use openAI codex & gpt 5.5 completely FREE (the full guide) 100% legit. no subscription, zero API cost. up to 1M+ token/day. you need just an openAI account and here's how to set it up in 5mins. openAI has a program that gives eligible developers free API usage every day in exchange for sharing API data that helps improve future models. it's not a one-time credit, your allowance refreshes daily. depending on your usage tier, you can get access to hundreds of thousands, or even millions, of free tokens every single day on supported models. here's how to activate it: 1️⃣open your API dashboard: 2️⃣go to settings → data controls 3️⃣enable data sharing for your organization or project 4️⃣make sure your account has a positive API balance 5️⃣save the settings if your account is eligible, you'll see a message confirming access to complimentary daily usage. before you turn it on, know the tradeoff: • prompts and outputs from shared projects can be used to improve openai's models • don't use it for confidential information, client work, or sensitive data • eligibility depends on your account type and settings for everyone else, it's an incredible deal. use it to: • learn AI development • build side projects • experiment with codex • test agents and automations • prototype ideas without worrying about API costs most developers burn money testing ideas. this lets you experiment at scale while spending little to nothing.

m0h

71,147 Aufrufe • vor 4 Monaten

It is very easy to make mistakes when creating evals for your AI product. Shreya Shankar and I run through the most common mistakes in this talk (with memes 🌶️!) . Chapter summaries below: 00:51 Foundation model benchmarks are not the same as your application evals 03:00 Generic Evals Are Useless 04:00 Do not outsource labeling & prompting to non domain experts 09:28 You should make your own data annotation app 12:40 Your LLM prompts should be specific and grounded in error analysis 15:25 Use binary labels 18:57 Look at your data 23:41 Be careful of overfitting to test data 25:40 Do online tests Links more resources in the reply
28:15

It is very easy to make mistakes when creating evals for your AI product. Shreya Shankar and I run through the most common mistakes in this talk (with memes 🌶️!) . Chapter summaries below: 00:51 Foundation model benchmarks are not the same as your application evals 03:00 Generic Evals Are Useless 04:00 Do not outsource labeling & prompting to non domain experts 09:28 You should make your own data annotation app 12:40 Your LLM prompts should be specific and grounded in error analysis 15:25 Use binary labels 18:57 Look at your data 23:41 Be careful of overfitting to test data 25:40 Do online tests Links more resources in the reply

Hamel Husain

46,184 Aufrufe • vor 1 Jahr

10 agent evals every AI engineer should know 1) golden set a frozen set of cases you run after every prompt model or tool change use it as your baseline to see whether the system improved or quietly broke OpenAI Evals helps you build repeatable benchmark sets and compare model changes → 2) llm as judge a second model scores open ended answers against a written rubric use it when there is no exact output to compare against OpenEvals provides ready made evaluators for LLM applications → 3) rubric scoring score correctness tone safety and cost separately one quality number hides the problem DeepEval helps you create custom metrics and score every dimension independently → 4) trajectory eval grade the path the agent took not only the final answer AgentEvals checks agent actions decisions and tool calls across the full trajectory → 5) tool unit tests test every tool with fixed inputs and outputs no model in the loop MCP Inspector helps you inspect and test MCP servers tools and responses separately → 6) regression suite replay previous runs against every new prompt model or toolset then compare the results Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI → 7) a b in prod split real traffic between two versions and compare actual outcomes GrowthBook provides feature flags controlled experiments and product analytics → 8) human review sample real runs and let a person grade them honestly use it to calibrate your automated judge Argilla helps teams collect human feedback review outputs and build better datasets → 9) shadow run let the candidate run on real traffic while its output is shown to nobody use it before a risky rollout Langfuse helps you trace production runs compare candidates and monitor eval results → 10) red team attack the system before somebody else does jailbreaks prompt injection data leaks and tool abuse Garak scans LLM systems for vulnerabilities and unsafe behavior → offline evals tell you it works online evals tell you it still works you probably do not need all ten today start with the two that would have caught your last outage bookmark this
0:21

10 agent evals every AI engineer should know 1) golden set a frozen set of cases you run after every prompt model or tool change use it as your baseline to see whether the system improved or quietly broke OpenAI Evals helps you build repeatable benchmark sets and compare model changes → 2) llm as judge a second model scores open ended answers against a written rubric use it when there is no exact output to compare against OpenEvals provides ready made evaluators for LLM applications → 3) rubric scoring score correctness tone safety and cost separately one quality number hides the problem DeepEval helps you create custom metrics and score every dimension independently → 4) trajectory eval grade the path the agent took not only the final answer AgentEvals checks agent actions decisions and tool calls across the full trajectory → 5) tool unit tests test every tool with fixed inputs and outputs no model in the loop MCP Inspector helps you inspect and test MCP servers tools and responses separately → 6) regression suite replay previous runs against every new prompt model or toolset then compare the results Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI → 7) a b in prod split real traffic between two versions and compare actual outcomes GrowthBook provides feature flags controlled experiments and product analytics → 8) human review sample real runs and let a person grade them honestly use it to calibrate your automated judge Argilla helps teams collect human feedback review outputs and build better datasets → 9) shadow run let the candidate run on real traffic while its output is shown to nobody use it before a risky rollout Langfuse helps you trace production runs compare candidates and monitor eval results → 10) red team attack the system before somebody else does jailbreaks prompt injection data leaks and tool abuse Garak scans LLM systems for vulnerabilities and unsafe behavior → offline evals tell you it works online evals tell you it still works you probably do not need all ten today start with the two that would have caught your last outage bookmark this

elune

155,026 Aufrufe • vor 3 Monaten

Run AI prompts directly from your menubar 💥
0:23

Run AI prompts directly from your menubar 💥

Martin Rariga

20,866 Aufrufe • vor 2 Monaten

You can also use gpt-image-1 in the Playground to quickly iterate on prompts and images:
0:34

You can also use gpt-image-1 in the Playground to quickly iterate on prompts and images:

OpenAI Developers

69,117 Aufrufe • vor 1 Jahr

Here's a great use case for Claude and the Basecamp CLI. Pull together some inspirational photos of iconic furniture from around the world, and upload them to the appropriately named galleries inside a Basecamp project. Here's a quick walkthrough:
1:34

Here's a great use case for Claude and the Basecamp CLI. Pull together some inspirational photos of iconic furniture from around the world, and upload them to the appropriately named galleries inside a Basecamp project. Here's a quick walkthrough:

Jason Fried

140,204 Aufrufe • vor 1 Monat

Last week we launched Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency Watch our new video walkthrough to see how to build and run your own benchmark with Optima, and find the best model for your use case Start building your benchmark today at
17:43

Last week we launched Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency Watch our new video walkthrough to see how to build and run your own benchmark with Optima, and find the best model for your use case Start building your benchmark today at

Artificial Analysis

19,837 Aufrufe • vor 2 Monaten

Create datasets, run evals, and even train models directly in Cursor with the Hugging Face plugin. Here's Ben Burtenshaw to show you how:
1:53

Create datasets, run evals, and even train models directly in Cursor with the Hugging Face plugin. Here's Ben Burtenshaw to show you how:

edwin

17,119 Aufrufe • vor 7 Monaten

Today we're launching the PlanetScale hosted MCP server. Use your favorite AI agent to explore your data, identify slow queries, and improve database performance.
5:55

Today we're launching the PlanetScale hosted MCP server. Use your favorite AI agent to explore your data, identify slow queries, and improve database performance.

PlanetScale

29,528 Aufrufe • vor 9 Monaten

Model, generate, refine and iterate - that's the loop! Then export your model to use anywhere you like. #madeWithUnbound
1:21

Model, generate, refine and iterate - that's the loop! Then export your model to use anywhere you like. #madeWithUnbound

Unbound

11,703 Aufrufe • vor 5 Monaten

As promised, here's a quick clip of the OG Xbox dashboard running natively on a Nintendo Switch thanks to the amazing work from Milenko and TeamUIX
0:30

As promised, here's a quick clip of the OG Xbox dashboard running natively on a Nintendo Switch thanks to the amazing work from Milenko and TeamUIX

Generalkidd

55,572 Aufrufe • vor 6 Monaten

Not sure how to submit your monthly evaluation form? Here's a quick walkthrough to guide you through it, from logging into the portal to getting your form verified.
1:13

Not sure how to submit your monthly evaluation form? Here's a quick walkthrough to guide you through it, from logging into the portal to getting your form verified.

NSA GHANA

15,937 Aufrufe • vor 3 Monaten

The Tidewave Toolbar recently got the ability to visualise client and server updates on the page. Here is a quick video showing how you can use it to improve the performance of your LiveView apps!
5:12

The Tidewave Toolbar recently got the ability to visualise client and server updates on the page. Here is a quick video showing how you can use it to improve the performance of your LiveView apps!

José Valim

16,542 Aufrufe • vor 2 Monaten

you might've heard about context engineering, but nobody has explained what it really is... that's why I made a full practical walkthrough on how to use it to make your AI outputs personalized and consistent in this walkthrough, I go over: – what context engineering actually is (without the jargon) – how to train AI on your own data and preferences – how to get outputs that sound like you wrote them (not generic AI) reply "context" and I'll DM it to you (must be following)
0:36

you might've heard about context engineering, but nobody has explained what it really is... that's why I made a full practical walkthrough on how to use it to make your AI outputs personalized and consistent in this walkthrough, I go over: – what context engineering actually is (without the jargon) – how to train AI on your own data and preferences – how to get outputs that sound like you wrote them (not generic AI) reply "context" and I'll DM it to you (must be following)

Tyler

20,023 Aufrufe • vor 1 Jahr