Loading video...

Video Failed to Load

Go Home

Today, we are launching Hosted Evaluations on the platform. Running evals is an infra problem: harnesses, sandboxes, hours of compute, hundreds of parallel runs. Running evals is hard. Until now.

PrimeIntellect's profile picture

Prime Intellect

66,802 subscribers

125,107 views • 29 days ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:
0:53

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:

Awni Hannun

132,388 views • 1 year ago

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥
5:42

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

James Grugett

276,478 views • 9 months ago

Today, we shared evals for an early version of the next model in our o-model reasoning series: OpenAI o3
9:38

Today, we shared evals for an early version of the next model in our o-model reasoning series: OpenAI o3

OpenAI

1,668,791 views • 1 year ago

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals
1:18

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals

Haider.

45,058 views • 7 months ago

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,602 views • 2 months ago

How do we understand & evaluate the fuzzy space of LLM outputs? We clone your Subject Matter Expert annotator into a Judge. Introducing EVALS EVALS EVALS Create a custom Judge that works for you
1:26

How do we understand & evaluate the fuzzy space of LLM outputs? We clone your Subject Matter Expert annotator into a Judge. Introducing EVALS EVALS EVALS Create a custom Judge that works for you

Leonard Tang

36,496 views • 1 year ago

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 views • 9 months ago

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.
0:35

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.

Modal

32,334 views • 4 months ago

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna
1:03:59

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

246,941 views • 4 months ago

Today, we’re launching “The Dubbing Rubric” — a method for properly evaluating AI dubbing systems. Including a set of human evals comparing popular solutions on the market - check it out!
0:35

Today, we’re launching “The Dubbing Rubric” — a method for properly evaluating AI dubbing systems. Including a set of human evals comparing popular solutions on the market - check it out!

mok

12,061 views • 1 year ago

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,279 views • 3 months ago

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC
13:01

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

10,969 views • 9 months ago

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.
0:52

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.

Bel-Air on Peacock

14,334 views • 6 months ago

Evals now supports tool use. 🛠️ You can now use tools and Structured Outputs when completing eval runs, and evaluate tool calls based on the arguments passed and responses returned. This supports tools that are OpenAI-hosted, MCP, and non-hosted. Read more in our guides below.
0:36

Evals now supports tool use. 🛠️ You can now use tools and Structured Outputs when completing eval runs, and evaluate tool calls based on the arguments passed and responses returned. This supports tools that are OpenAI-hosted, MCP, and non-hosted. Read more in our guides below.

OpenAI Developers

65,080 views • 1 year ago

Hosted evals are finally live!! Smal vid showing how to use them, more to come : )
3:59

Hosted evals are finally live!! Smal vid showing how to use them, more to come : )

Florian Brand

32,593 views • 29 days ago

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!
1:15

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 views • 1 year ago

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇
0:25

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇

Marco Mascorro

44,302 views • 17 days ago

Docker is excited to partner with E2B, a platform for running AI-generated code in secure cloud sandboxes. Docker + E2B now gives AI agents access to real tools (GitHub, Perplexity, Browserbase) in secure sandboxes via the MCP Catalog.
2:38

Docker is excited to partner with E2B, a platform for running AI-generated code in secure cloud sandboxes. Docker + E2B now gives AI agents access to real tools (GitHub, Perplexity, Browserbase) in secure sandboxes via the MCP Catalog.

Docker

45,583 views • 8 months ago

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.
1:01

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.

Chrissy Houlahan

623,845 views • 2 years ago

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay
1:06

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay

runDisney

295,035 views • 3 years ago