正在加载视频...

视频加载失败

返回首页

Today, we are launching Hosted Evaluations on the platform. Running evals is an infra problem: harnesses, sandboxes, hours of compute, hundreds of parallel runs. Running evals is hard. Until now.

PrimeIntellect's profile picture

Prime Intellect

71,266 subscribers

125,935 次观看 • 2 个月前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:
0:53

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:

Awni Hannun

132,923 次观看 • 1 年前

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals
1:18

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals

Haider.

45,058 次观看 • 9 个月前

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,656 次观看 • 4 个月前

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.
0:35

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.

Modal

32,567 次观看 • 6 个月前

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 次观看 • 11 个月前

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna
1:03:59

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

247,328 次观看 • 6 个月前

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 次观看 • 5 个月前

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.
0:52

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.

Bel-Air on Peacock

14,334 次观看 • 8 个月前

Compute at Sea Francois Chaubard AI is running out of compute, and data centers are running out of electricity and land. One possible solution is to move compute offshore. The ocean offers abundant space, sunlight, and natural cooling, opening the door to fleets of modular data centers operating together as a global cloud.
0:47

Compute at Sea Francois Chaubard AI is running out of compute, and data centers are running out of electricity and land. One possible solution is to move compute offshore. The ocean offers abundant space, sunlight, and natural cooling, opening the door to fleets of modular data centers operating together as a global cloud.

Y Combinator

64,621 次观看 • 1 个月前

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇
0:25

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇

Marco Mascorro

44,641 次观看 • 2 个月前

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.
1:01

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.

Chrissy Houlahan

623,911 次观看 • 2 年前

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay
1:06

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay

runDisney

295,035 次观看 • 3 年前

Today we're open-sourcing nac, an agent harness for long-running tasks, and launching the Arcee open models API beta. Nac is available now under Apache 2.0 on GitHub, built for developers running complex, multi-step engineering workloads.
0:58

Today we're open-sourcing nac, an agent harness for long-running tasks, and launching the Arcee open models API beta. Nac is available now under Apache 2.0 on GitHub, built for developers running complex, multi-step engineering workloads.

Arcee.ai

1,055,169 次观看 • 8 天前

In the latest episode of Fortune's #TermSheet, Whatnot CEO and cofounder Grant LaFontaine told Fortune that he and his cofounders sold hundreds of Funko Pops within the first hours of launching. Today, the live shopping platform is valued at $11.5 billion.
0:55

In the latest episode of Fortune's #TermSheet, Whatnot CEO and cofounder Grant LaFontaine told Fortune that he and his cofounders sold hundreds of Funko Pops within the first hours of launching. Today, the live shopping platform is valued at $11.5 billion.

FORTUNE

16,977 次观看 • 19 天前

today i'm launching an open source RAG platform built on top of turbopuffer fast ingestion, fast retrieval, clean developer experience. one of the few AI infra products that genuinely feels modern. more soon 👀
0:59

today i'm launching an open source RAG platform built on top of turbopuffer fast ingestion, fast retrieval, clean developer experience. one of the few AI infra products that genuinely feels modern. more soon 👀

Yoginth

45,910 次观看 • 3 个月前

“The fact that people are running away because of money is a problem to me” - Dr. Antoinette
0:34

“The fact that people are running away because of money is a problem to me” - Dr. Antoinette

@𝗼𝗻𝗲𝗷𝗼𝗯𝗹𝗲𝘀𝘀𝗯𝗼𝘆

1,315,627 次观看 • 2 年前

The list of all the problems in mathematics that Terence Tao *could* solve is absolutely enormous. The problem is we can't make copies of him, dump millions of dollars of inference compute into each copy, and have them each tackle a different Millennium Prize Problem for a hundred subjective years. But we can do that with AI. So in that way, we may be underrating how significant it will be when we have AIs that are not superintelligent but rather simply as good as the best humans today. It’s actually an extremely powerful property of AI that by just running copies in parallel, you can immediately fill in the waterline of accomplishments that are feasible with a given level of capabilities.
2:53

The list of all the problems in mathematics that Terence Tao *could* solve is absolutely enormous. The problem is we can't make copies of him, dump millions of dollars of inference compute into each copy, and have them each tackle a different Millennium Prize Problem for a hundred subjective years. But we can do that with AI. So in that way, we may be underrating how significant it will be when we have AIs that are not superintelligent but rather simply as good as the best humans today. It’s actually an extremely powerful property of AI that by just running copies in parallel, you can immediately fill in the waterline of accomplishments that are feasible with a given level of capabilities.

Dwarkesh Patel

99,378 次观看 • 4 个月前

Are we really at the End? Is Jesus coming soon? We are running out of time.
1:09

Are we really at the End? Is Jesus coming soon? We are running out of time.

Brandon Biggs

34,220 次观看 • 1 年前

We also shared evals on Open AI o3-mini — a faster, distilled version of o3 which is optimized for coding, and the first version of o3 we expect to make available for use in early 2025.
4:29

We also shared evals on Open AI o3-mini — a faster, distilled version of o3 which is optimized for coding, and the first version of o3 we expect to make available for use in early 2025.

OpenAI

340,205 次观看 • 1 年前