Загрузка видео...

Не удалось загрузить видео

На главную

Today, we are launching Hosted Evaluations on the platform. Running evals is an infra problem: harnesses, sandboxes, hours of compute, hundreds of parallel runs. Running evals is hard. Until now.

PrimeIntellect's profile picture

Prime Intellect

71,266 subscribers

125,935 просмотров • 2 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:
0:53

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:

Awni Hannun

132,388 просмотров • 1 год назад

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals
1:18

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals

Haider.

45,058 просмотров • 9 месяцев назад

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,656 просмотров • 3 месяцев назад

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 просмотров • 10 месяцев назад

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.
0:35

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.

Modal

32,334 просмотров • 5 месяцев назад

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna
1:03:59

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

247,328 просмотров • 5 месяцев назад

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 просмотров • 4 месяцев назад

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.
0:52

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.

Bel-Air on Peacock

14,334 просмотров • 8 месяцев назад

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇
0:25

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇

Marco Mascorro

44,641 просмотров • 1 месяц назад

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.
1:01

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.

Chrissy Houlahan

623,911 просмотров • 2 лет назад

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay
1:06

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay

runDisney

295,035 просмотров • 3 лет назад

🔥 Launching the Fireworks AI Experimentation Platform (GA) and Build SDK (Beta)🔥 Top AI teams don't win by chance - they win by running 10X more experiments than competitors. Launching early, collecting feedback, tuning and selecting models, running offline evals, and validating impact with product metrics in a fast, continuous cycle is critical to build successful AI products. But, the need to manage complex infrastructure, wait for GPUs, reconcile differences between training and serving, and maintain reliability, slow AI teams down. Now, with the Firework’s experimentation platform, experimenting with open models just got a whole lot easier. With the help of our brand-new Python SDK (in beta), you can: → Stop waiting for GPUs, fine-tune and deploy 1000s of models on a single GPU deployment using Multi-LoRA → Programmatically run large-scale fine-tuning experiments → Scale evals with fully managed and autoscaling GPU deployments Check out our blog post to learn more and see a demo of running 125 experiments in 3 minutes on a single deployment:
4:41

🔥 Launching the Fireworks AI Experimentation Platform (GA) and Build SDK (Beta)🔥 Top AI teams don't win by chance - they win by running 10X more experiments than competitors. Launching early, collecting feedback, tuning and selecting models, running offline evals, and validating impact with product metrics in a fast, continuous cycle is critical to build successful AI products. But, the need to manage complex infrastructure, wait for GPUs, reconcile differences between training and serving, and maintain reliability, slow AI teams down. Now, with the Firework’s experimentation platform, experimenting with open models just got a whole lot easier. With the help of our brand-new Python SDK (in beta), you can: → Stop waiting for GPUs, fine-tune and deploy 1000s of models on a single GPU deployment using Multi-LoRA → Programmatically run large-scale fine-tuning experiments → Scale evals with fully managed and autoscaling GPU deployments Check out our blog post to learn more and see a demo of running 125 experiments in 3 minutes on a single deployment:

Fireworks AI

17,619 просмотров • 1 год назад

In the latest episode of Fortune's #TermSheet, Whatnot CEO and cofounder Grant LaFontaine told Fortune that he and his cofounders sold hundreds of Funko Pops within the first hours of launching. Today, the live shopping platform is valued at $11.5 billion.
0:55

In the latest episode of Fortune's #TermSheet, Whatnot CEO and cofounder Grant LaFontaine told Fortune that he and his cofounders sold hundreds of Funko Pops within the first hours of launching. Today, the live shopping platform is valued at $11.5 billion.

FORTUNE

16,977 просмотров • 8 дней назад

today i'm launching an open source RAG platform built on top of turbopuffer fast ingestion, fast retrieval, clean developer experience. one of the few AI infra products that genuinely feels modern. more soon 👀
0:59

today i'm launching an open source RAG platform built on top of turbopuffer fast ingestion, fast retrieval, clean developer experience. one of the few AI infra products that genuinely feels modern. more soon 👀

Yoginth

45,910 просмотров • 2 месяцев назад

“The fact that people are running away because of money is a problem to me” - Dr. Antoinette
0:34

“The fact that people are running away because of money is a problem to me” - Dr. Antoinette

@𝗼𝗻𝗲𝗷𝗼𝗯𝗹𝗲𝘀𝘀𝗯𝗼𝘆

1,315,585 просмотров • 2 лет назад

The list of all the problems in mathematics that Terence Tao *could* solve is absolutely enormous. The problem is we can't make copies of him, dump millions of dollars of inference compute into each copy, and have them each tackle a different Millennium Prize Problem for a hundred subjective years. But we can do that with AI. So in that way, we may be underrating how significant it will be when we have AIs that are not superintelligent but rather simply as good as the best humans today. It’s actually an extremely powerful property of AI that by just running copies in parallel, you can immediately fill in the waterline of accomplishments that are feasible with a given level of capabilities.
2:53

The list of all the problems in mathematics that Terence Tao *could* solve is absolutely enormous. The problem is we can't make copies of him, dump millions of dollars of inference compute into each copy, and have them each tackle a different Millennium Prize Problem for a hundred subjective years. But we can do that with AI. So in that way, we may be underrating how significant it will be when we have AIs that are not superintelligent but rather simply as good as the best humans today. It’s actually an extremely powerful property of AI that by just running copies in parallel, you can immediately fill in the waterline of accomplishments that are feasible with a given level of capabilities.

Dwarkesh Patel

99,378 просмотров • 4 месяцев назад

Are we really at the End? Is Jesus coming soon? We are running out of time.
1:09

Are we really at the End? Is Jesus coming soon? We are running out of time.

Brandon Biggs

34,220 просмотров • 1 год назад

We also shared evals on Open AI o3-mini — a faster, distilled version of o3 which is optimized for coding, and the first version of o3 we expect to make available for use in early 2025.
4:29

We also shared evals on Open AI o3-mini — a faster, distilled version of o3 which is optimized for coding, and the first version of o3 we expect to make available for use in early 2025.

OpenAI

340,205 просмотров • 1 год назад

Edwin Sifuna on whether he is considering running for president: As Linda Mwananchi, we are very strongly considering the prospect of running a presidential candidate. That is clear; it is coming from the people themselves. If you are the one who runs around saying Ruto must go, you must be available to go him. We are hoping that once we complete the tour of the country, we should be able to give you a straight answer #CitizenExplainer
2:35

Edwin Sifuna on whether he is considering running for president: As Linda Mwananchi, we are very strongly considering the prospect of running a presidential candidate. That is clear; it is coming from the people themselves. If you are the one who runs around saying Ruto must go, you must be available to go him. We are hoping that once we complete the tour of the country, we should be able to give you a straight answer #CitizenExplainer

Citizen TV Kenya

18,645 просмотров • 12 дней назад

The United States military’s blockade of the Strait of Hormuz is costing Iran hundreds of millions of dollars every day. Iran’s economy is on the ropes, and the regime is running out of time.
0:54

The United States military’s blockade of the Strait of Hormuz is costing Iran hundreds of millions of dollars every day. Iran’s economy is on the ropes, and the regime is running out of time.

Tom Cotton

21,202 просмотров • 3 месяцев назад