Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Today, we are launching Hosted Evaluations on the platform. Running evals is an infra problem: harnesses, sandboxes, hours of compute, hundreds of parallel runs. Running evals is hard. Until now.

Prime Intellect

71,266 subscribers

125,935 Aufrufe • vor 1 Monat •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:

QwQ-32B evals on par with Deep Seek R1 680B but runs fast on a laptop. Delivery accepted. Here it is running nicely on a M4 Max with MLX. A snippet of its 8k token long thought process:

Awni Hannun

132,388 Aufrufe • vor 1 Jahr

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals

Microsoft AI CEO, Mustafa Suleyman: We can already imagine a time, just a few years ahead, when AI systems are trained on gigawatt-scale compute runs that are capable of self-improvement, setting goals, managing resources, and writing their own evals

Haider.

45,058 Aufrufe • vor 8 Monaten

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,653 Aufrufe • vor 3 Monaten

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 Aufrufe • vor 9 Monaten

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.

We sat down with Ramp to talk about their internal coding agent, Ramp Inspect, that now writes over half of their merged PRs. - Runs on Modal Sandboxes. - Full dev environments spin up in seconds. - Hundreds of sessions in parallel.

Modal

32,334 Aufrufe • vor 5 Monaten

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

247,208 Aufrufe • vor 5 Monaten

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 Aufrufe • vor 4 Monaten

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.

Time is running out. PSA this is a spoiler free zone until Tuesday 🤫 All episodes of the final season of #BelAir are streaming now on Peacock.

Bel-Air on Peacock

14,334 Aufrufe • vor 7 Monaten

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇

After coding is solved, the next frontier is computer use. Today, we are launching Use Computer, the infra for evaluating and training models to use all kinds of computers 👇

Marco Mascorro

44,641 Aufrufe • vor 1 Monat

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.

In 24 hours, the House will adjourn yet again without having voted on aid to Ukraine. It's insanity — Ukraine is literally running out of bullets and Republicans are running out of Washington. Respectfully, Speaker Johnson, the time is now to vote for Ukraine aid.

Chrissy Houlahan

623,845 Aufrufe • vor 2 Jahren

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay

Because every mile is truly MAGIC with the #runDisney running community! ✨ 👟 Today, on Global Running Day, we celebrate YOU! Thanks for being a part of the running magic! #GlobalRunningDay

runDisney

295,035 Aufrufe • vor 3 Jahren

🔥 Launching the Fireworks AI Experimentation Platform (GA) and Build SDK (Beta)🔥 Top AI teams don't win by chance - they win by running 10X more experiments than competitors. Launching early, collecting feedback, tuning and selecting models, running offline evals, and validating impact with product metrics in a fast, continuous cycle is critical to build successful AI products. But, the need to manage complex infrastructure, wait for GPUs, reconcile differences between training and serving, and maintain reliability, slow AI teams down. Now, with the Firework’s experimentation platform, experimenting with open models just got a whole lot easier. With the help of our brand-new Python SDK (in beta), you can: → Stop waiting for GPUs, fine-tune and deploy 1000s of models on a single GPU deployment using Multi-LoRA → Programmatically run large-scale fine-tuning experiments → Scale evals with fully managed and autoscaling GPU deployments Check out our blog post to learn more and see a demo of running 125 experiments in 3 minutes on a single deployment:

Fireworks AI

17,619 Aufrufe • vor 1 Jahr

“The fact that people are running away because of money is a problem to me” - Dr. Antoinette

“The fact that people are running away because of money is a problem to me” - Dr. Antoinette

@𝗼𝗻𝗲𝗷𝗼𝗯𝗹𝗲𝘀𝘀𝗯𝗼𝘆

1,315,534 Aufrufe • vor 2 Jahren

The list of all the problems in mathematics that Terence Tao *could* solve is absolutely enormous. The problem is we can't make copies of him, dump millions of dollars of inference compute into each copy, and have them each tackle a different Millennium Prize Problem for a hundred subjective years. But we can do that with AI. So in that way, we may be underrating how significant it will be when we have AIs that are not superintelligent but rather simply as good as the best humans today. It’s actually an extremely powerful property of AI that by just running copies in parallel, you can immediately fill in the waterline of accomplishments that are feasible with a given level of capabilities.

The list of all the problems in mathematics that Terence Tao could solve is absolutely enormous. The problem is we can't make copies of him, dump millions of dollars of inference compute into each copy, and have them each tackle a different Millennium Prize Problem for a hundred subjective years. But we can do that with AI. So in that way, we may be underrating how significant it will be when we have AIs that are not superintelligent but rather simply as good as the best humans today. It’s actually an extremely powerful property of AI that by just running copies in parallel, you can immediately fill in the waterline of accomplishments that are feasible with a given level of capabilities.

Dwarkesh Patel

99,378 Aufrufe • vor 3 Monaten

Are we really at the End? Is Jesus coming soon? We are running out of time.

Are we really at the End? Is Jesus coming soon? We are running out of time.

Brandon Biggs

34,220 Aufrufe • vor 1 Jahr

We also shared evals on Open AI o3-mini — a faster, distilled version of o3 which is optimized for coding, and the first version of o3 we expect to make available for use in early 2025.

We also shared evals on Open AI o3-mini — a faster, distilled version of o3 which is optimized for coding, and the first version of o3 we expect to make available for use in early 2025.

OpenAI

340,099 Aufrufe • vor 1 Jahr

The United States military’s blockade of the Strait of Hormuz is costing Iran hundreds of millions of dollars every day. Iran’s economy is on the ropes, and the regime is running out of time.

The United States military’s blockade of the Strait of Hormuz is costing Iran hundreds of millions of dollars every day. Iran’s economy is on the ropes, and the regime is running out of time.

Tom Cotton

21,202 Aufrufe • vor 3 Monaten

The paddock is full of jobs. The problem is finding them. Until now. is now live, the first hiring platform built specifically for the motorcycle paddock. Create your profile today and find your next role in the paddock 🚀

Taylor Mackenzie

53,734 Aufrufe • vor 7 Monaten

Today on Training Data: Sanjit Biswas, founder & CEO of Samsara (NYSE:IOT) and former Sequoia Capital backed founder of Cisco Meraki Sanjit shares the ups & downs of running neural nets on constrained compute and power footprints in the real world, ~2-10 watts Physical AI is hard 🫡

Today on Training Data: Sanjit Biswas, founder & CEO of Samsara (NYSE:IOT) and former Sequoia Capital backed founder of Cisco Meraki Sanjit shares the ups & downs of running neural nets on constrained compute and power footprints in the real world, ~2-10 watts Physical AI is hard 🫡

Sonya Huang 🐥

35,714 Aufrufe • vor 7 Monaten

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:

LangChain

13,322 Aufrufe • vor 9 Monaten