Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

DSPy is great at classification tasks, but it's hard to make a formal eval metric for 'fuzzy' creative tasks like writing a blog post or telling a joke. Solution: train a judge with DSPy to agree with you 90%+, then use that judge as the eval metric for the... show more

Mike Taylor

9,523 subscribers

18,654 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:

Andrew Ng

181,457 views • 1 year ago

Glad that you guys both agree with me. When someone can't identify things that seem obvious, it's make you go hmmmm... seems like a good metric to use when "judging" objectively. 🤷‍♂️ Proper Doinks

Glad that you guys both agree with me. When someone can't identify things that seem obvious, it's make you go hmmmm... seems like a good metric to use when "judging" objectively. 🤷‍♂️ Proper Doinks

Archive Seed Bank

12,562 views • 4 months ago

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

Hao AI Lab

69,004 views • 1 year ago

You're so good at drunk dancing that you accidentally confess This officer needed proof for the judge that her suspect was intoxicated, so she got creative with the sobriety test. First, a catwalk. Then she asked him to dance. And this man DELIVERED like he was auditioning for a reality show.

You're so good at drunk dancing that you accidentally confess This officer needed proof for the judge that her suspect was intoxicated, so she got creative with the sobriety test. First, a catwalk. Then she asked him to dance. And this man DELIVERED like he was auditioning for a reality show.

Brown World

46,419 views • 5 months ago

Okay, folks, insane things are happening. Imagine if, instead of writing your DSPy programs, you could just specify what you want and get a custom BIG BEAUTIFUL DSPY PROGRAM generated for you ? That's now possible thanks to my new project: METADSPY. Link to github in second post. 1/

Okay, folks, insane things are happening. Imagine if, instead of writing your DSPy programs, you could just specify what you want and get a custom BIG BEAUTIFUL DSPY PROGRAM generated for you ? That's now possible thanks to my new project: METADSPY. Link to github in second post. 1/

Franck SN

56,734 views • 1 year ago

LLMs are great for human in the loop applications, but fail at deterministic developer tasks. Interfaze (YC P26) is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more. Congrats on the launch, Yoeven and Harsha!

LLMs are great for human in the loop applications, but fail at deterministic developer tasks. Interfaze (YC P26) is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more. Congrats on the launch, Yoeven and Harsha!

Y Combinator

69,326 views • 2 months ago

I get LOTS of questions about deploying DSPy programs, so isaac 🧩 and I built dspy-cli: a tool that serves DSPy programs as HTTP APIs with Docker config, OpenAPI specs, MCP support, and more. Here's a quick intro:

I get LOTS of questions about deploying DSPy programs, so isaac 🧩 and I built dspy-cli: a tool that serves DSPy programs as HTTP APIs with Docker config, OpenAPI specs, MCP support, and more. Here's a quick intro:

Drew Breunig

18,676 views • 7 months ago

I always preach for independent wrestlers to set up merch tables at shows… But what should they sell? EVERYTHING! Be creative. Make merch that you think your fans would want. Make different merch that you sell at various prices. The merch table is a great way to make more money. But more importantly, it’s a great way to make a connection with fans.

I always preach for independent wrestlers to set up merch tables at shows… But what should they sell? EVERYTHING! Be creative. Make merch that you think your fans would want. Make different merch that you sell at various prices. The merch table is a great way to make more money. But more importantly, it’s a great way to make a connection with fans.

Matt Cardona

15,667 views • 11 months ago

Can we get robots to improve at long-horizon tasks without supervision? Our latest work tackles this problem by planning to practice! Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)

Can we get robots to improve at long-horizon tasks without supervision? Our latest work tackles this problem by planning to practice! Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)

Nishanth Kumar

18,704 views • 2 years ago

OH. MY. GOSH!!! The Charlotte NC judge who gave a $5k bond to a woman for assaulting her boyfriend who then ST*BBED HIM TO DE*TH 1 MONTH LATER... ...IS A DEMOCRAT JUDGE WITH PRIOR CHARGES HIMSELF FOR FALSIFYING LEGAL DOCUMENTS!!!! He served only 90 days of unsupervised probation and KEPT HIS JOB AS A JUDGE!!! This. Is. Insanity.

OH. MY. GOSH!!! The Charlotte NC judge who gave a $5k bond to a woman for assaulting her boyfriend who then STBBED HIM TO DETH 1 MONTH LATER... ...IS A DEMOCRAT JUDGE WITH PRIOR CHARGES HIMSELF FOR FALSIFYING LEGAL DOCUMENTS!!!! He served only 90 days of unsupervised probation and KEPT HIS JOB AS A JUDGE!!! This. Is. Insanity.

Matt Van Swol

123,577 views • 2 months ago

Ever Heard of a Metric in Differential Geometry? How About a Geodesic? These two ideas are central to how Mathematics and Physics describe Spacetime. A metric is how a surface measures distance and angle. A geodesic is the path that counts as "straight" after that measuring rule is chosen. Once the metric changes from point to point, straightness is no longer something you judge by eye. On a curved surface, the shortest paths are geodesics. They are chosen by the metric. Take a surface written as r(x,y) = (x, y, f(x,y)) Its metric is gᵢⱼ = ∂ᵢr · ∂ⱼr A geodesic q(t) = (x(t), y(t)) obeys q̈ᵏ + Γᵏᵢⱼ q̇ᶦq̇ʲ = 0 where Γᵏᵢⱼ are the Christoffel symbols of the surface metric. Launch many almost-parallel paths. At the start, they look like a clean beam until curvature takes over. Some paths focus, others separate and some slide around ridges. The surface is essentially the rulebook for motion. The shortest path is local. The geometry remembers everything. #DifferentialGeometry #Geodesics #RiemannianGeometry #MathematicalPhysics #Curvature #TensorCalculus #AppliedMathematics #Geometry #ScienceAnimation

Ever Heard of a Metric in Differential Geometry? How About a Geodesic? These two ideas are central to how Mathematics and Physics describe Spacetime. A metric is how a surface measures distance and angle. A geodesic is the path that counts as "straight" after that measuring rule is chosen. Once the metric changes from point to point, straightness is no longer something you judge by eye. On a curved surface, the shortest paths are geodesics. They are chosen by the metric. Take a surface written as r(x,y) = (x, y, f(x,y)) Its metric is gᵢⱼ = ∂ᵢr · ∂ⱼr A geodesic q(t) = (x(t), y(t)) obeys q̈ᵏ + Γᵏᵢⱼ q̇ᶦq̇ʲ = 0 where Γᵏᵢⱼ are the Christoffel symbols of the surface metric. Launch many almost-parallel paths. At the start, they look like a clean beam until curvature takes over. Some paths focus, others separate and some slide around ridges. The surface is essentially the rulebook for motion. The shortest path is local. The geometry remembers everything. #DifferentialGeometry #Geodesics #RiemannianGeometry #MathematicalPhysics #Curvature #TensorCalculus #AppliedMathematics #Geometry #ScienceAnimation

Mathelirium

22,170 views • 6 days ago

Teachers can use Canva with great ease, To make poetry lessons a delightful breeze. With templates for poems and creative art, Students' creativity can shine from the start. 🎤💥

Teachers can use Canva with great ease, To make poetry lessons a delightful breeze. With templates for poems and creative art, Students' creativity can shine from the start. 🎤💥

Canva Education

19,336 views • 3 years ago

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

Santiago

36,273 views • 1 year ago

Made myself a no code interface for optimizing prompts with DSPy. Thought I'd open source it.

Made myself a no code interface for optimizing prompts with DSPy. Thought I'd open source it.

Mike Taylor

73,097 views • 1 year ago

Judge President Selby Mbenenge says it’s hard to accept that a judge shouldn’t pursue a romantic relationship with someone of a different class, age, or job status at work.

Judge President Selby Mbenenge says it’s hard to accept that a judge shouldn’t pursue a romantic relationship with someone of a different class, age, or job status at work.

MDN NEWS

25,033 views • 11 months ago

.Dave Rubin: "How do you sort of quantify or analyze the behavior around [the pressures to make content only for clicks?]" Gad Saad: "I think the problem with Candace and a lot of those folks, is that the metric that they're trying to optimize is, as you've guys have said, clickbait."

.Dave Rubin: "How do you sort of quantify or analyze the behavior around [the pressures to make content only for clicks?]" Gad Saad: "I think the problem with Candace and a lot of those folks, is that the metric that they're trying to optimize is, as you've guys have said, clickbait."

The Rubin Report

23,589 views • 6 months ago

🚨 ELON MUSK: "It's not like, I agree with everything the administration does. I agree with much of what the administration does, but we have differences of opinion, you know, the things that I don't entirely agree with. But it's difficult for me to bring that up in an interview, because then it creates a bone of contention. So then I'm in a little stuck, in a bind where I'm like, Well, I don't want to, you know, speak up against the administration, but I don't want to, you also don't want to take responsibility for everything, the administration is doing."

🚨 ELON MUSK: "It's not like, I agree with everything the administration does. I agree with much of what the administration does, but we have differences of opinion, you know, the things that I don't entirely agree with. But it's difficult for me to bring that up in an interview, because then it creates a bone of contention. So then I'm in a little stuck, in a bind where I'm like, Well, I don't want to, you know, speak up against the administration, but I don't want to, you also don't want to take responsibility for everything, the administration is doing."

DogeDesigner

98,616 views • 1 year ago

🚀 Introducing Queue: A Smarter Way to Build with Replit Agent Today, we’re excited to launch Queue, a powerful new capability designed to enhance your experience with Replit Agent. With Queue, you can submit multiple tasks while your Agent is still working—no more waiting around or breaking your flow. As each task completes, the Agent automatically moves to the next. You can: ✅ Add multiple tasks for the Agent to process in order. ✅ Edit, reorder, or reprioritize messages on the fly. ✅ Attach files and adjust settings—including toggling extended thinking or selecting a different model—per task.

🚀 Introducing Queue: A Smarter Way to Build with Replit Agent Today, we’re excited to launch Queue, a powerful new capability designed to enhance your experience with Replit Agent. With Queue, you can submit multiple tasks while your Agent is still working—no more waiting around or breaking your flow. As each task completes, the Agent automatically moves to the next. You can: ✅ Add multiple tasks for the Agent to process in order. ✅ Edit, reorder, or reprioritize messages on the fly. ✅ Attach files and adjust settings—including toggling extended thinking or selecting a different model—per task.

Replit ⠕

75,576 views • 11 months ago