Загрузка видео...

Не удалось загрузить видео

На главную

DSPy is great at classification tasks, but it's hard to make a formal eval metric for 'fuzzy' creative tasks like writing a blog post or telling a joke. Solution: train a judge with DSPy to agree with you 90%+, then use that judge as the eval metric for the...

hammer_mt's profile picture

Mike Taylor

9,523 subscribers

18,654 просмотров • 10 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:
3:27

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:

Andrew Ng

181,457 просмотров • 1 год назад

Glad that you guys both agree with me. When someone can't identify things that seem obvious, it's make you go hmmmm... seems like a good metric to use when "judging" objectively. 🤷‍♂️ Proper Doinks
0:58

Glad that you guys both agree with me. When someone can't identify things that seem obvious, it's make you go hmmmm... seems like a good metric to use when "judging" objectively. 🤷‍♂️ Proper Doinks

Archive Seed Bank

12,562 просмотров • 5 месяцев назад

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:
0:41

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

Hao AI Lab

69,037 просмотров • 1 год назад

You're so good at drunk dancing that you accidentally confess This officer needed proof for the judge that her suspect was intoxicated, so she got creative with the sobriety test. First, a catwalk. Then she asked him to dance. And this man DELIVERED like he was auditioning for a reality show.
0:52

You're so good at drunk dancing that you accidentally confess This officer needed proof for the judge that her suspect was intoxicated, so she got creative with the sobriety test. First, a catwalk. Then she asked him to dance. And this man DELIVERED like he was auditioning for a reality show.

Brown World

46,630 просмотров • 6 месяцев назад

LLMs are great for human in the loop applications, but fail at deterministic developer tasks. Interfaze (YC P26) is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more. Congrats on the launch, Yoeven and Harsha!
1:21

LLMs are great for human in the loop applications, but fail at deterministic developer tasks. Interfaze (YC P26) is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more. Congrats on the launch, Yoeven and Harsha!

Y Combinator

71,358 просмотров • 3 месяцев назад

I always preach for independent wrestlers to set up merch tables at shows… But what should they sell? EVERYTHING! Be creative. Make merch that you think your fans would want. Make different merch that you sell at various prices. The merch table is a great way to make more money. But more importantly, it’s a great way to make a connection with fans.
0:12

I always preach for independent wrestlers to set up merch tables at shows… But what should they sell? EVERYTHING! Be creative. Make merch that you think your fans would want. Make different merch that you sell at various prices. The merch table is a great way to make more money. But more importantly, it’s a great way to make a connection with fans.

Matt Cardona

15,667 просмотров • 1 год назад

OH. MY. GOSH!!! The Charlotte NC judge who gave a $5k bond to a woman for assaulting her boyfriend who then ST*BBED HIM TO DE*TH 1 MONTH LATER... ...IS A DEMOCRAT JUDGE WITH PRIOR CHARGES HIMSELF FOR FALSIFYING LEGAL DOCUMENTS!!!! He served only 90 days of unsupervised probation and KEPT HIS JOB AS A JUDGE!!! This. Is. Insanity.
0:57

OH. MY. GOSH!!! The Charlotte NC judge who gave a $5k bond to a woman for assaulting her boyfriend who then ST*BBED HIM TO DE*TH 1 MONTH LATER... ...IS A DEMOCRAT JUDGE WITH PRIOR CHARGES HIMSELF FOR FALSIFYING LEGAL DOCUMENTS!!!! He served only 90 days of unsupervised probation and KEPT HIS JOB AS A JUDGE!!! This. Is. Insanity.

Matt Van Swol

123,600 просмотров • 3 месяцев назад

Ever Heard of a Metric in Differential Geometry? How About a Geodesic? These two ideas are central to how Mathematics and Physics describe Spacetime. A metric is how a surface measures distance and angle. A geodesic is the path that counts as "straight" after that measuring rule is chosen. Once the metric changes from point to point, straightness is no longer something you judge by eye. On a curved surface, the shortest paths are geodesics. They are chosen by the metric. Take a surface written as r(x,y) = (x, y, f(x,y)) Its metric is gᵢⱼ = ∂ᵢr · ∂ⱼr A geodesic q(t) = (x(t), y(t)) obeys q̈ᵏ + Γᵏᵢⱼ q̇ᶦq̇ʲ = 0 where Γᵏᵢⱼ are the Christoffel symbols of the surface metric. Launch many almost-parallel paths. At the start, they look like a clean beam until curvature takes over. Some paths focus, others separate and some slide around ridges. The surface is essentially the rulebook for motion. The shortest path is local. The geometry remembers everything. #DifferentialGeometry #Geodesics #RiemannianGeometry #MathematicalPhysics #Curvature #TensorCalculus #AppliedMathematics #Geometry #ScienceAnimation
0:33

Ever Heard of a Metric in Differential Geometry? How About a Geodesic? These two ideas are central to how Mathematics and Physics describe Spacetime. A metric is how a surface measures distance and angle. A geodesic is the path that counts as "straight" after that measuring rule is chosen. Once the metric changes from point to point, straightness is no longer something you judge by eye. On a curved surface, the shortest paths are geodesics. They are chosen by the metric. Take a surface written as r(x,y) = (x, y, f(x,y)) Its metric is gᵢⱼ = ∂ᵢr · ∂ⱼr A geodesic q(t) = (x(t), y(t)) obeys q̈ᵏ + Γᵏᵢⱼ q̇ᶦq̇ʲ = 0 where Γᵏᵢⱼ are the Christoffel symbols of the surface metric. Launch many almost-parallel paths. At the start, they look like a clean beam until curvature takes over. Some paths focus, others separate and some slide around ridges. The surface is essentially the rulebook for motion. The shortest path is local. The geometry remembers everything. #DifferentialGeometry #Geodesics #RiemannianGeometry #MathematicalPhysics #Curvature #TensorCalculus #AppliedMathematics #Geometry #ScienceAnimation

Mathelirium

22,170 просмотров • 1 месяц назад

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:
6:33

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

Santiago

36,273 просмотров • 1 год назад

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang
2:41

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MTS

100,425 просмотров • 17 дней назад

Teachers can use Canva with great ease, To make poetry lessons a delightful breeze. With templates for poems and creative art, Students' creativity can shine from the start. 🎤💥
0:21

Teachers can use Canva with great ease, To make poetry lessons a delightful breeze. With templates for poems and creative art, Students' creativity can shine from the start. 🎤💥

Canva Education

19,336 просмотров • 3 лет назад

.Dave Rubin: "How do you sort of quantify or analyze the behavior around [the pressures to make content only for clicks?]" Gad Saad: "I think the problem with Candace and a lot of those folks, is that the metric that they're trying to optimize is, as you've guys have said, clickbait."
2:17

.Dave Rubin: "How do you sort of quantify or analyze the behavior around [the pressures to make content only for clicks?]" Gad Saad: "I think the problem with Candace and a lot of those folks, is that the metric that they're trying to optimize is, as you've guys have said, clickbait."

The Rubin Report

23,589 просмотров • 7 месяцев назад

On a previous #TeawithGaryVee I was asked "What is more important to you as a leader? Challenging your employees with new tasks to grow or making sure they to give them tasks that they excel at?" if you have a question for me, drop it here so my team can add it to the next Rapid Fire Q&A and follow me on
1:01

On a previous #TeawithGaryVee I was asked "What is more important to you as a leader? Challenging your employees with new tasks to grow or making sure they to give them tasks that they excel at?" if you have a question for me, drop it here so my team can add it to the next Rapid Fire Q&A and follow me on

Gary Vaynerchuk

22,719 просмотров • 28 дней назад

🚨 ELON MUSK: "It's not like, I agree with everything the administration does. I agree with much of what the administration does, but we have differences of opinion, you know, the things that I don't entirely agree with. But it's difficult for me to bring that up in an interview, because then it creates a bone of contention. So then I'm in a little stuck, in a bind where I'm like, Well, I don't want to, you know, speak up against the administration, but I don't want to, you also don't want to take responsibility for everything, the administration is doing."
0:38

🚨 ELON MUSK: "It's not like, I agree with everything the administration does. I agree with much of what the administration does, but we have differences of opinion, you know, the things that I don't entirely agree with. But it's difficult for me to bring that up in an interview, because then it creates a bone of contention. So then I'm in a little stuck, in a bind where I'm like, Well, I don't want to, you know, speak up against the administration, but I don't want to, you also don't want to take responsibility for everything, the administration is doing."

DogeDesigner

98,616 просмотров • 1 год назад

🚀 Introducing Queue: A Smarter Way to Build with Replit Agent Today, we’re excited to launch Queue, a powerful new capability designed to enhance your experience with Replit Agent. With Queue, you can submit multiple tasks while your Agent is still working—no more waiting around or breaking your flow. As each task completes, the Agent automatically moves to the next. You can: ✅ Add multiple tasks for the Agent to process in order. ✅ Edit, reorder, or reprioritize messages on the fly. ✅ Attach files and adjust settings—including toggling extended thinking or selecting a different model—per task.
0:27

🚀 Introducing Queue: A Smarter Way to Build with Replit Agent Today, we’re excited to launch Queue, a powerful new capability designed to enhance your experience with Replit Agent. With Queue, you can submit multiple tasks while your Agent is still working—no more waiting around or breaking your flow. As each task completes, the Agent automatically moves to the next. You can: ✅ Add multiple tasks for the Agent to process in order. ✅ Edit, reorder, or reprioritize messages on the fly. ✅ Attach files and adjust settings—including toggling extended thinking or selecting a different model—per task.

Replit ⠕

75,576 просмотров • 1 год назад

Aaron Judge was hit coming off the field with a throw from Anthony Volpe. It is normal for a ball to be thrown to Judge as he comes off the field
0:27

Aaron Judge was hit coming off the field with a throw from Anthony Volpe. It is normal for a ball to be thrown to Judge as he comes off the field

Talkin' Yanks

693,242 просмотров • 1 год назад

Elon Musk: “You’d think that if you’re going to be appointed as a judge, you have to have proven that you have an excellent knowledge of the law and that you will make your decisions according to the law. That’s what we assume should be a requirement. You’ve got to know what the law is, and then you need to make decisions in accordance with the law. Not vibes. You can’t be just vibing as a judge.”
0:58

Elon Musk: “You’d think that if you’re going to be appointed as a judge, you have to have proven that you have an excellent knowledge of the law and that you will make your decisions according to the law. That’s what we assume should be a requirement. You’ve got to know what the law is, and then you need to make decisions in accordance with the law. Not vibes. You can’t be just vibing as a judge.”

Mars University

13,097,700 просмотров • 3 месяцев назад

Ferrari wants Melanie to be a judge at the auditions for #YoungAndReckless season 3 👀! Do you agree?
0:20

Ferrari wants Melanie to be a judge at the auditions for #YoungAndReckless season 3 👀! Do you agree?

Baddies + More!

47,896 просмотров • 1 год назад