正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

DSPy is great at classification tasks, but it's hard to make a formal eval metric for 'fuzzy' creative tasks like writing a blog post or telling a joke. Solution: train a judge with DSPy to agree with you 90%+, then use that judge as the eval metric for the... show more

Mike Taylor

9,523 subscribers

18,654 次观看 • 10 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:

Andrew Ng

181,822 次观看 • 1 年前

Glad that you guys both agree with me. When someone can't identify things that seem obvious, it's make you go hmmmm... seems like a good metric to use when "judging" objectively. 🤷‍♂️ Proper Doinks

Glad that you guys both agree with me. When someone can't identify things that seem obvious, it's make you go hmmmm... seems like a good metric to use when "judging" objectively. 🤷‍♂️ Proper Doinks

Archive Seed Bank

12,562 次观看 • 5 个月前

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

Hao AI Lab

69,037 次观看 • 1 年前

You're so good at drunk dancing that you accidentally confess This officer needed proof for the judge that her suspect was intoxicated, so she got creative with the sobriety test. First, a catwalk. Then she asked him to dance. And this man DELIVERED like he was auditioning for a reality show.

You're so good at drunk dancing that you accidentally confess This officer needed proof for the judge that her suspect was intoxicated, so she got creative with the sobriety test. First, a catwalk. Then she asked him to dance. And this man DELIVERED like he was auditioning for a reality show.

Brown World

46,689 次观看 • 6 个月前

LLMs are great for human in the loop applications, but fail at deterministic developer tasks. Interfaze (YC P26) is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more. Congrats on the launch, Yoeven and Harsha!

LLMs are great for human in the loop applications, but fail at deterministic developer tasks. Interfaze (YC P26) is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more. Congrats on the launch, Yoeven and Harsha!

Y Combinator

71,358 次观看 • 3 个月前

I always preach for independent wrestlers to set up merch tables at shows… But what should they sell? EVERYTHING! Be creative. Make merch that you think your fans would want. Make different merch that you sell at various prices. The merch table is a great way to make more money. But more importantly, it’s a great way to make a connection with fans.

I always preach for independent wrestlers to set up merch tables at shows… But what should they sell? EVERYTHING! Be creative. Make merch that you think your fans would want. Make different merch that you sell at various prices. The merch table is a great way to make more money. But more importantly, it’s a great way to make a connection with fans.

Matt Cardona

15,667 次观看 • 1 年前

Meta's chief AI officer says a swarm of agents out-produced a team of 100 engineers, and the machinery behind it is markdown files and cron jobs: "Like I think we've seen internally in Meta cases where if you can develop the right agentic loop and you have the right eval or the right metric for the agents to optimize, you can have a swarm of agents accomplish more than like a team of 100 engineers." "Very, very handily. Actually, very, very easily." "And so I think figuring out what the world looks like with lots of these agentic coordination problems, I think that is one of the most interesting problems today." "It just comes down to skills, markdown files, cron jobs." "I think it's always funny how mundane everything is once you really dig into it." If he is right, writing the metric now matters more than writing the code, and one judgment call about that metric stands in for 100 engineers' worth of output. Wang says Meta has already run this internally. - Alexandr Wang (Alexandr Wang), Chief AI Officer at Meta, with Garry Tan (Garry Tan) at Y Combinator (Y Combinator) Startup School 2026.

Meta's chief AI officer says a swarm of agents out-produced a team of 100 engineers, and the machinery behind it is markdown files and cron jobs: "Like I think we've seen internally in Meta cases where if you can develop the right agentic loop and you have the right eval or the right metric for the agents to optimize, you can have a swarm of agents accomplish more than like a team of 100 engineers." "Very, very handily. Actually, very, very easily." "And so I think figuring out what the world looks like with lots of these agentic coordination problems, I think that is one of the most interesting problems today." "It just comes down to skills, markdown files, cron jobs." "I think it's always funny how mundane everything is once you really dig into it." If he is right, writing the metric now matters more than writing the code, and one judgment call about that metric stands in for 100 engineers' worth of output. Wang says Meta has already run this internally. - Alexandr Wang (Alexandr Wang), Chief AI Officer at Meta, with Garry Tan (Garry Tan) at Y Combinator (Y Combinator) Startup School 2026.

Karl Mehta

26,779 次观看 • 12 天前

OH. MY. GOSH!!! The Charlotte NC judge who gave a $5k bond to a woman for assaulting her boyfriend who then ST*BBED HIM TO DE*TH 1 MONTH LATER... ...IS A DEMOCRAT JUDGE WITH PRIOR CHARGES HIMSELF FOR FALSIFYING LEGAL DOCUMENTS!!!! He served only 90 days of unsupervised probation and KEPT HIS JOB AS A JUDGE!!! This. Is. Insanity.

OH. MY. GOSH!!! The Charlotte NC judge who gave a $5k bond to a woman for assaulting her boyfriend who then STBBED HIM TO DETH 1 MONTH LATER... ...IS A DEMOCRAT JUDGE WITH PRIOR CHARGES HIMSELF FOR FALSIFYING LEGAL DOCUMENTS!!!! He served only 90 days of unsupervised probation and KEPT HIS JOB AS A JUDGE!!! This. Is. Insanity.

Matt Van Swol

123,600 次观看 • 4 个月前

Ever Heard of a Metric in Differential Geometry? How About a Geodesic? These two ideas are central to how Mathematics and Physics describe Spacetime. A metric is how a surface measures distance and angle. A geodesic is the path that counts as "straight" after that measuring rule is chosen. Once the metric changes from point to point, straightness is no longer something you judge by eye. On a curved surface, the shortest paths are geodesics. They are chosen by the metric. Take a surface written as r(x,y) = (x, y, f(x,y)) Its metric is gᵢⱼ = ∂ᵢr · ∂ⱼr A geodesic q(t) = (x(t), y(t)) obeys q̈ᵏ + Γᵏᵢⱼ q̇ᶦq̇ʲ = 0 where Γᵏᵢⱼ are the Christoffel symbols of the surface metric. Launch many almost-parallel paths. At the start, they look like a clean beam until curvature takes over. Some paths focus, others separate and some slide around ridges. The surface is essentially the rulebook for motion. The shortest path is local. The geometry remembers everything. #DifferentialGeometry #Geodesics #RiemannianGeometry #MathematicalPhysics #Curvature #TensorCalculus #AppliedMathematics #Geometry #ScienceAnimation

Ever Heard of a Metric in Differential Geometry? How About a Geodesic? These two ideas are central to how Mathematics and Physics describe Spacetime. A metric is how a surface measures distance and angle. A geodesic is the path that counts as "straight" after that measuring rule is chosen. Once the metric changes from point to point, straightness is no longer something you judge by eye. On a curved surface, the shortest paths are geodesics. They are chosen by the metric. Take a surface written as r(x,y) = (x, y, f(x,y)) Its metric is gᵢⱼ = ∂ᵢr · ∂ⱼr A geodesic q(t) = (x(t), y(t)) obeys q̈ᵏ + Γᵏᵢⱼ q̇ᶦq̇ʲ = 0 where Γᵏᵢⱼ are the Christoffel symbols of the surface metric. Launch many almost-parallel paths. At the start, they look like a clean beam until curvature takes over. Some paths focus, others separate and some slide around ridges. The surface is essentially the rulebook for motion. The shortest path is local. The geometry remembers everything. #DifferentialGeometry #Geodesics #RiemannianGeometry #MathematicalPhysics #Curvature #TensorCalculus #AppliedMathematics #Geometry #ScienceAnimation

Mathelirium

22,170 次观看 • 1 个月前

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

Santiago

36,273 次观看 • 1 年前

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MTS

100,519 次观看 • 27 天前

Teachers can use Canva with great ease, To make poetry lessons a delightful breeze. With templates for poems and creative art, Students' creativity can shine from the start. 🎤💥

Teachers can use Canva with great ease, To make poetry lessons a delightful breeze. With templates for poems and creative art, Students' creativity can shine from the start. 🎤💥

Canva Education

19,336 次观看 • 3 年前

.Dave Rubin: "How do you sort of quantify or analyze the behavior around [the pressures to make content only for clicks?]" Gad Saad: "I think the problem with Candace and a lot of those folks, is that the metric that they're trying to optimize is, as you've guys have said, clickbait."

.Dave Rubin: "How do you sort of quantify or analyze the behavior around [the pressures to make content only for clicks?]" Gad Saad: "I think the problem with Candace and a lot of those folks, is that the metric that they're trying to optimize is, as you've guys have said, clickbait."

The Rubin Report

23,589 次观看 • 8 个月前

On a previous #TeawithGaryVee I was asked "What is more important to you as a leader? Challenging your employees with new tasks to grow or making sure they to give them tasks that they excel at?" if you have a question for me, drop it here so my team can add it to the next Rapid Fire Q&A and follow me on

On a previous #TeawithGaryVee I was asked "What is more important to you as a leader? Challenging your employees with new tasks to grow or making sure they to give them tasks that they excel at?" if you have a question for me, drop it here so my team can add it to the next Rapid Fire Q&A and follow me on

Gary Vaynerchuk

22,719 次观看 • 1 个月前

🚨 ELON MUSK: "It's not like, I agree with everything the administration does. I agree with much of what the administration does, but we have differences of opinion, you know, the things that I don't entirely agree with. But it's difficult for me to bring that up in an interview, because then it creates a bone of contention. So then I'm in a little stuck, in a bind where I'm like, Well, I don't want to, you know, speak up against the administration, but I don't want to, you also don't want to take responsibility for everything, the administration is doing."

🚨 ELON MUSK: "It's not like, I agree with everything the administration does. I agree with much of what the administration does, but we have differences of opinion, you know, the things that I don't entirely agree with. But it's difficult for me to bring that up in an interview, because then it creates a bone of contention. So then I'm in a little stuck, in a bind where I'm like, Well, I don't want to, you know, speak up against the administration, but I don't want to, you also don't want to take responsibility for everything, the administration is doing."

DogeDesigner

98,616 次观看 • 1 年前

Elon Musk: “You’d think that if you’re going to be appointed as a judge, you have to have proven that you have an excellent knowledge of the law and that you will make your decisions according to the law. That’s what we assume should be a requirement. You’ve got to know what the law is, and then you need to make decisions in accordance with the law. Not vibes. You can’t be just vibing as a judge.”

Elon Musk: “You’d think that if you’re going to be appointed as a judge, you have to have proven that you have an excellent knowledge of the law and that you will make your decisions according to the law. That’s what we assume should be a requirement. You’ve got to know what the law is, and then you need to make decisions in accordance with the law. Not vibes. You can’t be just vibing as a judge.”

Mars University

13,098,555 次观看 • 4 个月前

Aaron Judge was hit coming off the field with a throw from Anthony Volpe. It is normal for a ball to be thrown to Judge as he comes off the field

Aaron Judge was hit coming off the field with a throw from Anthony Volpe. It is normal for a ball to be thrown to Judge as he comes off the field

Talkin' Yanks

693,316 次观看 • 1 年前