Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Iterating... Evals take time and are boring: but are a fundamental validation step of sane LLM inference. Let's try to make them as easy and fun to run as possible.

antirez's profile picture

antirez

68,635 subscribers

32,727 görüntüleme • 1 ay önce •via X (Twitter)

Bilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Unpopular opinion: Most agent evals are theatre. You run them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →
0:13

Unpopular opinion: Most agent evals are theatre. You run them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →

Swapna Kumar Panda

49,557 görüntüleme • 2 ay önce

okay I've made these available to preorder, these are limited and I don't know if I'll be making anymore after this batch as they take a lot of time and effort lmao. however I will try and make some of the Breach ones to go with them next |-/
0:16

okay I've made these available to preorder, these are limited and I don't know if I'll be making anymore after this batch as they take a lot of time and effort lmao. however I will try and make some of the Breach ones to go with them next |-/

chris |-/

11,170 görüntüleme • 7 ay önce

Would you? Could you? Challenge your balance and speed with the thrilling water stone run! These stones dip into the water if you linger too long, requiring quick reflexes and steady footing to make it across. A fun and exciting test of agility and balance-can you master the art of running on water stones? This looks fun, but things are never as easy as they look.
0:15

Would you? Could you? Challenge your balance and speed with the thrilling water stone run! These stones dip into the water if you linger too long, requiring quick reflexes and steady footing to make it across. A fun and exciting test of agility and balance-can you master the art of running on water stones? This looks fun, but things are never as easy as they look.

NancyH

4,956,471 görüntüleme • 3 ay önce

“When I step onto the pitch, I try to manage my emotions as much as possible to be clinical. And of course, when you score or assist, there are some emotions that come in — but they go away just as quickly because you have to immediately focus on the next play. So you enjoy the moment, but it’s extremely quick.”
0:18

“When I step onto the pitch, I try to manage my emotions as much as possible to be clinical. And of course, when you score or assist, there are some emotions that come in — but they go away just as quickly because you have to immediately focus on the next play. So you enjoy the moment, but it’s extremely quick.”

Ous

27,998 görüntüleme • 1 yıl önce

It's amazing that women are actually confused as to why men don't approach them and flirt them anymore. All you had to do was be sane.
0:18

It's amazing that women are actually confused as to why men don't approach them and flirt them anymore. All you had to do was be sane.

Jedediah Bila

12,121 görüntüleme • 4 gün önce

A million children once lived here. They are lying about the dead and the plan was never to expel, but to murder as many as possible because Palestine only exists as long as there are Palestinians. They will leave a small amount and claim to "safely evacuated 2 million people."
0:18

A million children once lived here. They are lying about the dead and the plan was never to expel, but to murder as many as possible because Palestine only exists as long as there are Palestinians. They will leave a small amount and claim to "safely evacuated 2 million people."

Khaled 𐌔𐌔oliman

1,030,647 görüntüleme • 11 ay önce

TRUMP: The U.S. will run Venezuela. “We are going to run [Venezuela] until such time as we can do a safe, proper, and judicious transition [of power.]”
0:17

TRUMP: The U.S. will run Venezuela. “We are going to run [Venezuela] until such time as we can do a safe, proper, and judicious transition [of power.]”

Wall Street Mav

35,757 görüntüleme • 6 ay önce

Trump on Venezuela: We are going to run it essentially until such time as a proper transition can take place.
0:12

Trump on Venezuela: We are going to run it essentially until such time as a proper transition can take place.

Acyn

20,681 görüntüleme • 6 ay önce

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread
0:17

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Linus ✦ Ekenstam

915,787 görüntüleme • 2 yıl önce

🐾 Trying to follow in mom’s footsteps isn’t always easy… but our red panda boys are figuring it out one step (and stumble) at a time.
0:12

🐾 Trying to follow in mom’s footsteps isn’t always easy… but our red panda boys are figuring it out one step (and stumble) at a time.

Zoo Knoxville

13,509 görüntüleme • 10 ay önce

A little girl in a bomb shelter in Israel is going from dog to dog and petting them to make them calmer, as they are afraid of the sirens and explosions. I love this.
0:11

A little girl in a bomb shelter in Israel is going from dog to dog and petting them to make them calmer, as they are afraid of the sirens and explosions. I love this.

Vivid.🇮🇱

242,041 görüntüleme • 4 ay önce

60yds+ 5.3’s+ are easy these days when I have so much time to punt and workout as a free agent lol
0:18

60yds+ 5.3’s+ are easy these days when I have so much time to punt and workout as a free agent lol

Michael Turk

189,986 görüntüleme • 2 yıl önce

Elon Musk: “Spend less time on finance, in conference rooms and on PowerPoint, and more time just trying to make your product as amazing as possible.”
0:12

Elon Musk: “Spend less time on finance, in conference rooms and on PowerPoint, and more time just trying to make your product as amazing as possible.”

ELON CLIPS

42,004 görüntüleme • 1 yıl önce

hello i have watched the entire episode of the bbc reality show where british people are dropped in vancouver and asked to get to haida gwaii as fast as possible without flying or having phones/computers to guide them and it is just as wonderful as i hoped
0:18

hello i have watched the entire episode of the bbc reality show where british people are dropped in vancouver and asked to get to haida gwaii as fast as possible without flying or having phones/computers to guide them and it is just as wonderful as i hoped

Justin McElroy

223,491 görüntüleme • 3 yıl önce

girl YOU are hybe’s target audience keeping them employed and all.. might as well make monthly despots to them to keep them in business
0:19

girl YOU are hybe’s target audience keeping them employed and all.. might as well make monthly despots to them to keep them in business

khloe⁷

49,659 görüntüleme • 27 gün önce