正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Iterating... Evals take time and are boring: but are a fundamental validation step of sane LLM inference. Let's try to make them as easy and fun to run as possible.

antirez

79,908 subscribers

32,815 次观看 • 5 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Unpopular opinion: Most agent evals are theatre. You run them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →

Unpopular opinion: Most agent evals are theatre. You run them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →

Swapna Kumar Panda

50,466 次观看 • 5 个月前

okay I've made these available to preorder, these are limited and I don't know if I'll be making anymore after this batch as they take a lot of time and effort lmao. however I will try and make some of the Breach ones to go with them next |-/

okay I've made these available to preorder, these are limited and I don't know if I'll be making anymore after this batch as they take a lot of time and effort lmao. however I will try and make some of the Breach ones to go with them next |-/

chris |-/

11,170 次观看 • 10 个月前

Would you? Could you? Challenge your balance and speed with the thrilling water stone run! These stones dip into the water if you linger too long, requiring quick reflexes and steady footing to make it across. A fun and exciting test of agility and balance-can you master the art of running on water stones? This looks fun, but things are never as easy as they look.

Would you? Could you? Challenge your balance and speed with the thrilling water stone run! These stones dip into the water if you linger too long, requiring quick reflexes and steady footing to make it across. A fun and exciting test of agility and balance-can you master the art of running on water stones? This looks fun, but things are never as easy as they look.

NancyH

4,957,937 次观看 • 6 个月前

NEFERTITI Something as irrelevant and as incompetent as Tulumbu dey talk and you wan make human being take notes??? Like WTAF are you expecting him to write? Balablue? Dasindasundagoden? Make u na try dey pity pesin o!

NEFERTITI Something as irrelevant and as incompetent as Tulumbu dey talk and you wan make human being take notes??? Like WTAF are you expecting him to write? Balablue? Dasindasundagoden? Make u na try dey pity pesin o!

velele (#NDC_OK_2027)

27,726 次观看 • 1 年前

“When I step onto the pitch, I try to manage my emotions as much as possible to be clinical. And of course, when you score or assist, there are some emotions that come in — but they go away just as quickly because you have to immediately focus on the next play. So you enjoy the moment, but it’s extremely quick.”

“When I step onto the pitch, I try to manage my emotions as much as possible to be clinical. And of course, when you score or assist, there are some emotions that come in — but they go away just as quickly because you have to immediately focus on the next play. So you enjoy the moment, but it’s extremely quick.”

Ous

27,998 次观看 • 1 年前

It's amazing that women are actually confused as to why men don't approach them and flirt them anymore. All you had to do was be sane.

It's amazing that women are actually confused as to why men don't approach them and flirt them anymore. All you had to do was be sane.

Jedediah Bila

12,121 次观看 • 3 个月前

What do you guys think about private jets? I know haters build narratives around them... and try to make it as if you never flew on one... But... I do think they are fun and way too expensive. My life is a movie bayyyy bayyyyy! 😎

What do you guys think about private jets? I know haters build narratives around them... and try to make it as if you never flew on one... But... I do think they are fun and way too expensive. My life is a movie bayyyy bayyyyy! 😎

𝕁𝕠𝕜𝕖𝕣🃏

131,922 次观看 • 19 天前

TRUMP: The U.S. will run Venezuela. “We are going to run [Venezuela] until such time as we can do a safe, proper, and judicious transition [of power.]”

TRUMP: The U.S. will run Venezuela. “We are going to run [Venezuela] until such time as we can do a safe, proper, and judicious transition [of power.]”

Wall Street Mav

35,757 次观看 • 9 个月前

Trump on Venezuela: We are going to run it essentially until such time as a proper transition can take place.

Trump on Venezuela: We are going to run it essentially until such time as a proper transition can take place.

Acyn

20,681 次观看 • 9 个月前

Markwayne Mullin: "The Democrats throw out all these fearmongels to try to make ICE and President Trump's administration look as worse as possible" 🥴

Markwayne Mullin: "The Democrats throw out all these fearmongels to try to make ICE and President Trump's administration look as worse as possible" 🥴

Aaron Rupar

570,632 次观看 • 1 个月前

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Linus ✦ Ekenstam

916,022 次观看 • 2 年前

🐾 Trying to follow in mom’s footsteps isn’t always easy… but our red panda boys are figuring it out one step (and stumble) at a time.

🐾 Trying to follow in mom’s footsteps isn’t always easy… but our red panda boys are figuring it out one step (and stumble) at a time.

Zoo Knoxville

13,723 次观看 • 1 年前

People don’t realize that they need as many data centers as possible to store data and train AI to perfection. Once they are finished and achieve true AGI, they won’t need that much space. They will run their AI and robots in their local underground doomsday bunkers with limited power and data centers, but with enough capacity for them to serve as their workers.

People don’t realize that they need as many data centers as possible to store data and train AI to perfection. Once they are finished and achieve true AGI, they won’t need that much space. They will run their AI and robots in their local underground doomsday bunkers with limited power and data centers, but with enough capacity for them to serve as their workers.

Open Minded Approach

16,216 次观看 • 1 个月前

A little girl in a bomb shelter in Israel is going from dog to dog and petting them to make them calmer, as they are afraid of the sirens and explosions. I love this.

A little girl in a bomb shelter in Israel is going from dog to dog and petting them to make them calmer, as they are afraid of the sirens and explosions. I love this.

Vivid.🇮🇱

242,283 次观看 • 7 个月前

I compiled over 11k bug bounty reports into one library a little bit ago, I wanted to run some analysis on bounty reports, but had a hard time finding and compiling everything especially across platforms so I decided to just make a library + api for anyone who wants to run any sort of tests/analysis etc lemme know what yall think + anything to add to make it as useful as possible :) link in replies

I compiled over 11k bug bounty reports into one library a little bit ago, I wanted to run some analysis on bounty reports, but had a hard time finding and compiling everything especially across platforms so I decided to just make a library + api for anyone who wants to run any sort of tests/analysis etc lemme know what yall think + anything to add to make it as useful as possible :) link in replies

Perly 🦈

76,141 次观看 • 1 个月前