Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

Charles 🎉 Frye

19,138 subscribers

18,764 views • 2 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang

Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration! - 5x🚀than vanilla (on HF) - 1.4x🚀than EAGLE-2 (on HF) - A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang) - 1.65x🚀in latency even for large bs=64 (on SGLang) - A new scaling law: more training data, better speedup - Apache 2.0 Paper: Code: SGLang version: ⚒️Takeaway: Introducing training-time test, a novel draft model training technique: we replace feature prediction with direct token prediction and shift from top-layer-only features to multi-layer feature fusion. This approach unlocks a new scaling law previously undiscovered in EAGLE and EAGLE-2. 🙏Acknowledge: We would like to thank the SGLang team (zhyncs Lianmin Zheng Ying Sheng James Liu, Ke Bao, and others LMSYS Org) for their merge and careful evaluation of EAGLE-3 on SGLang. 🤝Want to collaborate? We're a small academic group with limited GPU resources. If you're interested in supporting our next version of EAGLE or would like us to train a preliminary version tailored to a specific model, please get in touch! Joint work with Yuhui Li, Fangyun Wei, and Chao Zhang

Hongyang Zhang

42,200 views • 1 year ago

In the Halo games, we never really see a Pelican land and takeoff using their landing gear, apart from 1 shot in Halo Infinite, and Microsoft Flight simulator where you can fly the pelican. So animating this shot trying to get it feel realistic with no real reference was fun.

In the Halo games, we never really see a Pelican land and takeoff using their landing gear, apart from 1 shot in Halo Infinite, and Microsoft Flight simulator where you can fly the pelican. So animating this shot trying to get it feel realistic with no real reference was fun.

MotoGinji

65,764 views • 4 months ago

Realized that visuals can get busy when a large number of lines are on the screen, so I added a 1-bit "fog" to dissolve lines in the distance. This will be useful for simulating dark areas.

Realized that visuals can get busy when a large number of lines are on the screen, so I added a 1-bit "fog" to dissolve lines in the distance. This will be useful for simulating dark areas.

Romanus Surt （スルト゠ロマヌス）

14,219 views • 1 year ago

I wanted to give a better look at what Adaptive Hardness actually means,so here’s a quick video to show it in action. The base material is a 40 Shore A polymer, but thanks to this technique, I can control the hardness dynamically within the same part.

I wanted to give a better look at what Adaptive Hardness actually means,so here’s a quick video to show it in action. The base material is a 40 Shore A polymer, but thanks to this technique, I can control the hardness dynamically within the same part.

Simone Fontana

89,720 views • 1 year ago

Here's a fun one with Seedance 2.0 You can just tell the model to end with the first frame, so you can create looping videos without using first/last frames, a feature especially helpful when using the Omni variant of SD2. Generated in CapCut with early access. Prompt below (it's a big one):

Here's a fun one with Seedance 2.0 You can just tell the model to end with the first frame, so you can create looping videos without using first/last frames, a feature especially helpful when using the Omni variant of SD2. Generated in CapCut with early access. Prompt below (it's a big one):

Alex Patrascu

42,179 views • 5 months ago

SOMEONE TURNED THEIR TEAM'S TASK TRACKER INTO A 3D ISLAND instead of a boring list of tasks, your teams work is a little island that grows as you get stuff done > you assign tasks right in slack, just type who its for, the points, and the due date > finish a task and you get to place a building on the island > get your work rejected and the building collapses into rubble > the rubble stays there forever, so everyone can see it > each new sprint starts a fresh island so over time the island fills up with buildings for all the work your team actually finished, and the rubble is a reminder of what got rejected. its open source, so any team can set it up. way more fun than staring at a to do list all day

SOMEONE TURNED THEIR TEAM'S TASK TRACKER INTO A 3D ISLAND instead of a boring list of tasks, your teams work is a little island that grows as you get stuff done > you assign tasks right in slack, just type who its for, the points, and the due date > finish a task and you get to place a building on the island > get your work rejected and the building collapses into rubble > the rubble stays there forever, so everyone can see it > each new sprint starts a fresh island so over time the island fills up with buildings for all the work your team actually finished, and the rubble is a reminder of what got rejected. its open source, so any team can set it up. way more fun than staring at a to do list all day

Om Patel

12,526 views • 20 days ago

"It was a lot of fun. The guys have welcomed me in with open arms. It's just fun to get out there, start, and play with a good defense behind me ... a lot of fun today." David Peterson has loved his time as a Cub so far 👏

"It was a lot of fun. The guys have welcomed me in with open arms. It's just fun to get out there, start, and play with a good defense behind me ... a lot of fun today." David Peterson has loved his time as a Cub so far 👏

Marquee Sports Network

44,801 views • 1 month ago

Max Recorded TPS (1 Block) is now on Chainspect! 🚀 So far, we've measured Max Recorded TPS only for a period of 100 blocks. You take the total transactions over the time and divide them by cumulative block time to get at estimate of what TPS a blockchain can churn out consistently during peak workload periods. Now, we're adding the Max Recorded TPS per block. That's the highest-ever burst TPS a chain has handled on record — a measure of its peak performance under high stress. Check out how your fav chain has fared:

Max Recorded TPS (1 Block) is now on Chainspect! 🚀 So far, we've measured Max Recorded TPS only for a period of 100 blocks. You take the total transactions over the time and divide them by cumulative block time to get at estimate of what TPS a blockchain can churn out consistently during peak workload periods. Now, we're adding the Max Recorded TPS per block. That's the highest-ever burst TPS a chain has handled on record — a measure of its peak performance under high stress. Check out how your fav chain has fared:

Chainspect

49,252 views • 1 year ago

Had a super fun time animating/directing this video for GLITCH with Zach P. 🎉 Was really fun to write these lil' guys. #TheAmazingDigitalCircus Here's all the animation I did before I got in a car accident lol:

Had a super fun time animating/directing this video for GLITCH with Zach P. 🎉 Was really fun to write these lil' guys. #TheAmazingDigitalCircus Here's all the animation I did before I got in a car accident lol:

Micah Preciado

85,507 views • 2 years ago

𝗔𝗻𝘆𝗼𝗻𝗲 𝗰𝗮𝗻 𝗹𝗮𝘂𝗻𝗰𝗵 𝗮 𝗠𝟯𝗠𝟯. We’ve just updated the ‘Mint a coin’ interface to allow creators to launch a M3M3, with or without an existing token. Here's how you can mint a coin, create a pool, and launch a M3M3 in one simple process: 👇

𝗔𝗻𝘆𝗼𝗻𝗲 𝗰𝗮𝗻 𝗹𝗮𝘂𝗻𝗰𝗵 𝗮 𝗠𝟯𝗠𝟯. We’ve just updated the ‘Mint a coin’ interface to allow creators to launch a M3M3, with or without an existing token. Here's how you can mint a coin, create a pool, and launch a M3M3 in one simple process: 👇

Meteora

30,064 views • 1 year ago

🐿: as a long time pirates of the caribbean fan, i really feel like i put in the work for this 🐇: i’m a bit disappointed you didn’t put on a beard though ok so what am i supposed to do with the information that seonghwa prefers pirate hongjoong with a beard now??!?🤓

🐿: as a long time pirates of the caribbean fan, i really feel like i put in the work for this 🐇: i’m a bit disappointed you didn’t put on a beard though ok so what am i supposed to do with the information that seonghwa prefers pirate hongjoong with a beard now??!?🤓

skie ⋆. 𐙚 ˚

33,993 views • 5 months ago

omg.. just found a way to install&use Clawdbot in 2 mins no need mac mini, API keys, just one click to set up everything automatically and get personal AI assistant to work for you 24/7 here's how and what you can do with it:

omg.. just found a way to install&use Clawdbot in 2 mins no need mac mini, API keys, just one click to set up everything automatically and get personal AI assistant to work for you 24/7 here's how and what you can do with it:

el.cine

3,322,427 views • 5 months ago

Excited to share the news that i'm a part of Rare Atom! A new adventure for me and i'm looking forward to show the world what we can do 👊 Glad to be back with mr Szedlár Kornél aswell here's a lil deagle 1v5 from practice

Excited to share the news that i'm a part of Rare Atom! A new adventure for me and i'm looking forward to show the world what we can do 👊 Glad to be back with mr Szedlár Kornél aswell here's a lil deagle 1v5 from practice

phzy

153,267 views • 2 years ago

Do you have any healthy tricks to handle exponential scaling in #AfterEffects without baking keyframes with the Exponential Scale feature? Always on the lookout for a smoother workflow. Here's the technique I've been using so far 👇 #MotionDesign #CreativeSolutions

Do you have any healthy tricks to handle exponential scaling in #AfterEffects without baking keyframes with the Exponential Scale feature? Always on the lookout for a smoother workflow. Here's the technique I've been using so far 👇 #MotionDesign #CreativeSolutions

François Lefebvre

39,121 views • 2 years ago

The newest version of our Almanac preprint is out, and just in time for our demo at the Stanford AIMI Symposium 2023! Almanac is a retrieval-augmented LLM that provides up-to-date and verifiable answers to medical queries. Link: We benchmark our approach on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified & resident physicians, and demonstrate significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties. More interestingly, because the retrieved data acts as a single source of truth, we find retrieval-based LLMs to be more robust to prompt injection and manipulation! Future work will involve expanding the scope of our dataset to more specialties and multimodal settings. #Medtwitter #MedEd

The newest version of our Almanac preprint is out, and just in time for our demo at the Stanford AIMI Symposium 2023! Almanac is a retrieval-augmented LLM that provides up-to-date and verifiable answers to medical queries. Link: We benchmark our approach on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified & resident physicians, and demonstrate significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties. More interestingly, because the retrieved data acts as a single source of truth, we find retrieval-based LLMs to be more robust to prompt injection and manipulation! Future work will involve expanding the scope of our dataset to more specialties and multimodal settings. #Medtwitter #MedEd

Cyril Zakka, MD

18,128 views • 3 years ago

What happens next? We took a photo with Lens and asked Nano Banana to generate the next 5 seconds of a scene. While it’s not predicting the future in a literal sense, the model draws upon its vast world knowledge to generate a plausible outcome in each of these examples. It's a fun peek at how our latest AI image editing model can be used for creative forecasting and storytelling. Nano Banana is now live in Search – accessible through Lens and AI Mode – so you can try this out for yourself!

What happens next? We took a photo with Lens and asked Nano Banana to generate the next 5 seconds of a scene. While it’s not predicting the future in a literal sense, the model draws upon its vast world knowledge to generate a plausible outcome in each of these examples. It's a fun peek at how our latest AI image editing model can be used for creative forecasting and storytelling. Nano Banana is now live in Search – accessible through Lens and AI Mode – so you can try this out for yourself!

Rajan Patel

74,826 views • 9 months ago

Thank you so so much to Dooby for giving me the chance of making your 3D Model! It was so much fun and a joy to work with you again!! 🚃💨 #doobart

Thank you so so much to Dooby for giving me the chance of making your 3D Model! It was so much fun and a joy to work with you again!! 🚃💨 #doobart

Yoolie

500,567 views • 1 year ago

🔗 Watch the full class: 📚 Third ropes get added to TKs for a few different reasons: a decorative touch, extra reinforcement to keep the harness in place on certain body types, or added support that gives you more options if you're playing with partials or taking the harness into the air. On our website, you'll find a range of TK variations taught by different instructors, each with their own technical and visual flavor, so you can find the approach that fits your body and what you're going for.

🔗 Watch the full class: 📚 Third ropes get added to TKs for a few different reasons: a decorative touch, extra reinforcement to keep the harness in place on certain body types, or added support that gives you more options if you're playing with partials or taking the harness into the air. On our website, you'll find a range of TK variations taught by different instructors, each with their own technical and visual flavor, so you can find the approach that fits your body and what you're going for.

Shibari Study

16,743 views • 15 days ago

introducing a new, very fun, LLM benchmark- the Game-of-Life Bench! the rules are simple: given an 8x8 grid following Conway's game of life rules, the goal is to create an initial pattern with at most 32 cells that can last the longest number of turns before dying/repeating. some results to highlight (with caveats detailed below): - gpt 5.1 lasts the longest with a 106 step run - claude models are really bad at this! they refuse to reason about this task and score < 25 points - deepseek r1 is the best open model with 102 steps. why? because i wanted to create a benchmark that has (i think) no practicality, but is still fun to look at, cheap, and still measures something interesting. i also am a big fan of the game of life. its absurdly simple rules leading to intractability is extremely cool to me. also, i saw a lot of work with LLMs trying to "predict" the next state in Conway's game of life, I think game-of-life bench is more fun because it's pretty open ended and only asks the LLM for the initial state. I also think this could be an RL env? but idk why you would ever train on this task haha i don't think this is a "serious" benchmark because it doesnt measure anything practical, but i still think it's a hard benchmark exactly because you can't predict what happens with your initial state many turns into the future; this is why i was initially expecting all LLMs to be bad at it, but turns out, some are clearly better than the others (the ordering may surprise you!) reminder: this is still a work-in-progress; (1) i am gpu-poor so could only do 10 runs for each model, even though total running cost is relatively low. maybe with some more credits i can run more seeds for each model. (2) i handpicked models which i think are at the frontier right now, plus some others that were on my mind. so, if you'd like to see a model on here, let me know. (3) i currently only do an 8x8 grid because i thought that by itself would be pretty hard for current LLMs, but of course we can increase grid sizes! (4) the coolest thing is, i dont think we can calculate the max possible number of states (yay undecidability!) you can go without repeating, so this is essentially a no-ceiling task, which is pretty cool! again, i did this mostly out of a desire to make LLMs do something fun. if this keeps me entertained for a few more days, i'd likely release a blog post on it. if it keeps me entertained for a week (and someone sponsors me), i'll put more work into it :P lastly, this is fully open sourced, so feel free to run this on your own!

introducing a new, very fun, LLM benchmark- the Game-of-Life Bench! the rules are simple: given an 8x8 grid following Conway's game of life rules, the goal is to create an initial pattern with at most 32 cells that can last the longest number of turns before dying/repeating. some results to highlight (with caveats detailed below): - gpt 5.1 lasts the longest with a 106 step run - claude models are really bad at this! they refuse to reason about this task and score < 25 points - deepseek r1 is the best open model with 102 steps. why? because i wanted to create a benchmark that has (i think) no practicality, but is still fun to look at, cheap, and still measures something interesting. i also am a big fan of the game of life. its absurdly simple rules leading to intractability is extremely cool to me. also, i saw a lot of work with LLMs trying to "predict" the next state in Conway's game of life, I think game-of-life bench is more fun because it's pretty open ended and only asks the LLM for the initial state. I also think this could be an RL env? but idk why you would ever train on this task haha i don't think this is a "serious" benchmark because it doesnt measure anything practical, but i still think it's a hard benchmark exactly because you can't predict what happens with your initial state many turns into the future; this is why i was initially expecting all LLMs to be bad at it, but turns out, some are clearly better than the others (the ordering may surprise you!) reminder: this is still a work-in-progress; (1) i am gpu-poor so could only do 10 runs for each model, even though total running cost is relatively low. maybe with some more credits i can run more seeds for each model. (2) i handpicked models which i think are at the frontier right now, plus some others that were on my mind. so, if you'd like to see a model on here, let me know. (3) i currently only do an 8x8 grid because i thought that by itself would be pretty hard for current LLMs, but of course we can increase grid sizes! (4) the coolest thing is, i dont think we can calculate the max possible number of states (yay undecidability!) you can go without repeating, so this is essentially a no-ceiling task, which is pretty cool! again, i did this mostly out of a desire to make LLMs do something fun. if this keeps me entertained for a few more days, i'd likely release a blog post on it. if it keeps me entertained for a week (and someone sponsors me), i'll put more work into it :P lastly, this is fully open sourced, so feel free to run this on your own!

Akshit

13,722 views • 5 months ago

Token Generation Ceremonies are now live - Liqfid kicks off the first one very soon What are TGCs? TGCs are permissionless markets - it’s a way to get validation and liquidity for your appcoin, idea or memecoin and ensure that your core community gets to get into the token on the ground floor. Remember: TGCs are permissionless, so anyone can launch a TGC, about anything - so go wild Start a Ceremony at Welcome to the era of Token Generation Ceremonies. ✳️

Token Generation Ceremonies are now live - Liqfid kicks off the first one very soon What are TGCs? TGCs are permissionless markets - it’s a way to get validation and liquidity for your appcoin, idea or memecoin and ensure that your core community gets to get into the token on the ground floor. Remember: TGCs are permissionless, so anyone can launch a TGC, about anything - so go wild Start a Ceremony at Welcome to the era of Token Generation Ceremonies. ✳️

fey

17,971 views • 6 months ago