Charles 🎉 Frye's banner
Charles 🎉 Frye's profile picture

Charles 🎉 Frye

@charles_irl • 18,202 subscribers

memer of technical staff at @modal. he/him. ex @full_stack_dl, @weights_biases (acq. @CoreWeave), phd Berkeley @Redwood_Neuro. try https://t.co/SYWVMCb7OB

Shorts

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

18,561 views

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

17,384 views

nother banger in the pipeline btw

nother banger in the pipeline btw

12,600 views

Videos

LIVE
1.2k

Anya Rossi

sweetdream.ai

SweetDream.aiSponsored•Livecam
Streaming Now

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

HD live stream
Exclusive private shows
1.2k viewers online

Free preview available • Premium content

Dozens of teams have asked my advice on running LLMs. How fast is DeepSeek V3 with vLLM on 8 GPUs? What's the max throughput of Qwen 2.5 Coder with SGLang on one H100? Running & sharing benchmarks ad hoc was too slow So we built a tiny app, the LLM Engine Advisor
0:38
charles_irl's profile picture

Dozens of teams have asked my advice on running LLMs. How fast is DeepSeek V3 with vLLM on 8 GPUs? What's the max throughput of Qwen 2.5 Coder with SGLang on one H100? Running & sharing benchmarks ad hoc was too slow So we built a tiny app, the LLM Engine Advisor

Charles 🎉 Frye

86,348 views • 1 year ago

Announcing Twitter '95, an AI simulation of Twitter, if it had existed in 1995. - LLaMA 3.1 405B + FastAPI on Modal ✅ - Next.js app on Vercel ✅ - PostgreSQL on Supabase ✅ - Jordan dunking on Clinton ✅ - MVP written in 26 hours at crossover hackathon/marathon ✅
0:23
charles_irl's profile picture

Announcing Twitter '95, an AI simulation of Twitter, if it had existed in 1995. - LLaMA 3.1 405B + FastAPI on Modal ✅ - Next.js app on Vercel ✅ - PostgreSQL on Supabase ✅ - Jordan dunking on Clinton ✅ - MVP written in 26 hours at crossover hackathon/marathon ✅

Charles 🎉 Frye

55,162 views • 1 year ago

We've run thousands of LLM inference serving benchmarks at Modal. We're releasing the results so you don't have to. We're releasing the code so that you can. Introducing: The LLM Engineer's Almanac. Just in time for the AI Engineer World's Fair.
0:28
charles_irl's profile picture

We've run thousands of LLM inference serving benchmarks at Modal. We're releasing the results so you don't have to. We're releasing the code so that you can. Introducing: The LLM Engineer's Almanac. Just in time for the AI Engineer World's Fair.

Charles 🎉 Frye

21,976 views • 1 year ago

No more content to load