Charles 🎉 Frye's banner

Charles 🎉 Frye

@charles_irl • 18,293 subscribers

memer of technical staff at @modal. he/him. ex @full_stack_dl, @weights_biases (acq. @CoreWeave), phd Berkeley @Redwood_Neuro. try https://t.co/SYWVMCb7OB

Shorts

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

Added a fun lil widget to the LLM Engineer's Almanac -- a "Token Timing Simulator" so you can get a visceral feel for what a benchmark perf number means. Here's David Wang's latest work with Zhijian Liu's DFlash technique in SGLang -- ~1k TPS!

18,561 просмотров

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

17,384 просмотров

nother banger in the pipeline btw

nother banger in the pipeline btw

12,600 просмотров

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

Dozens of teams have asked my advice on running LLMs. How fast is DeepSeek V3 with vLLM on 8 GPUs? What's the max throughput of Qwen 2.5 Coder with SGLang on one H100? Running & sharing benchmarks ad hoc was too slow So we built a tiny app, the LLM Engine Advisor

Dozens of teams have asked my advice on running LLMs. How fast is DeepSeek V3 with vLLM on 8 GPUs? What's the max throughput of Qwen 2.5 Coder with SGLang on one H100? Running & sharing benchmarks ad hoc was too slow So we built a tiny app, the LLM Engine Advisor

Charles 🎉 Frye

86,360 просмотров • 1 год назад

Announcing Twitter '95, an AI simulation of Twitter, if it had existed in 1995. - LLaMA 3.1 405B + FastAPI on Modal ✅ - Next.js app on Vercel ✅ - PostgreSQL on Supabase ✅ - Jordan dunking on Clinton ✅ - MVP written in 26 hours at crossover hackathon/marathon ✅

Announcing Twitter '95, an AI simulation of Twitter, if it had existed in 1995. - LLaMA 3.1 405B + FastAPI on Modal ✅ - Next.js app on Vercel ✅ - PostgreSQL on Supabase ✅ - Jordan dunking on Clinton ✅ - MVP written in 26 hours at crossover hackathon/marathon ✅

Charles 🎉 Frye

55,162 просмотров • 1 год назад

We've run thousands of LLM inference serving benchmarks at Modal. We're releasing the results so you don't have to. We're releasing the code so that you can. Introducing: The LLM Engineer's Almanac. Just in time for the AI Engineer World's Fair.

We've run thousands of LLM inference serving benchmarks at Modal. We're releasing the results so you don't have to. We're releasing the code so that you can. Introducing: The LLM Engineer's Almanac. Just in time for the AI Engineer World's Fair.

Charles 🎉 Frye

21,976 просмотров • 1 год назад

Больше нет контента для загрузки