Scaling past a trillion parameters requires inference engineers to solve 3 issues: - Width: Routing between hundreds of sparse experts - Depth: Mixing global and local linear attention - Length: Processing 1M-token sequences I asked Ali and Bryce to draw it up. They delivered.

what a chat

Keep drawing on the whiteboard and they might promote you to @ waterloo_professor if you're not careful.

@waterloo_intern how I imagine @waterloo_intern in office hours

@waterloo_intern @waterloo_intern dream team

@waterloo_intern taking notes ✍️

@waterloo_intern Gold

@waterloo_intern banger drop

@waterloo_intern trillion parameters mostly proves nobody profits from serving them

@waterloo_intern I am solving the same bottleneck on the decode side

@waterloo_intern Unfortunate that this wasn't cross-posted to YT. X video experience is still awful for longer content.

@waterloo_intern Its on YT, Baseten channel

@waterloo_intern I'd guess at 1M tokens the few global layers' KV cache is most of the memory, since the local ones stay flat. Does that end up deciding how many global layers you can afford?

@waterloo_intern You guys are making the best technical ML content out there. @oneill_c @jxmnop one was great too

@waterloo_intern 把万亿参数的推理难题拆成宽度、深度、长度三维很清晰：专家路由决定算力怎么分，混合全局与局部注意力决定每层怎么省，百万 token 序列又把 KV 缓存和调度逼到极限。实际部署里最先卡住的往往不是单项峰值，而是三者叠加时的尾延迟，这一块你们有实测数据吗？

@waterloo_intern where's the white board from tho

@waterloo_intern If you talk about abstract concepts long enough one automatically spawns

@waterloo_intern this is awesome. thanks for this amazing video

A single parameter count can hide a lot of very different work. Separating these axes gives the profiling conversation somewhere concrete to start.

@waterloo_intern Please do more of it

@waterloo_intern i really like whiteboard sessions this is awesome!

@waterloo_intern Wish you would post this on youtube so its easier to share

@waterloo_intern Its on YouTube too, Baseten channel

@waterloo_intern aye, ty. will follow and share. good video btw!