bro casually walks and explains 5 GPU performance optimization methods for LLMs. one of the most simple and intuitive explanations for beginners.

the most casual conversation in SF

@lindavivah @robertnishihara If you can't explain PagedAttention or Quantization while dodging traffic on a sidewalk, you don't understand it well enough. The industry is full of 40-page whitepapers that say less than this guy says in a 2-minute walk. "Casual" is the new "Expert."

@lindavivah @robertnishihara this must be an interview standard take a walk with the HR

@robertnishihara @robertnishihara is the best! ⭐️

@robertnishihara you can see his mastery by just the way he explains

imagine having your feed full of vids like this absolute brain rot

@lindavivah @robertnishihara did i just get bamboozled into watching an ad for ray

Here’s the clean, structured version of what you’re saying, without the vlog artifacts and with the technical distinctions made explicit. ⸻ LLM inference vs. “regular” inference 1. Variable-length computation and continuous batching Traditional inference (e.g. CNNs for vision) assumes: •fixed-size inputs •fixed compute graphs •fixed latency per request LLMs break all of that: •prompts vary in length •outputs vary in length •total compute per request is unknown upfront Batching is the main lever for inference efficiency, but variable-length requests finish at different times. You can’t just wait for the slowest one without wasting GPU cycles. LLM systems therefore rely on continuous batching: dynamically inserting and removing requests from a batch as tokens complete. This problem simply doesn’t exist for fixed-shape models. ⸻ 2. Two fundamentally different phases: prefill vs. decode LLM inference is split into two stages: •Prefill: process the full input prompt •Decode: generate tokens one at a time These have very different characteristics: •Prefill is typically compute-bound •Decode is typically memory-bandwidth-bound Running both on the same GPUs causes interference and latency jitter. High-performance systems often use prefill–decode disaggregation, running them on separate compute pools and transferring state between them. This separation is largely irrelevant for conventional models. ⸻ 3. GPU memory management and KV caching LLMs depend heavily on caching: •Multi-turn conversations reuse shared prefixes •Intermediate attention states (KV cache) must persist across turns This introduces hard problems: •What to cache vs. recompute •When to evict or swap cache entries •How to avoid GPU memory fragmentation Modern LLM serving borrows ideas from virtual memory (paged KV caches) to manage this complexity efficiently. Regular models don’t have long-lived, request-specific internal state, so this entire class of problems barely exists there. ⸻ 4. Prefix-aware routing Scaling non-LLM inference usually means: •replicate the model •route requests round-robin or by load LLMs benefit from routing based on cached prefixes: •a request should go to the replica that already holds its KV cache •naive load balancing destroys cache locality and performance This forces routing logic to become state-aware rather than stateless. That’s a major architectural difference. ⸻ 5. Sharding and Mixture-of-Experts (MoE) Most classical models scale by replication. LLMs increasingly use Mixture-of-Experts architectures: •attention layers shared across GPUs •expert layers sharded across GPUs •tokens dynamically routed to experts during inference This results in: •one logical model spread across many devices •fine-grained, data-dependent routing inside a single request That’s not “many replicas.” It’s one distributed model with internal traffic patterns that change per token. ⸻ Why specialized LLM inference engines exist These differences are why general inference runtimes aren’t enough. Engines like vLLM, SGLang, and TensorRT-LLM exist specifically to handle: •continuous batching •KV cache management •prefix-aware routing •prefill/decode separation •MoE sharding Systems like Ray sit above these to orchestrate them at scale. ⸻ LLM inference isn’t just “bigger models.” It’s a different systems problem.

@lindavivah @robertnishihara What is this from and how do I watch more of these?

@lindavivah @robertnishihara Insane - probably the best explanation I’ve seen so far. It’s funny how you can totally ace this walking down the street in NYC. More proof you don’t need a podcast studio, lights, etc. it’s all about who is doing the explaining and how cracked they are 🔥

@lindavivah @robertnishihara That's no ordinary bro. That's one of the nicest and coolest CEOs in silicon valley. I've spoken with him and know people that know him well. Robert is a class act. Highly recommend his company, if you haven't used it.

@lindavivah @robertnishihara this is a great format normalize outdoor podcasts

@lindavivah @robertnishihara I understood 2.3% of this

@lindavivah @robertnishihara Meh. Very little of this stuff existed 10 minutes ago and most won’t exist 10 minutes from now, but I guess it’s cool to hyper specialize on the transitory

@lindavivah @robertnishihara Inference is inference.

@lindavivah @robertnishihara This is my favorite genre of explanation — zero slides, zero buzzwords, just vibes and first principles. If more ML content sounded like this, way fewer people would be intimidated.

@lindavivah @robertnishihara Brother casually pulls out all these facts, puts it together on the fly and does it in the most non chalant way possible

@lindavivah @robertnishihara He is the cofounder of Ray/anyscale which is used for ML training

@lindavivah @robertnishihara Sounds like a lot of concepts repurposed from operating systems.

@lindavivah @robertnishihara This was awesome. He’s gives a perfect amount of context

@lindavivah @robertnishihara If you cannot translate/word your response depending on the expertise of the person asking, you have not mastered your subject. Well done 👍

@lindavivah @robertnishihara Huge respect to @robertnishihara big fan of ray

@lindavivah @robertnishihara nice

@lindavivah @robertnishihara With a phd I’d hope he be able to explain it this way

@lindavivah @robertnishihara That's how we look when we're asked about a topic we're very passionate about

@lindavivah @robertnishihara Tell me you’re on a Hinge date in SF without telling me you’re on a Hibge date in SF

@lindavivah @robertnishihara Some baddie could go wife that guy up and be set for life. Somewhat surprised the host doesn’t to be honest. I’m turned on just listening to him.

@lindavivah @robertnishihara Bro casually explained these concepts while dodging traffic

@lindavivah @robertnishihara @robertnishihara are you doing more of these? That was super helpful

@lindavivah @robertnishihara Sounds a lot like they need middle out processing and arrange the data tip to tip facing each other to maintain optimal load batches

Spot on – Robert Nishihara dropping pure gold while strolling through the city. His breakdown of continuous batching, prefill/decode disaggregation, paged attention, prefix caching, and MoE sharding is one of the clearest intros to LLM serving optimizations I've seen. Everything he said checks out perfectly with current best practices in vLLM, Ray Serve, etc. Super accessible for beginners but still spot-on technically. More walking tech talks please! 🚀

@lindavivah @robertnishihara Man is a G

@lindavivah @robertnishihara This is one of the best explanations I've seen so far!

@lindavivah @robertnishihara Just made me realise , I know nothing about ml ops, off to learning it.

@lindavivah @robertnishihara OK, dude, you are hired!

@lindavivah @robertnishihara is this guy human? 😵‍💫

@lindavivah @robertnishihara I hope bro gets laid