正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now... show more

h100envy

3,740 subscribers

61,470 次观看 • 3 个月前 •via X (Twitter)

科学技术财经教育

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 次观看 • 3 个月前

Ex-Berkeley PhD who leads SGLang at xAI explained how they serve Grok on 100K GPUs in 23 minutes - better than $2000 inference-at-scale courses. split prefill and decode -> shard experts across GPUs -> route tokens per expert -> overlap comm and compute -> serve at DeepSeek-API-killing prices. That loop is why xAI runs Grok on SGLang and third parties beat DeepSeek's own API by 5x on cost. SGLang + prefill-decode disaggregation + expert parallelism + AMD MI300 - that's the stack. Watch and save it, then read the article below.

Ex-Berkeley PhD who leads SGLang at xAI explained how they serve Grok on 100K GPUs in 23 minutes - better than $2000 inference-at-scale courses. split prefill and decode -> shard experts across GPUs -> route tokens per expert -> overlap comm and compute -> serve at DeepSeek-API-killing prices. That loop is why xAI runs Grok on SGLang and third parties beat DeepSeek's own API by 5x on cost. SGLang + prefill-decode disaggregation + expert parallelism + AMD MI300 - that's the stack. Watch and save it, then read the article below.

h100envy

319,581 次观看 • 3 个月前

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 次观看 • 1 个月前

PyTorch core engineer at Meta turned CUDA kernel writing into a sport in 13 minutes - better than $1500 GPU programming bootcamps. profile the kernel -> find the bottleneck -> rewrite -> benchmark -> merge the winning code into PyTorch. That loop is how the open community now beats hand-tuned vendor kernels. GPU MODE community + KernelBot competition + winning kernel merged into the framework - that's the stack. Watch it, then steal the loop below.

h100envy

35,390 次观看 • 3 个月前

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 次观看 • 2 个月前

The creator of Pydantic explained how to make live agents smarter after they ship in 1 hour 21 minutes - better than $2500 agent evals bootcamps. collect traces from your live agent -> score them with an eval -> run GEPA to auto-optimize the prompt -> deploy the winner -> repeat forever. That loop is how Pydantic AI agents self-improve while running in production. Pydantic AI + Logfire tracing + GEPA prompt optimizer + evals + feedback loops - that's the stack. Watch and save it, then wire GEPA into your live agent this week.

The creator of Pydantic explained how to make live agents smarter after they ship in 1 hour 21 minutes - better than $2500 agent evals bootcamps. collect traces from your live agent -> score them with an eval -> run GEPA to auto-optimize the prompt -> deploy the winner -> repeat forever. That loop is how Pydantic AI agents self-improve while running in production. Pydantic AI + Logfire tracing + GEPA prompt optimizer + evals + feedback loops - that's the stack. Watch and save it, then wire GEPA into your live agent this week.

h100envy

49,768 次观看 • 2 个月前

OpenAI engineer explained every form of fine-tuning in 1 hour 46 minutes - better than $3000 fine-tuning bootcamps. pick your task -> try prompting first -> drop to SFT for style -> run DPO for preference -> run RFT for verifiable rewards -> ship a model 10x cheaper than GPT-5. That loop is why teams that fine-tune beat teams that only prompt on cost and speed. Prompting + SFT + DPO + RFT + OpenAI fine-tuning API - that's the stack. Watch and save it, then pick the right fine-tune for your task this week.

OpenAI engineer explained every form of fine-tuning in 1 hour 46 minutes - better than $3000 fine-tuning bootcamps. pick your task -> try prompting first -> drop to SFT for style -> run DPO for preference -> run RFT for verifiable rewards -> ship a model 10x cheaper than GPT-5. That loop is why teams that fine-tune beat teams that only prompt on cost and speed. Prompting + SFT + DPO + RFT + OpenAI fine-tuning API - that's the stack. Watch and save it, then pick the right fine-tune for your task this week.

h100envy

41,245 次观看 • 2 个月前

Ex-JPMorgan engineer who wrote the LLM Course explained everything about fine-tuning and merging in 18 minutes - better than $2500 fine-tuning bootcamps. pick the base -> LoRA or QLoRA -> then DPO or KTO for alignment -> merge two fine-tunes into one stronger model -> ship a model that beats the base on your task. That loop is why Labonne's merges sit at the top of the Hugging Face leaderboard. LoRA + QLoRA + DPO + KTO + mergekit - that's the stack. Watch and save it, then merge your first two fine-tunes this week.

Ex-JPMorgan engineer who wrote the LLM Course explained everything about fine-tuning and merging in 18 minutes - better than $2500 fine-tuning bootcamps. pick the base -> LoRA or QLoRA -> then DPO or KTO for alignment -> merge two fine-tunes into one stronger model -> ship a model that beats the base on your task. That loop is why Labonne's merges sit at the top of the Hugging Face leaderboard. LoRA + QLoRA + DPO + KTO + mergekit - that's the stack. Watch and save it, then merge your first two fine-tunes this week.

h100envy

13,856 次观看 • 2 个月前

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

h100envy

497,378 次观看 • 3 个月前

Google engineer explained how to fine-tune a tiny LLM from 46% to 90% accuracy on your phone in 21 minutes - better than $1500 on-device AI bootcamps. pick Gemma 270M -> generate synthetic task data -> fine-tune with LoRA -> quantize to int4 -> deploy to Pixel and hit 2000 tokens per second. That loop is how a 270M model beats a 70B one on your task, running fully offline in your pocket. Gemma 270M + synthetic data + LoRA + int4 quantization + on-device runtime - that's the stack. Watch and save it, then fine-tune your own tiny agent tonight.

Google engineer explained how to fine-tune a tiny LLM from 46% to 90% accuracy on your phone in 21 minutes - better than $1500 on-device AI bootcamps. pick Gemma 270M -> generate synthetic task data -> fine-tune with LoRA -> quantize to int4 -> deploy to Pixel and hit 2000 tokens per second. That loop is how a 270M model beats a 70B one on your task, running fully offline in your pocket. Gemma 270M + synthetic data + LoRA + int4 quantization + on-device runtime - that's the stack. Watch and save it, then fine-tune your own tiny agent tonight.

h100envy

710,607 次观看 • 2 个月前

Prime Intellect engineers explained how they train reasoning models over the open internet in 30 minutes - better than $3000 distributed training courses. split policy and rollouts across nodes -> run agents in parallel envs -> verify with LLM judges -> gradient-sync over the internet -> train Llama, Qwen, Gemma at cluster scale on rented GPUs. That loop is why open reasoning models are catching closed labs without owning a data center. Prime-RL + verifiers + distributed rollouts + LLM judges + multi-cloud GPUs - that's the stack. Watch and save it, then launch your first distributed RL run this week.

Prime Intellect engineers explained how they train reasoning models over the open internet in 30 minutes - better than $3000 distributed training courses. split policy and rollouts across nodes -> run agents in parallel envs -> verify with LLM judges -> gradient-sync over the internet -> train Llama, Qwen, Gemma at cluster scale on rented GPUs. That loop is why open reasoning models are catching closed labs without owning a data center. Prime-RL + verifiers + distributed rollouts + LLM judges + multi-cloud GPUs - that's the stack. Watch and save it, then launch your first distributed RL run this week.

h100envy

40,022 次观看 • 3 个月前

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

Modal engineer who teaches Full-Stack Deep Learning explained everything an AI engineer needs to know about GPUs in 20 minutes - better than $800 CUDA workshops. memory bandwidth -> SM occupancy -> kernel fusion -> arithmetic intensity -> know when you're compute-bound vs memory-bound. That loop is how you stop overpaying for GPUs you're barely using. VRAM math + roofline model + Modal serverless GPUs + kernel profiling - that's the stack. Watch and save it, then check your own GPU utilization tonight.

h100envy

18,442 次观看 • 3 个月前

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

Antid

23,745 次观看 • 2 个月前

Storage just stole the show. Between KV cache offload and SSD prices going vertical, storage went from afterthought to headliner. Vik Malyala walks through Supermicro's storage stack. "Storage, whether it be for KV cache offload or just the incredible rise in cost of SSDs, is the primary thing to focus on for a lot of users." "Initially it started with the form factors. You have U.2s that were popular, or still are popular for that matter. Then you have E1.S and E3.S drives, mainly because of the density that we can bring in a 1U or 2U form factor. And we have come up with so-called Petascale solutions, which you can get a very good balance between the amount of storage, the storage bandwidth as IO bandwidth." "Hardware is only one part of it, but what we are doing is working with the likes of VAST, WEKA, DDN, Qumulo, OSNEXUS and every one of these software vendors. We are able to bring different types of software defined storage into customer environments so that you don't necessarily get stuck with traditional storage. So super exciting times for that."

Storage just stole the show. Between KV cache offload and SSD prices going vertical, storage went from afterthought to headliner. Vik Malyala walks through Supermicro's storage stack. "Storage, whether it be for KV cache offload or just the incredible rise in cost of SSDs, is the primary thing to focus on for a lot of users." "Initially it started with the form factors. You have U.2s that were popular, or still are popular for that matter. Then you have E1.S and E3.S drives, mainly because of the density that we can bring in a 1U or 2U form factor. And we have come up with so-called Petascale solutions, which you can get a very good balance between the amount of storage, the storage bandwidth as IO bandwidth." "Hardware is only one part of it, but what we are doing is working with the likes of VAST, WEKA, DDN, Qumulo, OSNEXUS and every one of these software vendors. We are able to bring different types of software defined storage into customer environments so that you don't necessarily get stuck with traditional storage. So super exciting times for that."

SemiAnalysis

61,919 次观看 • 2 个月前

The Cost of Intelligence is Heading to Zero | Hyperspace P2P Distributed Cache We present to you our breakthrough cross-domain work across AI, distributed systems, cryptography, game theory to solve the primary structural inefficiency at the heart of AI infrastructure: most inference is redundant. Google has reported that only 15% of daily searches are truly novel. The rest are repeats or close variants. LLM inference inherits this same power-law distribution. Enterprise chatbots see 70-80% of queries fall into a handful of intent categories. System prompts are identical across 100% of requests within an application. The KV attention state for "You are a helpful assistant" has been computed billions of times, on millions of GPUs, identically. And yet every AI lab, every startup, every self-hosted deployment - computes and caches these results independently. There is no shared layer. No global memory. Every provider pays the full compute cost for every query, even when the answer already exists somewhere in the network. This is the problem Hyperspace solves where distributed cache operates at three levels, each catching a different class of redundancy: 1. Response cache Same prompt, same model, same parameters - instant cached response from any node in the network. SHA-256 hash lookup via DHT, with cryptographic cache proofs linking every response to its original inference execution. No trust required. Fetchers re-announce as providers, so popular responses replicate naturally across more nodes. 2. KV prefix cache Same system prompt tokens - skip the most expensive part of inference entirely. Prefill (computing Key-Value attention states) is deterministic: same model plus same tokens always produces identical KV state. The network caches these states using erasure coding and distributes them via the routing network. New questions that share a common prefix resume generation from cached state instead of recomputing from scratch. 3. Routing to cached nodes Instead of transferring KV state across the network for every request, Hyperspace routes the request to the node that already has the state loaded in VRAM. The request goes to the cache, not the cache to the request. Together, these three layers mean that 70-90% of inference requests at network scale never require full GPU computation. This work doesn't exist in isolation. It builds on research from across the industry: SGLang's RadixAttention demonstrated that automatic prefix sharing can yield up to 5x speedup on structured LLM workloads. Moonshot AI's Mooncake built an entire KV-cache-centric disaggregated architecture for production serving at Kimi. Anthropic, OpenAI, and Google all launched prompt caching products in 2024 - priced at 50-90% discounts - because system prompt reuse is so pervasive that it changes the economics of inference. What all of these systems share is a common limitation: they operate within a single organization's infrastructure. SGLang caches prefixes within one server. Mooncake disaggregates KV cache within one datacenter. Anthropic's prompt caching works within one API provider's fleet. None of them can share cached state across organizational boundaries. Hyperspace removes this boundary. The cache is global. A response computed by a node in Tokyo is immediately available to a node in Berlin. A KV prefix state generated for Qwen-32B on one machine is verifiable and reusable by any other machine running the same model. The routing network provides the delivery guarantees, the erasure coding provides the redundancy, and the cache proofs provide the trust. What this means for the cost of intelligence Big AI labs scale linearly: twice the users means twice the GPU spend. Every query is a cost center. Their internal caching helps, but it's siloed - Lab A's cache can't serve Lab B's users, and neither can serve a self-hosted Llama deployment. Hyperspace scales sub-linearly. Every new node that joins the network adds to the global cache. Every inference result enriches the cache for all future requests. The cache hit rate rises with network size because query distributions follow a power law - the most common questions are asked exponentially more often than rare ones. The implication is simple: as the network grows, the effective cost per inference drops. Not linearly. Logarithmically. At 10 million nodes, we estimate 75-90% of all inference requests can be served from cache, eliminating 400,000+ MWh of energy consumption per year and avoiding over 200,000 tons of CO2 emissions. The first person to ask a question pays the compute cost. Everyone after them gets the answer for free, with cryptographic proof that it's authentic. Training is competitive. Inference is shared Open-weight models are converging on quality with closed models. Labs will continue to differentiate on training - data curation, architecture innovation, RLHF tuning. That's where the real intellectual property lives. But inference is a commodity. Two copies of Qwen-32B running the same prompt produce the same KV state and the same response, byte for byte, regardless of whose GPU runs the matrix multiplication. There is no moat in multiplying matrices. The moat is in training the weights. A global distributed cache makes this separation explicit. It doesn't matter who trained the model. Once the weights are open, the inference cost approaches zero at scale - because the network remembers every answer and can prove it's correct. No lab, no matter how well-funded, can match this. They cannot share caches across competitors. They scale linearly. The network scales logarithmically. The marginal cost of intelligence approaches zero. That's the endgame.

The Cost of Intelligence is Heading to Zero | Hyperspace P2P Distributed Cache We present to you our breakthrough cross-domain work across AI, distributed systems, cryptography, game theory to solve the primary structural inefficiency at the heart of AI infrastructure: most inference is redundant. Google has reported that only 15% of daily searches are truly novel. The rest are repeats or close variants. LLM inference inherits this same power-law distribution. Enterprise chatbots see 70-80% of queries fall into a handful of intent categories. System prompts are identical across 100% of requests within an application. The KV attention state for "You are a helpful assistant" has been computed billions of times, on millions of GPUs, identically. And yet every AI lab, every startup, every self-hosted deployment - computes and caches these results independently. There is no shared layer. No global memory. Every provider pays the full compute cost for every query, even when the answer already exists somewhere in the network. This is the problem Hyperspace solves where distributed cache operates at three levels, each catching a different class of redundancy: 1. Response cache Same prompt, same model, same parameters - instant cached response from any node in the network. SHA-256 hash lookup via DHT, with cryptographic cache proofs linking every response to its original inference execution. No trust required. Fetchers re-announce as providers, so popular responses replicate naturally across more nodes. 2. KV prefix cache Same system prompt tokens - skip the most expensive part of inference entirely. Prefill (computing Key-Value attention states) is deterministic: same model plus same tokens always produces identical KV state. The network caches these states using erasure coding and distributes them via the routing network. New questions that share a common prefix resume generation from cached state instead of recomputing from scratch. 3. Routing to cached nodes Instead of transferring KV state across the network for every request, Hyperspace routes the request to the node that already has the state loaded in VRAM. The request goes to the cache, not the cache to the request. Together, these three layers mean that 70-90% of inference requests at network scale never require full GPU computation. This work doesn't exist in isolation. It builds on research from across the industry: SGLang's RadixAttention demonstrated that automatic prefix sharing can yield up to 5x speedup on structured LLM workloads. Moonshot AI's Mooncake built an entire KV-cache-centric disaggregated architecture for production serving at Kimi. Anthropic, OpenAI, and Google all launched prompt caching products in 2024 - priced at 50-90% discounts - because system prompt reuse is so pervasive that it changes the economics of inference. What all of these systems share is a common limitation: they operate within a single organization's infrastructure. SGLang caches prefixes within one server. Mooncake disaggregates KV cache within one datacenter. Anthropic's prompt caching works within one API provider's fleet. None of them can share cached state across organizational boundaries. Hyperspace removes this boundary. The cache is global. A response computed by a node in Tokyo is immediately available to a node in Berlin. A KV prefix state generated for Qwen-32B on one machine is verifiable and reusable by any other machine running the same model. The routing network provides the delivery guarantees, the erasure coding provides the redundancy, and the cache proofs provide the trust. What this means for the cost of intelligence Big AI labs scale linearly: twice the users means twice the GPU spend. Every query is a cost center. Their internal caching helps, but it's siloed - Lab A's cache can't serve Lab B's users, and neither can serve a self-hosted Llama deployment. Hyperspace scales sub-linearly. Every new node that joins the network adds to the global cache. Every inference result enriches the cache for all future requests. The cache hit rate rises with network size because query distributions follow a power law - the most common questions are asked exponentially more often than rare ones. The implication is simple: as the network grows, the effective cost per inference drops. Not linearly. Logarithmically. At 10 million nodes, we estimate 75-90% of all inference requests can be served from cache, eliminating 400,000+ MWh of energy consumption per year and avoiding over 200,000 tons of CO2 emissions. The first person to ask a question pays the compute cost. Everyone after them gets the answer for free, with cryptographic proof that it's authentic. Training is competitive. Inference is shared Open-weight models are converging on quality with closed models. Labs will continue to differentiate on training - data curation, architecture innovation, RLHF tuning. That's where the real intellectual property lives. But inference is a commodity. Two copies of Qwen-32B running the same prompt produce the same KV state and the same response, byte for byte, regardless of whose GPU runs the matrix multiplication. There is no moat in multiplying matrices. The moat is in training the weights. A global distributed cache makes this separation explicit. It doesn't matter who trained the model. Once the weights are open, the inference cost approaches zero at scale - because the network remembers every answer and can prove it's correct. No lab, no matter how well-funded, can match this. They cannot share caches across competitors. They scale linearly. The network scales logarithmically. The marginal cost of intelligence approaches zero. That's the endgame.

Varun

37,555 次观看 • 6 个月前

Alibaba engineer who leads Qwen explained the future of open agent models in 25 minutes - better than $2000 LLM training courses. pre-train the base ->SFT -> RLHF -> tool use -> multi-modal -> ship a whole family (chat / VL / coder / math / QwQ). That loop is why Qwen quietly became the most downloaded open model family on Hugging Face. Qwen base + Qwen-VL + Qwen-Coder + QwQ reasoning - that's the stack. Watch and save it, then read the article below.

Alibaba engineer who leads Qwen explained the future of open agent models in 25 minutes - better than $2000 LLM training courses. pre-train the base ->SFT -> RLHF -> tool use -> multi-modal -> ship a whole family (chat / VL / coder / math / QwQ). That loop is why Qwen quietly became the most downloaded open model family on Hugging Face. Qwen base + Qwen-VL + Qwen-Coder + QwQ reasoning - that's the stack. Watch and save it, then read the article below.

h100envy

114,423 次观看 • 3 个月前

Liquid AI's head of post-training explained how they built a small model that runs on-device under 1 GB in 20 minutes - better than $2500 small-model bootcamps. pick LFM2.5 base -> on-policy preference alignment -> agentic reinforcement learning -> curriculum training -> iterative model merging -> ship a 1B model that reliably calls tools on your phone. That loop is why frontier small models now beat 70B models on the tasks that actually matter. LFM2.5 + on-policy DPO + agentic RL + curriculum training + iterative merging - that's the stack. Watch and save it, then run a 1B agent on your phone tonight.

Liquid AI's head of post-training explained how they built a small model that runs on-device under 1 GB in 20 minutes - better than $2500 small-model bootcamps. pick LFM2.5 base -> on-policy preference alignment -> agentic reinforcement learning -> curriculum training -> iterative model merging -> ship a 1B model that reliably calls tools on your phone. That loop is why frontier small models now beat 70B models on the tasks that actually matter. LFM2.5 + on-policy DPO + agentic RL + curriculum training + iterative merging - that's the stack. Watch and save it, then run a 1B agent on your phone tonight.

h100envy

87,975 次观看 • 2 个月前

An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usually produces long rollouts, and when a rollout reaches its context budget, a compaction policy removes older turns and keeps the prompt, recent turns, or a summary. Most systems use re-prefill compaction. They create a shorter sequence and run every retained token through the model again. This rebuilds the KV cache after each compaction, so the same tokens may be processed several times. KV-streams is a new approach to solve this. Instead of rebuilding the shorter sequence, KV-streams removes discarded turns directly from the live KV cache. Generation continues from the entries that survived. In practice, you cannot just reuse those entries as is. It requires two fixes. > First, RoPE has already rotated each cached Key using its original token position. A Key may move from physical slot 1,000 to slot 500 after compaction, but the model must still treat it as position 1,000. KV-streams keeps physical storage and logical position separate. Retained Keys keep their rotations. New Queries and Keys continue from the original position count. > Second, vLLM does not store each KV entry independently. It groups them into fixed 16-token blocks and uses a block table to find those blocks in GPU memory. KV-streams cannot leave empty slots inside that table because the attention kernel may read the wrong memory. It therefore removes complete blocks and joins the remaining block references together. This introduces another problem. A conversation turn may end halfway through a 16-token block. In their vLLM setup, that partial block is not committed to the prefix cache and must be recomputed when the next turn arrives. If compaction happens first, those tokens may have originally attended to context that is no longer present. For the vLLM experiments, the researchers pad each completion to a multiple of 16 tokens. This commits every generated token to the cache before the next turn or compaction event. The SGLang setup uses single-token blocks and needs no padding. Separately, the researchers found that token-level eviction produced degenerate text and required substantial supervised fine-tuning. They suspect that removing message-boundary tokens disrupted the chat format learned during training. The agent experiments therefore remove complete turns, including their boundary tags. Once inference works this way, training must follow the same process. A normal trainer would either process the full rollout or rebuild the shorter sequence after compaction. Both options produce different KV entries from those used during generation. KV-streams records every compaction event during generation. The trainer then reproduces that history: retained tokens keep the states they formed before compaction, while later tokens are blocked from attending to spans that inference had already removed. In the repository’s validation runs, the difference between training and generation stayed near a mismatch KL of 0.001, indicating that both paths produced almost identical token probabilities. On SWE-bench Verified, the Markovian Thinker setup with KV-streams reached peak performance in about 32 hours. The same compaction strategy with re-prefill took roughly 60 hours, while both reached comparable final scores. The speedup mainly applies to RL training, where one rollout may be compacted several times. At production inference, batching can hide some prefill cost, and keeping caches for inactive users reduces serving capacity. There is one more detail. The retained KV entries were created while the removed turns were still visible. Future tokens cannot attend to those turns directly, but some of their influence may remain in the entries that were not evicted. In a controlled test, RL recovered an assignment after its original message was removed. Recall reached 100% when enough cache remained. KV-streams does not decide what the model should forget. Instead, it makes existing compaction policies cheaper to train by preserving the KV state they choose to retain. Repo: If you want to dive deeper into the broader KV-cache engineering stack, I wrote an article that covers eviction, paging, quantization, offloading, and architectural compression. Read it below.

An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usually produces long rollouts, and when a rollout reaches its context budget, a compaction policy removes older turns and keeps the prompt, recent turns, or a summary. Most systems use re-prefill compaction. They create a shorter sequence and run every retained token through the model again. This rebuilds the KV cache after each compaction, so the same tokens may be processed several times. KV-streams is a new approach to solve this. Instead of rebuilding the shorter sequence, KV-streams removes discarded turns directly from the live KV cache. Generation continues from the entries that survived. In practice, you cannot just reuse those entries as is. It requires two fixes. > First, RoPE has already rotated each cached Key using its original token position. A Key may move from physical slot 1,000 to slot 500 after compaction, but the model must still treat it as position 1,000. KV-streams keeps physical storage and logical position separate. Retained Keys keep their rotations. New Queries and Keys continue from the original position count. > Second, vLLM does not store each KV entry independently. It groups them into fixed 16-token blocks and uses a block table to find those blocks in GPU memory. KV-streams cannot leave empty slots inside that table because the attention kernel may read the wrong memory. It therefore removes complete blocks and joins the remaining block references together. This introduces another problem. A conversation turn may end halfway through a 16-token block. In their vLLM setup, that partial block is not committed to the prefix cache and must be recomputed when the next turn arrives. If compaction happens first, those tokens may have originally attended to context that is no longer present. For the vLLM experiments, the researchers pad each completion to a multiple of 16 tokens. This commits every generated token to the cache before the next turn or compaction event. The SGLang setup uses single-token blocks and needs no padding. Separately, the researchers found that token-level eviction produced degenerate text and required substantial supervised fine-tuning. They suspect that removing message-boundary tokens disrupted the chat format learned during training. The agent experiments therefore remove complete turns, including their boundary tags. Once inference works this way, training must follow the same process. A normal trainer would either process the full rollout or rebuild the shorter sequence after compaction. Both options produce different KV entries from those used during generation. KV-streams records every compaction event during generation. The trainer then reproduces that history: retained tokens keep the states they formed before compaction, while later tokens are blocked from attending to spans that inference had already removed. In the repository’s validation runs, the difference between training and generation stayed near a mismatch KL of 0.001, indicating that both paths produced almost identical token probabilities. On SWE-bench Verified, the Markovian Thinker setup with KV-streams reached peak performance in about 32 hours. The same compaction strategy with re-prefill took roughly 60 hours, while both reached comparable final scores. The speedup mainly applies to RL training, where one rollout may be compacted several times. At production inference, batching can hide some prefill cost, and keeping caches for inactive users reduces serving capacity. There is one more detail. The retained KV entries were created while the removed turns were still visible. Future tokens cannot attend to those turns directly, but some of their influence may remain in the entries that were not evicted. In a controlled test, RL recovered an assignment after its original message was removed. Recall reached 100% when enough cache remained. KV-streams does not decide what the model should forget. Instead, it makes existing compaction policies cheaper to train by preserving the KV state they choose to retain. Repo: If you want to dive deeper into the broader KV-cache engineering stack, I wrote an article that covers eviction, paging, quantization, offloading, and architectural compression. Read it below.

Avi Chawla

18,120 次观看 • 11 天前

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 次观看 • 4 个月前

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

102,747 次观看 • 6 个月前