Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

@TryTrustAI is building the fastest, cheapest lightweight inference by optimizing KV cache at Y Combinator. Our group of MIT grads and olympiad medalists from Google DeepMind, Jane Street, and NeurIPS/ICML publishers are setting a new standard for cost and latency efficiency.

Hannah Chung

1,867 subscribers

33,169 просмотров • 25 дней назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 36

Фото профиля Claire Mao

Claire Mao25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv WOWW so impressive congrats!!

Фото профиля Justin Hong

Justin Hong25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv been waiting for this

Фото профиля Hannah Chung

Hannah Chung25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv me too 🥲

Фото профиля medha

medha25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup Exciting!

Фото профиля Hannah Chung

Hannah Chung25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup I hope you're hyped

Фото профиля Jeremy

Jeremy25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv So gmi 🚀

Фото профиля Nour Zahzah

Nour Zahzah25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Let’s go

Фото профиля Shaurya Aggarwal

Shaurya Aggarwal25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv cracked team! congrats guys

Фото профиля Hannah Chung

Hannah Chung25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv thank you!!!

Фото профиля aryan mahajan

aryan mahajan25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv 🙌

Фото профиля Hannah Chung

Hannah Chung25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv 🙏

Фото профиля Sumit Singh🇮🇳

Sumit Singh🇮🇳25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Soo great 🤝👨🏻‍💻🌐 looking to know more . Let's connect

Фото профиля Nikhil Gupta

Nikhil Gupta24 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv wait IPhO camp does not the mean medalist

Фото профиля Rick A.F.

Rick A.F.25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Amazing, cache hits different when its optimised! Lets go, lets improve the efficiency of interference!

Фото профиля Akira

Akira25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Nice

Фото профиля Ruiyang Chen

Ruiyang Chen25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Curious what’s the use case?

Фото профиля Evan Dickinson

Evan Dickinson25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Mit!

Фото профиля Luke Gainey

Luke Gainey24 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv FANTASTIC

Фото профиля Dheeraj Kaim

Dheeraj Kaim25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv congrats, hope it works out for you guys

Фото профиля An Engineer's Log

An Engineer's Log25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv what's the largest model you can host, and the price per million token for it ?

Фото профиля Rahul Wagh

Rahul Wagh25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv interesting!!

Фото профиля om_lathiya_

om_lathiya_25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Interesting

Фото профиля Evï Anton

Evï Anton25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv is this any different from

Фото профиля Niraj Adhikari

Niraj Adhikari25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Looks good!

Фото профиля just a monolith

just a monolith25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv I have nothing to add. But goddamn, you're cute

Фото профиля Marco H

Marco H25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv So interesting.. i was just running research on this as I just heard KV cache for first time today and algorithm is showing it to me now.

Фото профиля Yoonseok Yang

Yoonseok Yang25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv lets go hannah!!

Фото профиля Santz

Santz24 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Congrats on the launch!

Фото профиля John

John25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Dope

Фото профиля AkinAyo | UI/UX Designer

AkinAyo | UI/UX Designer25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Impressive

Фото профиля 𝕐𝕖𝕤𝕙𝕦𝕒

𝕐𝕖𝕤𝕙𝕦𝕒24 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Great team 👏🏻💯

Фото профиля E

E25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Bubble

Фото профиля mr ceo

mr ceo25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Good luck

Фото профиля Chandan kumar

Chandan kumar25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Hii

Фото профиля Lokesh

Lokesh21 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Any opening role in product management with visa sponsorship?

Фото профиля Amritesh Anand

Amritesh Anand25 дней назад

@TryTrustAI @ycombinator @GoogleDeepMind @JaneStreetGroup @medha_rv Is this something we can try out on our GPU's or you serve endpoints to end clients?

Похожие видео

OneTriangle (YC S26) (YC S26) is building the fastest lightweight, cheap inference and hosting DeepSeek V4 Flash today at $0.15/M input and $0.35/M output tokens. Live at Our group of Massachusetts Institute of Technology (MIT) grads and olympiad medalists from Google DeepMind, Jane Street, and NeurIPS Conference /ICML publishers are setting a new standard for cost and latency efficiency using KV cache optimizations.

OneTriangle (YC S26) (YC S26) is building the fastest lightweight, cheap inference and hosting DeepSeek V4 Flash today at $0.15/M input and $0.35/M output tokens. Live at Our group of Massachusetts Institute of Technology (MIT) grads and olympiad medalists from Google DeepMind, Jane Street, and NeurIPS Conference /ICML publishers are setting a new standard for cost and latency efficiency using KV cache optimizations.

Hannah Chung

94,626 просмотров • 21 дней назад

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 просмотров • 4 месяцев назад

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

Alec Helbling

36,159 просмотров • 11 дней назад

$We partnered with Harvey to train a model for Review Table, achieving state-of-the-art accuracy at a fraction of the cost and latency of frontier alternatives. Review Table is one of Harvey’s highest inference volume products. Training an open source model for production traffic required building a representative eval and understanding how real lawyers use the product. Our teams discuss the research process.$

We partnered with Harvey to train a model for Review Table, achieving state-of-the-art accuracy at a fraction of the cost and latency of frontier alternatives. Review Table is one of Harvey’s highest inference volume products. Training an open source model for production traffic required building a representative eval and understanding how real lawyers use the product. Our teams discuss the research process.

Applied Compute

33,776 просмотров • 1 месяц назад

AI that responds in under 100ms doesn't happen by accident. Together AI VP of Kernels Dan Fu shares how they use NVIDIA CUDA, TensorRT-LLM, Dynamo and Together ATLAS on NVIDIA Blackwell to power ultra-low-latency inference and long context code generation for Cursor. From building megakernels that fuse an entire model's forward pass into a single launch to enabling real-time voice agents, Together AI is optimizing latency at every layer of the stack.

AI that responds in under 100ms doesn't happen by accident. Together AI VP of Kernels Dan Fu shares how they use NVIDIA CUDA, TensorRT-LLM, Dynamo and Together ATLAS on NVIDIA Blackwell to power ultra-low-latency inference and long context code generation for Cursor. From building megakernels that fuse an entire model's forward pass into a single launch to enabling real-time voice agents, Together AI is optimizing latency at every layer of the stack.

NVIDIA AI Infrastructure

20,712 просмотров • 2 месяцев назад

Realtime-VLA FLASH tackles one of the biggest deployment bottlenecks for diffusion-based VLAs: inference latency. The key idea is speculative inference for flow-matching VLAs. A lightweight draft model predicts an action chunk, while the main model’s Action Expert verifies it in parallel using flow-consistency checks instead of running full denoising every replanning round. This lets the system replace many expensive 58 ms full inference rounds with speculative rounds as fast as 7.8 ms, reducing average latency to 19.1 ms and achieving a 3.04× speedup on LIBERO while largely preserving success rate. Interesting systems insight: they profile π0 and show VLM prefill is compute-bound, while Action Denoise is memory-bound. FLASH exploits this by reusing KV cache and parallelizing verification instead of repeatedly running sequential denoising.

Robots Digest 🤖

14,576 просмотров • 4 месяцев назад

A 2.7x cost-per-token reduction on the same infrastructure, just from optimizing memory tiering and KV cache offload. That's the kind of number CIOs bring to the board, not a roadmap slide. At HPE Discover 2026, David Nicholson and Matthew Kimball sat down with Justin McGarry of HPE and Justin Christiansen of Intel to break down what's actually driving enterprise AI infrastructure decisions, including a new edge device running 80B parameter models from under an airplane seat.

A 2.7x cost-per-token reduction on the same infrastructure, just from optimizing memory tiering and KV cache offload. That's the kind of number CIOs bring to the board, not a roadmap slide. At HPE Discover 2026, David Nicholson and Matthew Kimball sat down with Justin McGarry of HPE and Justin Christiansen of Intel to break down what's actually driving enterprise AI infrastructure decisions, including a new edge device running 80B parameter models from under an airplane seat.

Six Five Media

100,819 просмотров • 2 месяцев назад

The Cost of Intelligence is Heading to Zero | Hyperspace P2P Distributed Cache We present to you our breakthrough cross-domain work across AI, distributed systems, cryptography, game theory to solve the primary structural inefficiency at the heart of AI infrastructure: most inference is redundant. Google has reported that only 15% of daily searches are truly novel. The rest are repeats or close variants. LLM inference inherits this same power-law distribution. Enterprise chatbots see 70-80% of queries fall into a handful of intent categories. System prompts are identical across 100% of requests within an application. The KV attention state for "You are a helpful assistant" has been computed billions of times, on millions of GPUs, identically. And yet every AI lab, every startup, every self-hosted deployment - computes and caches these results independently. There is no shared layer. No global memory. Every provider pays the full compute cost for every query, even when the answer already exists somewhere in the network. This is the problem Hyperspace solves where distributed cache operates at three levels, each catching a different class of redundancy: 1. Response cache Same prompt, same model, same parameters - instant cached response from any node in the network. SHA-256 hash lookup via DHT, with cryptographic cache proofs linking every response to its original inference execution. No trust required. Fetchers re-announce as providers, so popular responses replicate naturally across more nodes. 2. KV prefix cache Same system prompt tokens - skip the most expensive part of inference entirely. Prefill (computing Key-Value attention states) is deterministic: same model plus same tokens always produces identical KV state. The network caches these states using erasure coding and distributes them via the routing network. New questions that share a common prefix resume generation from cached state instead of recomputing from scratch. 3. Routing to cached nodes Instead of transferring KV state across the network for every request, Hyperspace routes the request to the node that already has the state loaded in VRAM. The request goes to the cache, not the cache to the request. Together, these three layers mean that 70-90% of inference requests at network scale never require full GPU computation. This work doesn't exist in isolation. It builds on research from across the industry: SGLang's RadixAttention demonstrated that automatic prefix sharing can yield up to 5x speedup on structured LLM workloads. Moonshot AI's Mooncake built an entire KV-cache-centric disaggregated architecture for production serving at Kimi. Anthropic, OpenAI, and Google all launched prompt caching products in 2024 - priced at 50-90% discounts - because system prompt reuse is so pervasive that it changes the economics of inference. What all of these systems share is a common limitation: they operate within a single organization's infrastructure. SGLang caches prefixes within one server. Mooncake disaggregates KV cache within one datacenter. Anthropic's prompt caching works within one API provider's fleet. None of them can share cached state across organizational boundaries. Hyperspace removes this boundary. The cache is global. A response computed by a node in Tokyo is immediately available to a node in Berlin. A KV prefix state generated for Qwen-32B on one machine is verifiable and reusable by any other machine running the same model. The routing network provides the delivery guarantees, the erasure coding provides the redundancy, and the cache proofs provide the trust. What this means for the cost of intelligence Big AI labs scale linearly: twice the users means twice the GPU spend. Every query is a cost center. Their internal caching helps, but it's siloed - Lab A's cache can't serve Lab B's users, and neither can serve a self-hosted Llama deployment. Hyperspace scales sub-linearly. Every new node that joins the network adds to the global cache. Every inference result enriches the cache for all future requests. The cache hit rate rises with network size because query distributions follow a power law - the most common questions are asked exponentially more often than rare ones. The implication is simple: as the network grows, the effective cost per inference drops. Not linearly. Logarithmically. At 10 million nodes, we estimate 75-90% of all inference requests can be served from cache, eliminating 400,000+ MWh of energy consumption per year and avoiding over 200,000 tons of CO2 emissions. The first person to ask a question pays the compute cost. Everyone after them gets the answer for free, with cryptographic proof that it's authentic. Training is competitive. Inference is shared Open-weight models are converging on quality with closed models. Labs will continue to differentiate on training - data curation, architecture innovation, RLHF tuning. That's where the real intellectual property lives. But inference is a commodity. Two copies of Qwen-32B running the same prompt produce the same KV state and the same response, byte for byte, regardless of whose GPU runs the matrix multiplication. There is no moat in multiplying matrices. The moat is in training the weights. A global distributed cache makes this separation explicit. It doesn't matter who trained the model. Once the weights are open, the inference cost approaches zero at scale - because the network remembers every answer and can prove it's correct. No lab, no matter how well-funded, can match this. They cannot share caches across competitors. They scale linearly. The network scales logarithmically. The marginal cost of intelligence approaches zero. That's the endgame.

The Cost of Intelligence is Heading to Zero | Hyperspace P2P Distributed Cache We present to you our breakthrough cross-domain work across AI, distributed systems, cryptography, game theory to solve the primary structural inefficiency at the heart of AI infrastructure: most inference is redundant. Google has reported that only 15% of daily searches are truly novel. The rest are repeats or close variants. LLM inference inherits this same power-law distribution. Enterprise chatbots see 70-80% of queries fall into a handful of intent categories. System prompts are identical across 100% of requests within an application. The KV attention state for "You are a helpful assistant" has been computed billions of times, on millions of GPUs, identically. And yet every AI lab, every startup, every self-hosted deployment - computes and caches these results independently. There is no shared layer. No global memory. Every provider pays the full compute cost for every query, even when the answer already exists somewhere in the network. This is the problem Hyperspace solves where distributed cache operates at three levels, each catching a different class of redundancy: 1. Response cache Same prompt, same model, same parameters - instant cached response from any node in the network. SHA-256 hash lookup via DHT, with cryptographic cache proofs linking every response to its original inference execution. No trust required. Fetchers re-announce as providers, so popular responses replicate naturally across more nodes. 2. KV prefix cache Same system prompt tokens - skip the most expensive part of inference entirely. Prefill (computing Key-Value attention states) is deterministic: same model plus same tokens always produces identical KV state. The network caches these states using erasure coding and distributes them via the routing network. New questions that share a common prefix resume generation from cached state instead of recomputing from scratch. 3. Routing to cached nodes Instead of transferring KV state across the network for every request, Hyperspace routes the request to the node that already has the state loaded in VRAM. The request goes to the cache, not the cache to the request. Together, these three layers mean that 70-90% of inference requests at network scale never require full GPU computation. This work doesn't exist in isolation. It builds on research from across the industry: SGLang's RadixAttention demonstrated that automatic prefix sharing can yield up to 5x speedup on structured LLM workloads. Moonshot AI's Mooncake built an entire KV-cache-centric disaggregated architecture for production serving at Kimi. Anthropic, OpenAI, and Google all launched prompt caching products in 2024 - priced at 50-90% discounts - because system prompt reuse is so pervasive that it changes the economics of inference. What all of these systems share is a common limitation: they operate within a single organization's infrastructure. SGLang caches prefixes within one server. Mooncake disaggregates KV cache within one datacenter. Anthropic's prompt caching works within one API provider's fleet. None of them can share cached state across organizational boundaries. Hyperspace removes this boundary. The cache is global. A response computed by a node in Tokyo is immediately available to a node in Berlin. A KV prefix state generated for Qwen-32B on one machine is verifiable and reusable by any other machine running the same model. The routing network provides the delivery guarantees, the erasure coding provides the redundancy, and the cache proofs provide the trust. What this means for the cost of intelligence Big AI labs scale linearly: twice the users means twice the GPU spend. Every query is a cost center. Their internal caching helps, but it's siloed - Lab A's cache can't serve Lab B's users, and neither can serve a self-hosted Llama deployment. Hyperspace scales sub-linearly. Every new node that joins the network adds to the global cache. Every inference result enriches the cache for all future requests. The cache hit rate rises with network size because query distributions follow a power law - the most common questions are asked exponentially more often than rare ones. The implication is simple: as the network grows, the effective cost per inference drops. Not linearly. Logarithmically. At 10 million nodes, we estimate 75-90% of all inference requests can be served from cache, eliminating 400,000+ MWh of energy consumption per year and avoiding over 200,000 tons of CO2 emissions. The first person to ask a question pays the compute cost. Everyone after them gets the answer for free, with cryptographic proof that it's authentic. Training is competitive. Inference is shared Open-weight models are converging on quality with closed models. Labs will continue to differentiate on training - data curation, architecture innovation, RLHF tuning. That's where the real intellectual property lives. But inference is a commodity. Two copies of Qwen-32B running the same prompt produce the same KV state and the same response, byte for byte, regardless of whose GPU runs the matrix multiplication. There is no moat in multiplying matrices. The moat is in training the weights. A global distributed cache makes this separation explicit. It doesn't matter who trained the model. Once the weights are open, the inference cost approaches zero at scale - because the network remembers every answer and can prove it's correct. No lab, no matter how well-funded, can match this. They cannot share caches across competitors. They scale linearly. The network scales logarithmically. The marginal cost of intelligence approaches zero. That's the endgame.

Varun

37,555 просмотров • 5 месяцев назад

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

Kleiner Perkins

17,495 просмотров • 2 месяцев назад

We're proud to be the fastest inference provider on Artificial Analysis, OpenRouter, and Hugging Face for GLM-5.3-Flash, at 122+ TPS. All served from the US only, starting on day 0, with ZDR by default. Stay tuned for updates as our engineers continue to optimize GLM-5.3-Flash for even higher throughput and lower latency.

We're proud to be the fastest inference provider on Artificial Analysis, OpenRouter, and Hugging Face for GLM-5.3-Flash, at 122+ TPS. All served from the US only, starting on day 0, with ZDR by default. Stay tuned for updates as our engineers continue to optimize GLM-5.3-Flash for even higher throughput and lower latency.

Baseten

23,041 просмотров • 18 дней назад

Before you ask: yes, everything is AI here. The video and sound both coming from a single text prompt using #Veo3 by Google DeepMind .Whoever is cooking the model, let him cook! Congrats matthieu lorrain and the team for the Google I/O live stream and the new Veo site!

Before you ask: yes, everything is AI here. The video and sound both coming from a single text prompt using #Veo3 by Google DeepMind .Whoever is cooking the model, let him cook! Congrats matthieu lorrain and the team for the Google I/O live stream and the new Veo site!

László Gaál

2,753,523 просмотров • 1 год назад

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 просмотров • 3 месяцев назад

🚨🇨🇳 Chinese modular building revolution goes global 🔸 Modular homes and pre-fabricated ‘Holon’ buildings can be built in days rather than months 🔸 Exports rose from $1.5 billion in 2015 to $4.3 billion in 2025, reaching the US, Australia, Russia and Southeast Asia 🔸 Use of lightweight steel B-core slabs for walls improve earthquake and typhoon resistance, and is claimed to cut carbon emissions by up to 90% 🔸 Rising labor and material costs, housing shortages and long build times have increased demand from developed countries 🔸 Demand in the developing world is due to supply chain problems, lack of skilled workers and an inefficient building trade The new Chinese prefab industry is setting a new global standard for speed, efficiency and environmental performance.

🚨🇨🇳 Chinese modular building revolution goes global 🔸 Modular homes and pre-fabricated ‘Holon’ buildings can be built in days rather than months 🔸 Exports rose from $1.5 billion in 2015 to $4.3 billion in 2025, reaching the US, Australia, Russia and Southeast Asia 🔸 Use of lightweight steel B-core slabs for walls improve earthquake and typhoon resistance, and is claimed to cut carbon emissions by up to 90% 🔸 Rising labor and material costs, housing shortages and long build times have increased demand from developed countries 🔸 Demand in the developing world is due to supply chain problems, lack of skilled workers and an inefficient building trade The new Chinese prefab industry is setting a new global standard for speed, efficiency and environmental performance.

Sputnik

14,844 просмотров • 3 месяцев назад

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

102,628 просмотров • 5 месяцев назад

Announcing our $16M Series A led by Standard Capital With participation from Y Combinator, Pioneer, Liquid 2 Ventures, Michael Grinich, Yuri Sagalov As part of this, we’re hiring for multiple roles across engineering, support and marketing ↓

Announcing our $16M Series A led by Standard Capital With participation from Y Combinator, Pioneer, Liquid 2 Ventures, Michael Grinich, Yuri Sagalov As part of this, we’re hiring for multiple roles across engineering, support and marketing ↓

Matt – Trigger.dev

81,255 просмотров • 9 месяцев назад

🚧 Pardon our progress as we make room for the new $82 million home of AAMU STEM. This state-of-the-art project will nearly triple the university’s instructional classroom and laboratory capacity in the areas of biology, chemistry and physics. The #AAMU150 campaign is setting a new standard for support. Join today at to be part of building the Alabama A&M of the future! #StartHere

🚧 Pardon our progress as we make room for the new $82 million home of AAMU STEM. This state-of-the-art project will nearly triple the university’s instructional classroom and laboratory capacity in the areas of biology, chemistry and physics. The #AAMU150 campaign is setting a new standard for support. Join today at to be part of building the Alabama A&M of the future! #StartHere

Alabama A&M University

12,767 просмотров • 1 год назад

Today, we’re launching Prodigy Research (YC S26), the frontier AI trading research lab. We’re training the world’s best foundation model for quantitative finance. Our AI quant outperforms a top 10% trader at Jane Street, and we've already achieved more than 100% returns in live trading over our YC batch, while major indices were flat or down. Prodigy’s model beats Claude Fable and GPT-5.6 Sol at autonomous quant research. Our founders are brothers who have built strategies generating billions in PnL on a Jane Street trading desk, trained SoTA foundation models at Google DeepMind, and shipped cutting-edge AI to billions of users at Apple. A new era has arrived in the world of quantitative finance. To learn more, contact founders@prodigy.markets. Prodigy Research (YC S26)

Today, we’re launching Prodigy Research (YC S26), the frontier AI trading research lab. We’re training the world’s best foundation model for quantitative finance. Our AI quant outperforms a top 10% trader at Jane Street, and we've already achieved more than 100% returns in live trading over our YC batch, while major indices were flat or down. Prodigy’s model beats Claude Fable and GPT-5.6 Sol at autonomous quant research. Our founders are brothers who have built strategies generating billions in PnL on a Jane Street trading desk, trained SoTA foundation models at Google DeepMind, and shipped cutting-edge AI to billions of users at Apple. A new era has arrived in the world of quantitative finance. To learn more, contact [email protected]. Prodigy Research (YC S26)

Michael Wang

1,941,743 просмотров • 1 месяц назад

Excited to announce AlphaEvolve A powerful AI coding agent developed by our team in Google DeepMind that is able to discover impactful new algorithms for important problems in Maths and Computing by combining the creativity of large language models with automated evaluators.

Excited to announce AlphaEvolve A powerful AI coding agent developed by our team in Google DeepMind that is able to discover impactful new algorithms for important problems in Maths and Computing by combining the creativity of large language models with automated evaluators.

Pushmeet Kohli

203,589 просмотров • 1 год назад

Last week we launched Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency Watch our new video walkthrough to see how to build and run your own benchmark with Optima, and find the best model for your use case Start building your benchmark today at

Last week we launched Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency Watch our new video walkthrough to see how to build and run your own benchmark with Optima, and find the best model for your use case Start building your benchmark today at

Artificial Analysis

19,837 просмотров • 27 дней назад