Loading video...

Video Failed to Load

Go Home

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B...

anirudhbv_ce's profile picture

ani

6,917 subscribers

17,269 views • 2 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🚀 Self-speculation brings 6.75x real speedup for LLM generation with SGLang inference! Same model drafts future tokens in Diffusion mode → then verifies them in AR (causal) mode. One model and one KV cache. Just different attention masks. Thanks to perfect alignment, we get 2× longer acceptance lengths than MTP techniques (Eagle-3, MTP, dFlash). We run 2 forward passes… but the 2× higher acceptance means we break even - and with zero overhead from extra drafter, KV cache, or LM head that comes with MTP - those are not free. Last week we released Nemotron-Labs-Diffusion + Tri-mode LLMs! We did continued pre-training on Ministral-3 models by switching attention patterns (block causal bidirectional). Result: one model that runs AR mode, Diffusion mode, and Self-Speculation. Diffusion mode already shows high benchmark accuracy - excited to see what happens when someone beats left-to-right acceptance! 🔥 Github: Paper: SGLang inference: Try the models on HF:
0:13

🚀 Self-speculation brings 6.75x real speedup for LLM generation with SGLang inference! Same model drafts future tokens in Diffusion mode → then verifies them in AR (causal) mode. One model and one KV cache. Just different attention masks. Thanks to perfect alignment, we get 2× longer acceptance lengths than MTP techniques (Eagle-3, MTP, dFlash). We run 2 forward passes… but the 2× higher acceptance means we break even - and with zero overhead from extra drafter, KV cache, or LM head that comes with MTP - those are not free. Last week we released Nemotron-Labs-Diffusion + Tri-mode LLMs! We did continued pre-training on Ministral-3 models by switching attention patterns (block causal bidirectional). Result: one model that runs AR mode, Diffusion mode, and Self-Speculation. Diffusion mode already shows high benchmark accuracy - excited to see what happens when someone beats left-to-right acceptance! 🔥 Github: Paper: SGLang inference: Try the models on HF:

Pavlo Molchanov

66,604 views • 2 months ago

A good technical LLM interview question: Your LLM chatbot takes 12s before it generates the first token, and the users are complaining. So you move the model onto a GPU with 3x the computing power. The time to first token barely improves. Why did this happen? (answer below) Latency in an LLM app is a placement problem disguised as a model problem. If you profile the 12 seconds, the model's prefill itself may only account for around 1.5 seconds of it. So halving the prefill step saves just 750ms out of 12000, which is under 7%. The rest is spread across stages that never touch the GPU. The request first travels to whatever region the app runs in, and a cross-continent round trip could cost over a second before any code executes. Then the request handler starts. On a container-based serverless platform under load, this adds several seconds of cold start, paid before auth, rate limiting, or prompt assembly even begins. Retrieval adds its own hop, and the response streams back across the same distance. Optimizing a stage that was already fast cannot alter the latency that's majorly affected by other stages. Those other stages are slow for a structural reason. An LLM app runs two workloads that want opposite machines. - The request path is short, spiky, and needs to sit close to users - Inference is long-running, GPU-bound, and billed hourly, whether requests arrive or not. So the actual decision is not which model to run, but where each of these two workloads runs. There are three options, each with its own tradeoffs: > A dedicated GPU box removes inference cold starts, but it bills around the clock and lives in one location, so distant users wait out the round trip on every request > Container-based serverless scales to zero, but the request path pays a cold start, and most of these platforms have no GPU behind them. > Edge runtimes start in under a millisecond, because a WebAssembly module carries no OS or container image to boot. They handle the request path well and cannot hold a model. So the answer is not to pick one, but to split the app across two of them. The request path runs close to users, and inference runs on a dedicated GPU it calls into. That also explains the failed upgrade. More compute made a stage that was already fast faster, and left the 10.5 seconds around it untouched. To actually learn how it's done in practice, Akamai's GitHub has a reference implementation for each half. - vllm-on-lke serves Qwen2.5-7B-Instruct behind an OpenAI-compatible endpoint on one RTX 4000 Ada GPU in Linode Kubernetes Engine, with Terraform creating the cluster, both firewalls, and the GPU operator in one apply. - akamai-functions-llm-chatbot covers the front, where a WebAssembly API checks a KV cache and only calls the GPU-backed instance on a miss. Both are available on Akamai’s new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post treats generation as a single 1.5s block, but that block has its own structure, and knowing it well tells you whether a model is slow to start or slow to stream. I wrote a first-principles walkthrough of it, covering the prefill and decode split, KV caching, and where the time actually goes inside each one. Read it below. Thanks to Akamai Cloud for partnering today!
0:18

A good technical LLM interview question: Your LLM chatbot takes 12s before it generates the first token, and the users are complaining. So you move the model onto a GPU with 3x the computing power. The time to first token barely improves. Why did this happen? (answer below) Latency in an LLM app is a placement problem disguised as a model problem. If you profile the 12 seconds, the model's prefill itself may only account for around 1.5 seconds of it. So halving the prefill step saves just 750ms out of 12000, which is under 7%. The rest is spread across stages that never touch the GPU. The request first travels to whatever region the app runs in, and a cross-continent round trip could cost over a second before any code executes. Then the request handler starts. On a container-based serverless platform under load, this adds several seconds of cold start, paid before auth, rate limiting, or prompt assembly even begins. Retrieval adds its own hop, and the response streams back across the same distance. Optimizing a stage that was already fast cannot alter the latency that's majorly affected by other stages. Those other stages are slow for a structural reason. An LLM app runs two workloads that want opposite machines. - The request path is short, spiky, and needs to sit close to users - Inference is long-running, GPU-bound, and billed hourly, whether requests arrive or not. So the actual decision is not which model to run, but where each of these two workloads runs. There are three options, each with its own tradeoffs: > A dedicated GPU box removes inference cold starts, but it bills around the clock and lives in one location, so distant users wait out the round trip on every request > Container-based serverless scales to zero, but the request path pays a cold start, and most of these platforms have no GPU behind them. > Edge runtimes start in under a millisecond, because a WebAssembly module carries no OS or container image to boot. They handle the request path well and cannot hold a model. So the answer is not to pick one, but to split the app across two of them. The request path runs close to users, and inference runs on a dedicated GPU it calls into. That also explains the failed upgrade. More compute made a stage that was already fast faster, and left the 10.5 seconds around it untouched. To actually learn how it's done in practice, Akamai's GitHub has a reference implementation for each half. - vllm-on-lke serves Qwen2.5-7B-Instruct behind an OpenAI-compatible endpoint on one RTX 4000 Ada GPU in Linode Kubernetes Engine, with Terraform creating the cluster, both firewalls, and the GPU operator in one apply. - akamai-functions-llm-chatbot covers the front, where a WebAssembly API checks a KV cache and only calls the GPU-backed instance on a miss. Both are available on Akamai’s new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post treats generation as a single 1.5s block, but that block has its own structure, and knowing it well tells you whether a model is slow to start or slow to stream. I wrote a first-principles walkthrough of it, covering the prefill and decode split, KV caching, and where the time actually goes inside each one. Read it below. Thanks to Akamai Cloud for partnering today!

Avi Chawla

21,163 views • 7 days ago

#WATCH | Delhi | An AAP worker says, "Dictatorship will not be tolerated. You are picking up students as if they were terrorists. Yet, those responsible for leaking the papers are not being picked up. The government is failing to apprehend the paper leak culprits."
0:10

#WATCH | Delhi | An AAP worker says, "Dictatorship will not be tolerated. You are picking up students as if they were terrorists. Yet, those responsible for leaking the papers are not being picked up. The government is failing to apprehend the paper leak culprits."

ANI

24,605 views • 3 months ago

#WATCH | Delhi: Lok Sabha LoP and Congress MP Rahul Gandhi, along with INDIA bloc MPs, reach Gandhi Smriti as they continue their protest over the NEET paper leak.
0:18

#WATCH | Delhi: Lok Sabha LoP and Congress MP Rahul Gandhi, along with INDIA bloc MPs, reach Gandhi Smriti as they continue their protest over the NEET paper leak.

ANI

102,587 views • 1 month ago

#WATCH | Muskan from Chandigarh says, "We are here to support an Army. They are doing so much for us, and we also want to do something for our Army."
0:17

#WATCH | Muskan from Chandigarh says, "We are here to support an Army. They are doing so much for us, and we also want to do something for our Army."

ANI

576,358 views • 1 year ago

#WATCH | TVK Chief Vijay meets Tamil Nadu Governor Rajendra Vishwanath Arlekar at the Lok Bhavan. With the support of Congress, CPI, CPI(M), VCK and IUML, 121 MLAs are now in support of TVK, and its path to form a government in the state is now clear. (Source: Lok Bhavan)
0:16

#WATCH | TVK Chief Vijay meets Tamil Nadu Governor Rajendra Vishwanath Arlekar at the Lok Bhavan. With the support of Congress, CPI, CPI(M), VCK and IUML, 121 MLAs are now in support of TVK, and its path to form a government in the state is now clear. (Source: Lok Bhavan)

ANI

234,426 views • 3 months ago

#WATCH | Patna | On tweet by Kerala unit of Congress, ' 'B for Beedi and B for Bihar...' (which stands deleted now), RJD leader Tejashwi Yadav says, "It was a wrong tweet. We don't support it."
0:11

#WATCH | Patna | On tweet by Kerala unit of Congress, ' 'B for Beedi and B for Bihar...' (which stands deleted now), RJD leader Tejashwi Yadav says, "It was a wrong tweet. We don't support it."

ANI

241,547 views • 11 months ago

#WATCH | Delhi: BJP MPs staged a protest outside the Parliament demanding the resignation of Punjab Education Minister Harjot Singh Bains, over the alleged paper leak in the state.
0:19

#WATCH | Delhi: BJP MPs staged a protest outside the Parliament demanding the resignation of Punjab Education Minister Harjot Singh Bains, over the alleged paper leak in the state.

ANI

63,386 views • 29 days ago

#WATCH | Delhi | BJP leader Smriti Irani says, "...Vikram talks about India's capacity to be the bridge not only between the global north and the global south, but also to ascend to a new formidable future. My compliments to the author and my gratitude for his service"
0:16

#WATCH | Delhi | BJP leader Smriti Irani says, "...Vikram talks about India's capacity to be the bridge not only between the global north and the global south, but also to ascend to a new formidable future. My compliments to the author and my gratitude for his service"

ANI

20,246 views • 9 months ago

#WATCH | Kolkata: Former CM and TMC chief Mamata Banerjee says, “If need be, I myself will go to the CJP protest. They had our support from the beginning, and will continue to have it.” (Source: TMC)
0:18

#WATCH | Kolkata: Former CM and TMC chief Mamata Banerjee says, “If need be, I myself will go to the CJP protest. They had our support from the beginning, and will continue to have it.” (Source: TMC)

ANI

124,861 views • 1 month ago

#WATCH | Auckland, New Zealand | Prime Minister Narendra Modi emplanes for India after concluding his three-nation visit to Indonesia, Australia, and New Zealand. (Source: ANI/DD)
0:18

#WATCH | Auckland, New Zealand | Prime Minister Narendra Modi emplanes for India after concluding his three-nation visit to Indonesia, Australia, and New Zealand. (Source: ANI/DD)

ANI

88,183 views • 1 month ago

#WATCH | On Bagram Airbase in Afghanistan, US President Donald J Trump says, "We're talking now to Afghanistan, and we want it back, and we want it back soon. If they don't do it, you're going to find out what I'm going to do." (Source: US Network Pool via Reuters)
0:19

#WATCH | On Bagram Airbase in Afghanistan, US President Donald J Trump says, "We're talking now to Afghanistan, and we want it back, and we want it back soon. If they don't do it, you're going to find out what I'm going to do." (Source: US Network Pool via Reuters)

ANI

461,828 views • 11 months ago

#WATCH | Guwahati, Assam: An FIR has been filed against the Assam Jatiya Parishad (AJP) candidate of the Guwahati Central seat, Kunki Chowdhury, at the Panbazar Police Station for violating the Model Code of Conduct on the polling day, April 9th. She says, "An FIR was filed against me, making some false allegations. So I came here to give a statement... I presented my side and gave my statement. I have full faith in the system, and it will investigate thoroughly..."
0:18

#WATCH | Guwahati, Assam: An FIR has been filed against the Assam Jatiya Parishad (AJP) candidate of the Guwahati Central seat, Kunki Chowdhury, at the Panbazar Police Station for violating the Model Code of Conduct on the polling day, April 9th. She says, "An FIR was filed against me, making some false allegations. So I came here to give a statement... I presented my side and gave my statement. I have full faith in the system, and it will investigate thoroughly..."

ANI

28,172 views • 4 months ago

#WATCH | Rudraprayag, Uttarakhand | Due to fresh snowfall in April, the Kedarnath temple complex and its surrounding areas are currently draped in a pristine white blanket of snow. Along the trekking route, glaciers have descended near Chhoti Lincholi and Badi Lincholi. As pilgrims advance from Gaurikund toward Lincholi, they will traverse paths carved directly through these glaciers, adding a thrilling edge to their journey. Furthermore, the view of Baba Kedarnath Dham from Dev Darshani is exceptionally divine and captivating this season, promising a deeply spiritual experience for all visitors. The portals (kapat) of Baba Kedarnath are set to open for devotees on April 22. With the opening of the gates, thousands of pilgrims are expected to arrive at the Dham, marking the formal commencement of this holy pilgrimage. (Source: Local Administration)
0:07

#WATCH | Rudraprayag, Uttarakhand | Due to fresh snowfall in April, the Kedarnath temple complex and its surrounding areas are currently draped in a pristine white blanket of snow. Along the trekking route, glaciers have descended near Chhoti Lincholi and Badi Lincholi. As pilgrims advance from Gaurikund toward Lincholi, they will traverse paths carved directly through these glaciers, adding a thrilling edge to their journey. Furthermore, the view of Baba Kedarnath Dham from Dev Darshani is exceptionally divine and captivating this season, promising a deeply spiritual experience for all visitors. The portals (kapat) of Baba Kedarnath are set to open for devotees on April 22. With the opening of the gates, thousands of pilgrims are expected to arrive at the Dham, marking the formal commencement of this holy pilgrimage. (Source: Local Administration)

ANI

15,161 views • 4 months ago

#WATCH | Andhra Pradesh Deputy CM Pawan Kalyan arrives in Delhi to participate in tomorrow’s “Sena Prasthanam… For National Integrity” programme. During the programme, Pawan Kalyan is expected to give direction and guidance to party leaders on the movement’s vision and future course of action. (Source: Janasena party office)
0:13

#WATCH | Andhra Pradesh Deputy CM Pawan Kalyan arrives in Delhi to participate in tomorrow’s “Sena Prasthanam… For National Integrity” programme. During the programme, Pawan Kalyan is expected to give direction and guidance to party leaders on the movement’s vision and future course of action. (Source: Janasena party office)

ANI

34,005 views • 2 months ago

#WATCH | Haryana: A video went viral where a man was throwing currency notes from his running car in Gurugram. Police file a case in the matter. (Police have verified the viral video)
0:15

#WATCH | Haryana: A video went viral where a man was throwing currency notes from his running car in Gurugram. Police file a case in the matter. (Police have verified the viral video)

ANI

895,111 views • 3 years ago

#WATCH | Hyderabad, Telangana | After the meeting with SP chief Akhilesh Yadav, BRS working president KT Rama Rao says, "We draw inspiration from you and your party about the manner in which you reached Delhi with 37 MPs. We will bounce back once again and try to come back to power..."
0:18

#WATCH | Hyderabad, Telangana | After the meeting with SP chief Akhilesh Yadav, BRS working president KT Rama Rao says, "We draw inspiration from you and your party about the manner in which you reached Delhi with 37 MPs. We will bounce back once again and try to come back to power..."

ANI

209,095 views • 8 months ago