pip install spectralquant ✂️ Up to 6.62x KV cache... compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inferenceshow more

🚀 Self-speculation brings 6.75x real speedup for LLM generation... with SGLang inference! Same model drafts future tokens in Diffusion mode → then verifies them in AR (causal) mode. One model and one KV cache. Just different attention masks. Thanks to perfect alignment, we get 2× longer acceptance lengths than MTP techniques (Eagle-3, MTP, dFlash). We run 2 forward passes… but the 2× higher acceptance means we break even - and with zero overhead from extra drafter, KV cache, or LM head that comes with MTP - those are not free. Last week we released Nemotron-Labs-Diffusion + Tri-mode LLMs! We did continued pre-training on Ministral-3 models by switching attention patterns (block causal bidirectional). Result: one model that runs AR mode, Diffusion mode, and Self-Speculation. Diffusion mode already shows high benchmark accuracy - excited to see what happens when someone beats left-to-right acceptance! 🔥 Github: Paper: SGLang inference: Try the models on HF:show more

A good technical LLM interview question: Your LLM chatbot... takes 12s before it generates the first token, and the users are complaining. So you move the model onto a GPU with 3x the computing power. The time to first token barely improves. Why did this happen? (answer below) Latency in an LLM app is a placement problem disguised as a model problem. If you profile the 12 seconds, the model's prefill itself may only account for around 1.5 seconds of it. So halving the prefill step saves just 750ms out of 12000, which is under 7%. The rest is spread across stages that never touch the GPU. The request first travels to whatever region the app runs in, and a cross-continent round trip could cost over a second before any code executes. Then the request handler starts. On a container-based serverless platform under load, this adds several seconds of cold start, paid before auth, rate limiting, or prompt assembly even begins. Retrieval adds its own hop, and the response streams back across the same distance. Optimizing a stage that was already fast cannot alter the latency that's majorly affected by other stages. Those other stages are slow for a structural reason. An LLM app runs two workloads that want opposite machines. - The request path is short, spiky, and needs to sit close to users - Inference is long-running, GPU-bound, and billed hourly, whether requests arrive or not. So the actual decision is not which model to run, but where each of these two workloads runs. There are three options, each with its own tradeoffs: > A dedicated GPU box removes inference cold starts, but it bills around the clock and lives in one location, so distant users wait out the round trip on every request > Container-based serverless scales to zero, but the request path pays a cold start, and most of these platforms have no GPU behind them. > Edge runtimes start in under a millisecond, because a WebAssembly module carries no OS or container image to boot. They handle the request path well and cannot hold a model. So the answer is not to pick one, but to split the app across two of them. The request path runs close to users, and inference runs on a dedicated GPU it calls into. That also explains the failed upgrade. More compute made a stage that was already fast faster, and left the 10.5 seconds around it untouched. To actually learn how it's done in practice, Akamai's GitHub has a reference implementation for each half. - vllm-on-lke serves Qwen2.5-7B-Instruct behind an OpenAI-compatible endpoint on one RTX 4000 Ada GPU in Linode Kubernetes Engine, with Terraform creating the cluster, both firewalls, and the GPU operator in one apply. - akamai-functions-llm-chatbot covers the front, where a WebAssembly API checks a KV cache and only calls the GPU-backed instance on a miss. Both are available on Akamai’s new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post treats generation as a single 1.5s block, but that block has its own structure, and knowing it well tells you whether a model is slow to start or slow to stream. I wrote a first-principles walkthrough of it, covering the prefill and decode split, KV caching, and where the time actually goes inside each one. Read it below. Thanks to Akamai Cloud for partnering today!show more

A good technical LLM interview question: Your RAG chatbot... is working as expected locally. You deploy it behind a load balancer with 3 replicas. Users report that it forgets what they just asked, and answers get worse with each restart. Why did this happen? (answer below) A local setup has one process that owns everything. - The vector index is a variable in memory. - Conversation history is a Python list. - The documents are on local disk. You never treat any of them as infrastructure, because restarting rebuilds all three in seconds and there is only ever one copy. The setup does not carry over to production directly. The vector index might disappear on restart, so the app re-embeds everything on boot and serves empty results until it finishes. Conversation history may belong to one replica, so a follow-up routed elsewhere has no memory of the previous turn. Documents could be on whichever container ingested them, so the three replicas hold three different corpora. None of this is evident with one user and one process. So the actual work in shipping RAG is not just the retrieval logic, but also storing the vector index, the conversation history, and the documents outside the app, where every replica reads and writes the same copy. Which comes down to three requirements: > The vector store needs persistence and has to be reachable from every replica. pgvector inside Postgres keeps embeddings next to the rest of the data instead of adding another system to operate. > Conversation state has to be checkpointed outside the app. LangGraph writes its state to Postgres, so any replica can pick up a thread mid-conversation. > Docs need shared object storage, so ingestion happens once instead of once per replica. If you get those three right, the retrieval logic you wrote in the notebook works unchanged. To learn how all of it is wired together, Akamai's GitHub has a working reference implementation. - rag-langgraph-k8s-quickstart is an airline policy Q&A assistant built with FastAPI, LangChain, and LangGraph. Terraform provisions the LKE cluster, a Postgres instance with pgvector for embeddings, a second Postgres for LangGraph checkpointing, and an object storage bucket for the policy documents, in one apply. - akamai-workshop-ai-inference covers the next step, running the model yourself instead of calling an API, with prefill and decode, KV cache tradeoffs, and continuous batching under real concurrency. Both are available on Akamai's new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post assumes the retrieval logic was right to begin with, and that is doing a lot of work. Most RAG systems fail earlier, at the point where a chunk gets treated as a self-contained unit of meaning. I wrote about the two skills that fix that gap, and why the chunk is usually the wrong thing to embed. Read it below. Thanks to Akamai Cloud for partnering today!show more

#WATCH | Delhi | An AAP worker says, "Dictatorship... will not be tolerated. You are picking up students as if they were terrorists. Yet, those responsible for leaking the papers are not being picked up. The government is failing to apprehend the paper leak culprits."show more

#WATCH | Delhi: Lok Sabha LoP and Congress MP... Rahul Gandhi, along with INDIA bloc MPs, reach Gandhi Smriti as they continue their protest over the NEET paper leak.show more

#WATCH | Muskan from Chandigarh says, "We are here... to support an Army. They are doing so much for us, and we also want to do something for our Army."show more

#WATCH | TVK Chief Vijay meets Tamil Nadu Governor... Rajendra Vishwanath Arlekar at the Lok Bhavan. With the support of Congress, CPI, CPI(M), VCK and IUML, 121 MLAs are now in support of TVK, and its path to form a government in the state is now clear. (Source: Lok Bhavan)show more

#WATCH | Patna | On tweet by Kerala unit... of Congress, ' 'B for Beedi and B for Bihar...' (which stands deleted now), RJD leader Tejashwi Yadav says, "It was a wrong tweet. We don't support it."show more

#WATCH | Delhi: BJP MPs staged a protest outside... the Parliament demanding the resignation of Punjab Education Minister Harjot Singh Bains, over the alleged paper leak in the state.show more

#WATCH | Kolkata: Former CM and TMC chief Mamata... Banerjee says, “If need be, I myself will go to the CJP protest. They had our support from the beginning, and will continue to have it.” (Source: TMC)show more

#WATCH | Auckland, New Zealand | Prime Minister Narendra... Modi emplanes for India after concluding his three-nation visit to Indonesia, Australia, and New Zealand. (Source: ANI/DD)show more

#WATCH | On Bagram Airbase in Afghanistan, US President... Donald J Trump says, "We're talking now to Afghanistan, and we want it back, and we want it back soon. If they don't do it, you're going to find out what I'm going to do." (Source: US Network Pool via Reuters)show more

#WATCH | Rudraprayag, Uttarakhand | Due to fresh snowfall... in April, the Kedarnath temple complex and its surrounding areas are currently draped in a pristine white blanket of snow. Along the trekking route, glaciers have descended near Chhoti Lincholi and Badi Lincholi. As pilgrims advance from Gaurikund toward Lincholi, they will traverse paths carved directly through these glaciers, adding a thrilling edge to their journey. Furthermore, the view of Baba Kedarnath Dham from Dev Darshani is exceptionally divine and captivating this season, promising a deeply spiritual experience for all visitors. The portals (kapat) of Baba Kedarnath are set to open for devotees on April 22. With the opening of the gates, thousands of pilgrims are expected to arrive at the Dham, marking the formal commencement of this holy pilgrimage. (Source: Local Administration)show more

#WATCH | Guwahati, Assam: An FIR has been filed... against the Assam Jatiya Parishad (AJP) candidate of the Guwahati Central seat, Kunki Chowdhury, at the Panbazar Police Station for violating the Model Code of Conduct on the polling day, April 9th. She says, "An FIR was filed against me, making some false allegations. So I came here to give a statement... I presented my side and gave my statement. I have full faith in the system, and it will investigate thoroughly..."show more

#WATCH | Andhra Pradesh Deputy CM Pawan Kalyan arrives... in Delhi to participate in tomorrow’s “Sena Prasthanam… For National Integrity” programme. During the programme, Pawan Kalyan is expected to give direction and guidance to party leaders on the movement’s vision and future course of action. (Source: Janasena party office)show more

#WATCH | Telangana: A fire broke out at an... ink manufacturing unit in Prasanth Nagar industrial area in Kukatpally, Hyderabad. Three fire tenders rushed to the spot and are currently trying to control the blaze. No casualties have been reported so far, and the cause of the fire is yet to be ascertained. According to a police official, "A fire broke out at an ink manufacturing unit. Three fire tenders reached the spot and are currently trying to control the fire. No casualties have been reported so far and the cause of the fire is yet to be known." (Video Source: Fire official, Hyderabad)show more