Avi Chawla
@_avichawla • 71,911 subscribers
Daily tutorials and insights on DS, ML, LLMs, and RAGs • Co-founder @dailydoseofds_ • IIT Varanasi • ex-AI Engineer @ MastercardAI
Shorts
Videos
A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
Avi Chawla267,206 views • 22 days ago
Anthropic's in trouble, again! They spent years building what's now fully open-source. What made Claude feel different from a normal app is that the agent could act inside the interface instead of only talking in a chat box. For instance, Claude Artifacts let an agent render real UI, charts, dashboards, and interactive components that assemble live inside the response. Every major AI product tried to replicate it. But the problem was that unlike reasoning, planning, tool-calling, etc., none of it shipped natively with LangGraph, CrewAI, or Google ADK. So teams started building an owned version that required engineering the entire interface layer from scratch. Most teams, however, just settled for shipping the agent as a backend API in a chat box since rendering the UI is only one piece of it. To actually make it work, the interface layer also needed real-time streaming, state kept in sync between agent and UI, conversations that persist across sessions, and reconnection when a user refreshes mid-run. CopilotKit🪁 is now the only open-source framework that actually lets you build your own full-stack Claude-like apps. It decouples the agent from the interface, talking over AG-UI (an open protocol for agent-to-user communication). Being a standard protocol, the frontend never needs to know whether it is talking to a LangGraph or a CrewAI agent. You can change the backend anytime and the UI will never notice. In practice, CopilotKit's interface layer gives several pre-implemented React building blocks that wire the agent directly into the app, like: - generative UI, so the agent renders real components instead of text - chat windows, sidebars, and popups, or a fully headless setup - shared state, so the agent and app stay in sync - human-in-the-loop approvals, where the agent waits before acting - persistent threads that store the whole session, including the agent-user interactions and generated UI, not just text And because that full history is captured, those interactions can feed a self-learning layer that also improves the agent from real usage over time. The interface layer that Anthropic spent years engineering in-house is now literally available to any developer/team. CopilotKit is open-source with 30k+ GitHub stars, and AG-UI, the protocol underneath, is already supported across every major agent framework: LangGraph, CrewAI, Mastra, Google ADK, and more. CopilotKit GitHub repo → (don't forget to star it ⭐ ) If you want to go deeper, I found a detailed breakdown by Shubham Saboo recently on the three Generative UI patterns, with implementation. Read it below.
Avi Chawla455,742 views • 1 month ago
Another blow to Anthropic! Devs built a free and better Claude alternative that: - runs locally - works with any LLM - beats it on deep research - has Cowork-like capabilities - connects to 40+ data sources - self-hosts via Docker, and more. 100% open-source (20k+ stars).
Avi Chawla669,996 views • 3 months ago
Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.
Avi Chawla349,743 views • 1 month ago
Researchers built a new RAG approach that: - does not need a vector DB. - does not embed data. - involves no chunking. - performs no similarity search. And it hit 98.7% accuracy on a financial benchmark (SOTA). Here's the core problem with RAG that this new approach solves: Traditional RAG chunks documents, embeds them into vectors, and retrieves based on semantic similarity. But similarity ≠ relevance. When you ask "What were the debt trends in 2023?", a vector search returns chunks that look similar. But the actual answer might be buried in some Appendix, referenced on some page, in a section that shares zero semantic overlap with your query. Traditional RAG would likely never find it. PageIndex (open-source) solves this. Instead of chunking and embedding, PageIndex builds a hierarchical tree structure from your documents, like an intelligent table of contents. Then it uses reasoning to traverse that tree. For instance, the model doesn't ask: "What text looks similar to this query?" Instead, it asks: "Based on this document's structure, where would a human expert look for this answer?" That's a fundamentally different approach with: - No arbitrary chunking that breaks context. - No vector DB infrastructure to maintain. - Traceable retrieval to see exactly why it chose a specific section. - The ability to see in-document references ("see Table 5.3") the way a human would. But here's the deeper issue that it solves. Vector search treats every query as independent. But documents have structure and logic, like sections that reference other sections and context that builds across pages. PageIndex respects that structure instead of flattening it into embeddings. Do note that this approach may not make sense in every use case since traditional vector search is still fast, simple, and works well for many applications. But for professional documents that require domain expertise and multi-step reasoning, this tree-based, reasoning-first approach shines. For instance, PageIndex achieved 98.7% accuracy on FinanceBench, significantly outperforming traditional vector-based RAG systems on complex financial document analysis. Everything is fully open-source, so you can see the full implementation in GitHub and try it yourself. I have shared the GitHub repo in the replies!
Avi Chawla972,347 views • 5 months ago
LLM inference speed with vs. without KV caching: (learn how and why it works below)
Avi Chawla395,319 views • 4 months ago
I cut Fable 5 token usage 2.5x with just one change! - Before: 5.5 M tokens · 7 errors · $8.94 - After: 2.3 M tokens · 0 errors · $4.17 The final build was the same for both, but the path the agent took wildly differed. In both runs, the agent started with the same thing, i.e., it understood the backend before building anything, like: - Permission policies - Available storage buckets - Auth providers configured - How edge functions are deployed The first run used Firebase, which was built for a human dev using a dashboard. While the dev can read the above state by clicking through tabs, an agent has no dashboard. So it gathered the same info through API calls. And there's no single Firebase call that returned this info. The agent required to query multiple times, and each query over-returned. For instance, when the agent asked how sign-in is configured, Firebase also returned the entire auth surface and every method it supported. This was far more context than what it needed. And it repeated across every part of the backend it inspected. Some states (like which auth providers are active) weren't queryable at all. I provided it myself. Otherwise, the agent would have guessed. Errors further compounded the token usage. When a dev sees "permission denied," they can look at the console and figure out whether it's a rule, a path, or an unauthenticated request. Firebase returned the same string to the agent as well, and it had none of that surrounding context to debug. So it guessed again, picked the most likely cause, and rewrote code, utilizing more tokens. This Firebase setup cost me 5.5M tokens and 7 manual interventions during errors on a full-stack RAG app. But I brought that down to 2.3M tokens and 0 manual interventions by using InsForge as the backend context engineering layer (open-source and self-hostable via Docker). It provides the same primitives as Supabase/Firebase, but structures the entire information layer for agents, instead of dashboards. In one CLI call that consumed ~500 tokens, the agent saw the full backend topology before writing a single line of code. This included auth, database, storage, edge functions, model gateway, micro VMs, and deployment. Also, instead of loading the entire product surface into context on every task, four narrowly scoped skills activated only when relevant to keep cognitive load minimal. And to ensure efficient retries if needed, every CLI operation returned structured JSON with meaningful exit codes, so the agent never guessed what to do next. Here's the InsForge GitHub Repo: (don't forget to star it ⭐) The video below depicts the final build, comparing Firebase and InsForge. To dive deeper, I recently published a full walkthrough building the same RAG app on both backends and inspected them end-to-end. Read it below.
Avi Chawla112,879 views • 1 month ago
Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
Avi Chawla157,390 views • 2 months ago
Finally, Python 3.14 lets you disable GIL! It's a big deal because earlier, even if you wrote multi-threaded code, Python could only run one thread at a time, giving no performance benefit. But now, Python can run your multi-threaded code in parallel. And uv fully supports it!
Avi Chawla546,660 views • 9 months ago
Anthropic's in trouble, again. The entire Claude experience is now available at 1/6th the price. Kimi now does everything Claude does, powered by K2.6, a 1-trillion-parameter MoE model that activates only 32B parameters per token. It covers all three features Claude has (Chat, Code, and Cowork): 1) Kimi Chat runs in four modes - Instant for fast responses - Thinking for deep reasoning - Agent for multi-step execution - and Agent Swarm for parallel workloads. There's a 262K context window across all of them. 2) Kimi Code is the open-source CLI coding agent with K2.6 as the default backend. K2.6 ranked #1 on OpenRouter's programming leaderboard by weekly usage. 3) Kimi Agent is the Cowork equivalent. It generates: - full websites with database and auth - presentation decks (editable PPTX output) - spreadsheets with formulas and charts - word docs and structured research reports. On top of this, Kimi K2.6 is also trained to decompose tasks into up to 300 parallel sub-agents. This helps it retain coherence even across 4,000+ tool calls in a single run, with sessions sustaining up to 13 hours. On SWE-Bench Pro: - Kimi K2.6 → 58.6 - GPT-5.4 xhigh → 57.7 - Gemini 3.1 Pro → 54.2 - Claude Opus 4.6 → 53.4 Kimi K2.6 model is open weights and self-hostable on 4x H100s in INT4. Find the link to the HuggingFace model page in the replies!
Avi Chawla109,238 views • 2 months ago
Finally, a proper chat UI for Hermes Agent (open-source)! Hermes ships an official dashboard, but it's primarily built for management, and its chat is just a terminal piped into a browser tab. Hermes Web UI is an open-source chat-first alternative. It's self-hosted and points at your existing ~/.hermes state, so there's nothing new to configure. - It's a native web chat, not a terminal in a tab - Sessions group by date with a context ring - Kanban renders the agent's task board - Spaces manages your workspaces - Skills panel lists the full catalog - Tasks panel shows cron jobs - Insights show usage and activity - Memory shows MEMORY and SOUL files - Logs tails the agent, gateway, and error logs The whole setup runs 100% locally, binds to localhost by default, and you reach it over an SSH tunnel or Tailscale from your phone. I have shared the Hermes Web UI GitHub repo in the replies. Do note that it's a community project, not official, so expect occasional rough edges (concurrent profile runs are blocked for now). To dive deeper into Hermes Agent, my co-founder wrote a full masterclass about it, covering the learning loop, the memory tiers, self-evolving skills, GEPA, and running multiple isolated agents. Read it below.
Avi Chawla77,513 views • 1 month ago
OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!
Avi Chawla138,554 views • 4 months ago
Pentesting firms don't want you to see this. An open-source AI agent just replicated their $50k service. A "normal" pentest today looks like this: - $20k-$50k per engagement - 4-6 weeks of scoping, NDAs, kickoff calls - A big PDF that's outdated the moment you ship a new feature Meanwhile, AI agents are quietly starting to perform on-par with human pentester on the stuff that actually matters day-to-day: ↳ Enumerating attack surface ↳ Fuzzing endpoints ↳ Chaining simple vulns into real impact ↳ Producing PoCs and remediation steps developers can actually use And they do it in hours instead of weeks and at a fraction of the cost. This approach is actually implemented in Strix, a recently-trending open-source framework (14k+ stars) for AI pentesting agent. The framework spins up a team of AI "attackers" that probe your web apps, APIs, and code. It then returns validated findings with exploit evidence, remediation steps, and a full PDF report that looks exactly like what you'd get from a traditional firm, but without a $50k invoice and a month-long wait time. You can see the full implementation on GitHub and try it yourself. Just run: `strix --target https: //your-app .com` and you are good to go. Human red teams aren't disappearing but the routine pentest (pre-launch, post-refactor, quarterly checks) is clearly shifting to AI. Strix is one of the first tools that makes that shift feel real instead of hypothetical. I've shared the GitHub repo in the replies.
Avi Chawla224,487 views • 7 months ago
Big moment for Postgres! AI coding tools have been surprisingly bad at writing Postgres code. Not because the models are dumb, but because of how they learned SQL in the first place. LLMs are trained on the internet, which is full of outdated Stack Overflow answers and quick-fix tutorials. So when you ask an AI to generate a schema, it gives you something that technically runs but misses decades of Postgres evolution, like: - No GENERATED ALWAYS AS IDENTITY (added in PG10) - No expression or partial indexes - No NULLS NOT DISTINCT (PG15) - Missing CHECK constraints and proper foreign keys - Generic naming that tells you nothing But this is actually a solvable problem. You can teach AI tools to write better Postgres by giving them access to the right documentation at inference time. This exact solution is actually implemented in the newly released pg-aiguide by Tiger Data - Creators of TimescaleDB, which is an open-source MCP server that provides coding tools access to 35 years of Postgres expertise. In a gist, the MCP server enables: - Semantic search over the official PostgreSQL manual (version-aware, so it knows PG14 vs PG17 differences) - Curated skills with opinionated best practices for schema design, indexing, and constraints. I ran an experiment with Claude Code to see how well this works, and worked with the team to put this together. Prompt: "Generate a schema for an e-commerce site twice, one with the MCP server disabled, one with it enabled. Finally, run an assessment to compare the generated schemas." The run with the MCP server led to: - 420% more indexes (including partial and expression indexes) - 235% more constraints - 60% more tables (proper normalization) - 11 automation functions and triggers - Modern PG17 patterns throughout The MCP-assisted schema had proper data integrity, performance optimizations baked in, and followed naming conventions that actually make sense in production. pg-aiguide works with Claude Code, Cursor, VS Code, and any MCP-compatible tool. It's free and fully open source. I have shared the repo in the replies!
Avi Chawla186,931 views • 6 months ago
Check this!! Microsoft open-sourced a no-code data analysis tool. It's called Data Formulator and it provides AI-powered data analysis and an drag-and-drop UI for viz tasks. It also works beyond the initial dataset by creating relevant fields and the corresponding viz.
Avi Chawla260,087 views • 1 year ago
An MCP server to create Grant Sanderson animations (open-source):
Avi Chawla192,133 views • 1 year ago
Big update for Claude Desktop and Cursor users! Now you can connect all AI apps via a common memory layer in a minute. I used the Graphiti MCP server that runs 100% locally to cross-operate across AI apps like Claude Desktop and Cursor without losing context. (setup below)
Avi Chawla122,824 views • 8 months ago