Tech with Mak's banner

Tech with Mak

@techNmak • 47,499 subscribers

AI, coding, software, and whatever’s on my mind.

Shorts

KV Caching explained!!

KV Caching explained!!

40,454 views

One of the odd things about learning mathematics is how long you can manipulate an equation without having much of a picture of it in your head. You learn the notation, follow the derivation, get the answer, and move on. Then somebody shows you the same idea visually and you realize there was a geometric intuition hiding underneath the symbols the whole time. A Fourier sum becomes rotating vectors added head-to-tail. Attention becomes a weighted combination you can watch being formed. SVD becomes a sequence of transformations you can follow one step at a time. Even the equation behind a diffusion process feels different when you can see a point following a changing vector field. None of this replaces the mathematics. The equations are still where the precision lives. But once you have a picture in your head, the notation has something to attach itself to. I think this is why some mathematical ideas suddenly become much easier years after we first learned them. We finally stop seeing the equation as a line of symbols to manipulate and start seeing the process it describes. I wish more mathematics had been taught that way.

One of the odd things about learning mathematics is how long you can manipulate an equation without having much of a picture of it in your head. You learn the notation, follow the derivation, get the answer, and move on. Then somebody shows you the same idea visually and you realize there was a geometric intuition hiding underneath the symbols the whole time. A Fourier sum becomes rotating vectors added head-to-tail. Attention becomes a weighted combination you can watch being formed. SVD becomes a sequence of transformations you can follow one step at a time. Even the equation behind a diffusion process feels different when you can see a point following a changing vector field. None of this replaces the mathematics. The equations are still where the precision lives. But once you have a picture in your head, the notation has something to attach itself to. I think this is why some mathematical ideas suddenly become much easier years after we first learned them. We finally stop seeing the equation as a line of symbols to manipulate and start seeing the process it describes. I wish more mathematics had been taught that way.

57,362 views

If you are trying to understand where AI agents are going, learn harness engineering. A capable model is only one part of an agent system. Once the model begins reading files, calling tools, modifying state and working across many steps, the quality of the system depends increasingly on the software around it. Consider a coding agent working through a large repository. The model can decide that it needs to inspect a file, search for a symbol, make an edit or run a test, but those decisions do not execute themselves. The surrounding runtime has to decide which resources are available, whether the requested action is permitted, how the operation should be performed, what result should be retained, and what information should be presented to the model on the next step. This becomes harder as the run gets longer. As history accumulates, replaying everything can become costly and less effective. The harness has to decide what should remain in context, what should be summarized or retrieved later, and what belongs in persistent state outside the context window. Execution has similar problems. A long-running agent may need to survive an interruption, avoid repeating completed work, enforce permissions around consequential actions, and preserve enough history to reconstruct what happened when the final result is wrong. These are harness problems. The harness is the layer that manages context, tools, execution, state, checkpoints, limits and traces around the model. Harness engineering is the work of designing and improving that layer. Engineers inspect execution traces, evaluate agents on representative tasks, look for recurring failure modes, and then change things such as context selection, tool interfaces, state handling or execution controls. That last part matters because agent failures are often not fixed by changing the model. Sometimes the useful change is in what the model sees, how a tool is exposed, what state is preserved, or what the runtime does after a failed step. As agents take on longer tasks, the demands on this surrounding software grow. Model capability remains essential, but harness engineering is what turns that capability into an execution process that can be controlled, inspected, tested and improved.

If you are trying to understand where AI agents are going, learn harness engineering. A capable model is only one part of an agent system. Once the model begins reading files, calling tools, modifying state and working across many steps, the quality of the system depends increasingly on the software around it. Consider a coding agent working through a large repository. The model can decide that it needs to inspect a file, search for a symbol, make an edit or run a test, but those decisions do not execute themselves. The surrounding runtime has to decide which resources are available, whether the requested action is permitted, how the operation should be performed, what result should be retained, and what information should be presented to the model on the next step. This becomes harder as the run gets longer. As history accumulates, replaying everything can become costly and less effective. The harness has to decide what should remain in context, what should be summarized or retrieved later, and what belongs in persistent state outside the context window. Execution has similar problems. A long-running agent may need to survive an interruption, avoid repeating completed work, enforce permissions around consequential actions, and preserve enough history to reconstruct what happened when the final result is wrong. These are harness problems. The harness is the layer that manages context, tools, execution, state, checkpoints, limits and traces around the model. Harness engineering is the work of designing and improving that layer. Engineers inspect execution traces, evaluate agents on representative tasks, look for recurring failure modes, and then change things such as context selection, tool interfaces, state handling or execution controls. That last part matters because agent failures are often not fixed by changing the model. Sometimes the useful change is in what the model sees, how a tool is exposed, what state is preserved, or what the runtime does after a failed step. As agents take on longer tasks, the demands on this surrounding software grow. Model capability remains essential, but harness engineering is what turns that capability into an execution process that can be controlled, inspected, tested and improved.

49,737 views

Software Engineers when they hear "you will be replaced by AI"

Software Engineers when they hear "you will be replaced by AI"

167,821 views

you don’t hate calculus. you just never saw what the gradient was doing.

you don’t hate calculus. you just never saw what the gradient was doing.

27,826 views

Web Scraping is dead. Web Agenting is here. Writing selectors (div > .class > span) breaks every time a site updates. Building custom bots for every new target is a waste of engineering hours. TinyFish turns the entire real-time web into a single API. Input: Natural Language ("Find availability for X"). Target: 1 or 100 URLs. Output: Structured JSON. This isn't a simulation. It visits the Real-Time Web. 1./ One API, Many Sites - Same contract whether you hit 1 URL or 50. You focus on the Goal (Business Logic). TinyFish handles the How (Navigation, Clicks, Inputs). 2./ Real Automation - It doesn't just "read." It interacts. It fills forms, navigates multi-step flows, and handles dynamic JS content. 3./ Production Ready - This is the same infrastructure used by large enterprises, now exposed as a clean developer primitive. Logs, error handling, and structured data are built-in. The web is finally a proper API.

Web Scraping is dead. Web Agenting is here. Writing selectors (div > .class > span) breaks every time a site updates. Building custom bots for every new target is a waste of engineering hours. TinyFish turns the entire real-time web into a single API. Input: Natural Language ("Find availability for X"). Target: 1 or 100 URLs. Output: Structured JSON. This isn't a simulation. It visits the Real-Time Web. 1./ One API, Many Sites - Same contract whether you hit 1 URL or 50. You focus on the Goal (Business Logic). TinyFish handles the How (Navigation, Clicks, Inputs). 2./ Real Automation - It doesn't just "read." It interacts. It fills forms, navigates multi-step flows, and handles dynamic JS content. 3./ Production Ready - This is the same infrastructure used by large enterprises, now exposed as a clean developer primitive. Logs, error handling, and structured data are built-in. The web is finally a proper API.

119,865 views

🚨 This is the best way to learn how LLMs work. Interactive. 3D. Step-by-step. Covers: → Embedding → Layer Norm → Self-Attention → MLP → Transformer layers → Softmax → Output Stop reading papers. Start seeing. Link in comments. Save this immediately.

🚨 This is the best way to learn how LLMs work. Interactive. 3D. Step-by-step. Covers: → Embedding → Layer Norm → Self-Attention → MLP → Transformer layers → Softmax → Output Stop reading papers. Start seeing. Link in comments. Save this immediately.

82,419 views

Microsoft made 100B parameter models run on a single CPU. bitnet.cpp: The official inference framework for 1-bit LLMs. The math behind 1-bit LLMs is what makes them revolutionary. Traditional LLMs use 16-bit floating point weights. Every parameter is a number like 0.0023847 or -1.4729. When you run inference, you multiply these floats together. Billions of times. That's why you need GPUs, they're optimized for floating point matrix multiplication. BitNet b1.58 uses ternary weights: {-1, 0, 1}. That's not a simplification. That's a fundamental change in the math. When your weights are only -1, 0, or 1: → Multiply by 1 = keep the value → Multiply by -1 = flip the sign → Multiply by 0 = skip entirely Matrix multiplication becomes addition and subtraction. No floating point operations. No GPU required. This is why bitnet.cpp achieves: → 2.37x to 6.17x speedup on x86 CPUs → 1.37x to 5.07x speedup on ARM CPUs → 71.9% to 82.2% energy reduction on x86 → 55.4% to 70.0% energy reduction on ARM The speedups scale with model size. Larger models see bigger gains because there are more operations to simplify. A 100B parameter model running at human reading speed (5-7 tokens/second) on a single CPU. That's not optimization. That's a different paradigm. Why 1.58 bits? Because log₂(3) ≈ 1.58. Three possible values = 1.58 bits of information per weight. The key insight: These models aren't quantized after training. They're trained from scratch with ternary weights. The model learns to work within the constraint. No precision loss. No quality tradeoff.

Microsoft made 100B parameter models run on a single CPU. bitnet.cpp: The official inference framework for 1-bit LLMs. The math behind 1-bit LLMs is what makes them revolutionary. Traditional LLMs use 16-bit floating point weights. Every parameter is a number like 0.0023847 or -1.4729. When you run inference, you multiply these floats together. Billions of times. That's why you need GPUs, they're optimized for floating point matrix multiplication. BitNet b1.58 uses ternary weights: {-1, 0, 1}. That's not a simplification. That's a fundamental change in the math. When your weights are only -1, 0, or 1: → Multiply by 1 = keep the value → Multiply by -1 = flip the sign → Multiply by 0 = skip entirely Matrix multiplication becomes addition and subtraction. No floating point operations. No GPU required. This is why bitnet.cpp achieves: → 2.37x to 6.17x speedup on x86 CPUs → 1.37x to 5.07x speedup on ARM CPUs → 71.9% to 82.2% energy reduction on x86 → 55.4% to 70.0% energy reduction on ARM The speedups scale with model size. Larger models see bigger gains because there are more operations to simplify. A 100B parameter model running at human reading speed (5-7 tokens/second) on a single CPU. That's not optimization. That's a different paradigm. Why 1.58 bits? Because log₂(3) ≈ 1.58. Three possible values = 1.58 bits of information per weight. The key insight: These models aren't quantized after training. They're trained from scratch with ternary weights. The model learns to work within the constraint. No precision loss. No quality tradeoff.

23,202 views

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

You can literally type a sentence and watch a Transformer process it. This interactive explainer runs a GPT-style model in the browser and lets you follow the whole path: tokenization → embeddings → Q/K/V → attention → MLP → logits → next-token probabilities You can inspect attention maps, intermediate values, and change temperature, top-k and top-p to see how generation changes. If Transformers still feel like boxes connected by arrows, this is worth opening.

You can literally type a sentence and watch a Transformer process it. This interactive explainer runs a GPT-style model in the browser and lets you follow the whole path: tokenization → embeddings → Q/K/V → attention → MLP → logits → next-token probabilities You can inspect attention maps, intermediate values, and change temperature, top-k and top-p to see how generation changes. If Transformers still feel like boxes connected by arrows, this is worth opening.

132,935 views • 7 days ago

The math behind getting 10% better every week. Nothing happens for a while. Then everyone calls you lucky.

The math behind getting 10% better every week. Nothing happens for a while. Then everyone calls you lucky.

252,070 views • 3 months ago

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

12,211 views • 1 month ago

Unpopular opinion: VS Code is holding back AI coding. I’ve been spinning up multiple Cline instances across tmux panes and terminal tabs. Each agent has its own isolated state and can run a different task, branch, or idea in parallel. They just keep going while I focus on something else, no context collisions, no babysitting a single chat thread. You can pipe input, chain output, or run them headless in CI/CD. It fits into the workflow I already use in the terminal. Getting started literally took me 30 seconds: npm install -g cline AI is finally just another process in my workflow, instead of a panel or plugin.

Unpopular opinion: VS Code is holding back AI coding. I’ve been spinning up multiple Cline instances across tmux panes and terminal tabs. Each agent has its own isolated state and can run a different task, branch, or idea in parallel. They just keep going while I focus on something else, no context collisions, no babysitting a single chat thread. You can pipe input, chain output, or run them headless in CI/CD. It fits into the workflow I already use in the terminal. Getting started literally took me 30 seconds: npm install -g cline AI is finally just another process in my workflow, instead of a panel or plugin.

43,106 views • 8 months ago

Most people first see Euler’s Formula as a strange equation in a textbook. Then years later, they realize it quietly powers the modern world. Leonhard Euler discovered that these seemingly unrelated mathematical ideas: → exponential growth → imaginary numbers → sine waves → cosine waves → rotation … are all deeply connected through one identity: e^(iθ) = cos(θ) + i·sin(θ) At first glance it looks impossible. How can an exponential function suddenly produce circles and waves? The key insight is that multiplying by a complex exponential creates rotation. As the angle θ changes: - the cosine term tracks horizontal motion - the sine term tracks vertical motion - together they trace a perfect circle in the complex plane Euler showed that waves and rotation are mathematically the same phenomenon viewed differently. That single idea changed science and engineering forever. Today, Euler’s Formula sits underneath: → Fourier Transforms → signal processing → wireless communication → MRI scanners → quantum mechanics → electrical engineering → audio compression → radar systems → GPS → neural network frequency analysis Even modern AI systems indirectly rely on mathematics built on top of these foundations. The famous special case is Euler’s Identity: e^(iπ) + 1 = 0 Richard Feynman reportedly called it: "our jewel." Because it revealed that mathematics is not a collection of separate topics. It is one connected language describing reality itself.

Most people first see Euler’s Formula as a strange equation in a textbook. Then years later, they realize it quietly powers the modern world. Leonhard Euler discovered that these seemingly unrelated mathematical ideas: → exponential growth → imaginary numbers → sine waves → cosine waves → rotation … are all deeply connected through one identity: e^(iθ) = cos(θ) + i·sin(θ) At first glance it looks impossible. How can an exponential function suddenly produce circles and waves? The key insight is that multiplying by a complex exponential creates rotation. As the angle θ changes: - the cosine term tracks horizontal motion - the sine term tracks vertical motion - together they trace a perfect circle in the complex plane Euler showed that waves and rotation are mathematically the same phenomenon viewed differently. That single idea changed science and engineering forever. Today, Euler’s Formula sits underneath: → Fourier Transforms → signal processing → wireless communication → MRI scanners → quantum mechanics → electrical engineering → audio compression → radar systems → GPS → neural network frequency analysis Even modern AI systems indirectly rely on mathematics built on top of these foundations. The famous special case is Euler’s Identity: e^(iπ) + 1 = 0 Richard Feynman reportedly called it: "our jewel." Because it revealed that mathematics is not a collection of separate topics. It is one connected language describing reality itself.

25,504 views • 5 months ago

No more content to load