Daily Dose of Data Science's banner

Daily Dose of Data Science

@DailyDoseOfDS_ • 50,338 subscribers

Delivering daily insights in DS, ML, RAGs, Agents & AI Engineering. Trusted by over 100k+ readers!

Shorts

$Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.$

Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.

55,488 views

NN-SVG: Create neural network architecture drawings parametrically! Export them to SVG files and use them in your work!

NN-SVG: Create neural network architecture drawings parametrically! Export them to SVG files and use them in your work!

109,894 views

NN-SVG: Create neural network architecture drawings parametrically! Export them to SVG files and use them in your work!

NN-SVG: Create neural network architecture drawings parametrically! Export them to SVG files and use them in your work!

71,347 views

K-Means is simple. Making it fast on GPU isn't. Flash-KMeans is an IO-aware implementation of exact k-means that rethinks the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 30x speedup over cuML - 200x speedup over FAISS Using the same exact algorithm, just engineered for today’s hardware. At the million-scale, Flash-KMeans can complete a k-means iteration in milliseconds. Here's why this matters today: K-means has always been an offline primitive. Something you run once to preprocess data and move on. These speedups change that. ↳ Vector databases like FAISS use k-means to build search indices. Faster k-means means you can re-index dynamically as data changes, not batch it overnight. ↳ LLM quantization methods need k-means to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Millisecond k-means makes it viable to run this inside the inference loop, not just in preprocessing. The 200x over FAISS is the number to internalize. FAISS is the industry standard. Most production vector search systems sit on top of it. Link to the paper and code in next tweet!

K-Means is simple. Making it fast on GPU isn't. Flash-KMeans is an IO-aware implementation of exact k-means that rethinks the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 30x speedup over cuML - 200x speedup over FAISS Using the same exact algorithm, just engineered for today’s hardware. At the million-scale, Flash-KMeans can complete a k-means iteration in milliseconds. Here's why this matters today: K-means has always been an offline primitive. Something you run once to preprocess data and move on. These speedups change that. ↳ Vector databases like FAISS use k-means to build search indices. Faster k-means means you can re-index dynamically as data changes, not batch it overnight. ↳ LLM quantization methods need k-means to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Millisecond k-means makes it viable to run this inside the inference loop, not just in preprocessing. The 200x over FAISS is the number to internalize. FAISS is the industry standard. Most production vector search systems sit on top of it. Link to the paper and code in next tweet!

23,748 views

Visualizing complex data in Python just got easier! Meet Cosmograph for Python 🪐: The widget brings GPU-accelerated, interactive layout graph rendering right inside your Jupyter notebooks. Here’s why it’s a game-changer: ⚡ GPU-accelerated performance ⛓️ Interactive network exploration with pan, zoom, hover & selection ⚙️ Rich configuration APIs for layout, color, size & more 📦 Seamless notebook integration & easy Python installation But that’s not all: ✅ Force-directed simulations for dynamic layouts ✅ Smooth handling of large-scale networks ✅ Minimal setup—just pip install cosmograph Link to the repo in next tweet! ______ Follow us → Daily Dose of Data Science ✔️ For more insights & tutorials on AI and Machine Learning.

Visualizing complex data in Python just got easier! Meet Cosmograph for Python 🪐: The widget brings GPU-accelerated, interactive layout graph rendering right inside your Jupyter notebooks. Here’s why it’s a game-changer: ⚡ GPU-accelerated performance ⛓️ Interactive network exploration with pan, zoom, hover & selection ⚙️ Rich configuration APIs for layout, color, size & more 📦 Seamless notebook integration & easy Python installation But that’s not all: ✅ Force-directed simulations for dynamic layouts ✅ Smooth handling of large-scale networks ✅ Minimal setup—just pip install cosmograph Link to the repo in next tweet! ______ Follow us → Daily Dose of Data Science ✔️ For more insights & tutorials on AI and Machine Learning.

27,283 views

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

Claude Code used 3x fewer tokens with one change. - Before: 10.4M tokens · 10 errors · $9.21 - After: 3.7M tokens · 0 errors · $2.81 (read the full setup guide below)

Claude Code used 3x fewer tokens with one change. - Before: 10.4M tokens · 10 errors · $9.21 - After: 3.7M tokens · 0 errors · $2.81 (read the full setup guide below)

Daily Dose of Data Science

377,641 views • 5 months ago

LLM inference speed with vs. without KV caching: (learn how and why it works below)

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Daily Dose of Data Science

59,218 views • 6 months ago

K-Means clustering, visually explained:

K-Means clustering, visually explained:

Daily Dose of Data Science

156,411 views • 2 years ago

This is the best way to understand how ML models actually work! Use Drawdata to draw a 2D dataset in Jupyter. Use it to actively pick data from the widget and update the model as the data is being drawn! Fully interactive, real-time, and open-source!

This is the best way to understand how ML models actually work! Use Drawdata to draw a 2D dataset in Jupyter. Use it to actively pick data from the widget and update the model as the data is being drawn! Fully interactive, real-time, and open-source!

Daily Dose of Data Science

52,132 views • 11 months ago

An MCP server to create Grant Sanderson animations (open-source):

An MCP server to create Grant Sanderson animations (open-source):

Daily Dose of Data Science

12,376 views • 11 months ago

Build RAG over excel sheets, a step-by-step guide:

Build RAG over excel sheets, a step-by-step guide:

Daily Dose of Data Science

15,737 views • 1 year ago

No more content to load