Semi Doped
@semidoped • 9,766 subscribers
The industry powering AI, explained daily. Pod and takes by @vikramskr and @austinsemis. Sign up at https://t.co/wSYNiNJYqL
Videos
Intel's advanced packaging business has "billions of dollars" in commitments. But what if a capability few are watching is the real story? Intel has a strong optics process. Combined with EMIB, could they build a major advantage in co-packaged optics (CPO)?
Semi Doped100,605 просмотров • 2 месяцев назад
🎙️ NEW EPISODE: MicroVCSELs vs MicroLEDs, a MicroExplainer Austin and Vik host an explainer on the two competing microemitter technologies for AI interconnects, microLEDs and microVCSELs. They break down Credo's surprising pivot at OCP from its microLED acquisition to a microVCSEL product, and the physics that give microVCSELs better speed and reach. - The new battleground is the 5-50 meter "reach gap," too long for copper limited to ~7 meters and too short for expensive, supply-constrained InP optics - MicroLEDs are limited to ~3-5 Gbps over 5 meters because their broad light spectrum causes chromatic dispersion that smears the signal - Hitting 1 Tbps with microLEDs takes ~300 slow lanes, and coupling hundreds of fibers adds a lot of manufacturing complexity - MicroVCSELs can hit 50 Gbps+ over 50 meters. Stimulated emission gives a narrow beam with less dispersion, for 10x the speed and reach - MicroVCSELs use a mature Gallium Arsenide (GaAs) process on 8-inch wafers, shared with cell phones, for a cost and scale advantage over InP - MicroLEDs' niche is heat. Gallium Nitride (GaN) is highly temperature-insensitive, a candidate for short-reach optics inside a hot server tray This episode is presented by Crusoe. Crusoe's Serverless Fine-Tuning trains an open model like Llama, DeepSeek, or GLM on your own data, and you pay per million tokens, not per GPU hour. About $2 buys roughly 5M training tokens on smaller models. Get $5 in credit to start: Chapters: 0:00 The Credo Pivot 3:49 The Talk of OCP 8:13 The 5-50 Meter Reach Gap 12:52 How MicroLEDs Work 21:10 MicroLEDs' Parallelism Problem 22:55 How MicroVCSELs Work 26:57 MicroVCSEL Performance Specs 31:51 The Physics of VCSELs 33:35 The Market Chooses MicroVCSELs 35:45 MicroLEDs' Niche: Heat 38:16 Teasing Future Interviews Mentioned in this episode: - PicoJool’s Al Yuen: The Case for GaAs VCSELs in Scale-Up Interconnects (Semi Doped episode): - A MicroLED Field Trip to Avicena (Vik's Newsletter, paid): - MicroLEDs vs. Lasers: The Linewidth Tradeoff (Vik's Newsletter, free): - Understanding Energy Bands in Semiconductors (Vik's Newsletter, free): Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar
Semi Doped18,249 просмотров • 12 дней назад
Is Lumentum's laser moat disappearing? Co-packaged optics (CPO) shifts modulation onto silicon photonics chips (like Nvidia's), replacing complex EMLs with simpler, high-power (300-400mW) continuous-wave lasers. Essentially, a flashlight. Does the real moat still exist?
Semi Doped36,974 просмотров • 1 месяц назад
Vik and Val Bercovici (WEKA) map where AI inference memory is headed. Every 100x cut in KV cache gets swallowed by ~10,000x more usage, so demand climbs. - NVLink beats the board: 128 lanes vs 32 PCIe - WEKA serves NAND-backed storage faster than DRAM over network - DeepSeek's cache reads run ~87x cheaper, China only - CXL needs a dedicated bus, WEKA pools NAND over RDMA/NVLink instead - Val's call: SaaS giants and Neoclouds must merge Chapters: 0:00 Intro and memory prices 3:00 Model routing and offloading 5:10 Network faster than motherboard 13:06 Four-tier memory hierarchy 19:40 KV cache and Jevons paradox 23:50 DeepSeek cache read pricing 31:40 Sliding window attention 34:49 NAND tiers, SLC vs QLC 43:57 High bandwidth flash use cases 49:59 CXL versus NVLink 53:44 AMD Mex and small models 1:03:14 SaaS, Neoclouds and tokenomics Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar
Semi Doped61,062 просмотров • 3 месяцев назад
A masterclass on Google's TPU v8 Networking. Two TPU chips? Pssh. We already knew workload-specific silicon was here. But two scale-up networking topologies? That's the actual Google TPU news. Workload-specific interconnects. Think about that. New Semi Doped with Vikram Sekar and Austin Lyons. Copper? Yep. Optics? Yep. What we cover: - TPU splits in two: 8t training, 8i inference. - Virgo: 47 Pb/s scale-out fabric, 100% OCS. - Boardfly scale-up: copper PCB + AECs inside racks, OCS between groups. 16 hops → 7. - Training uses 3D torus (Rubik's Cube). - Inference doesn't. Workload-specific topologies now. - Dragonfly traces to a 2008 paper by Kim, Dally, Scott, Abts. Abts went on to build Groq's interconnect before Nvidia. Chapters: 0:00 Intro 0:21 Two TPUs for two workloads 2:31 HBM, SRAM, and Axion CPUs 7:22 Why networking is the new bottleneck 17:14 Virgo: rebuilding scale-out on optics 25:24 3D torus Rubik's Cube scale-up for training 34:50 Boardfly: scale-up for MoE inference 42:07 Workload-specific everything $GOOGL
Semi Doped93,264 просмотров • 6 месяцев назад
New episode: Advanced packaging for AI chips, from wire bonds to TSMC CoWOS and Intel EMIB. Packaging is no longer an afterthought. It is the chip, and Intel's EMIB challenges TSMC's CoWOS. - Three CoWOS flavors: silicon, organic RDL, local bridges - EMIB embeds tiny bridges into the substrate, no interposer - EMIB-T and EMIB-M add through-silicon vias and power capacitors - Google is booking 3M TPUs on EMIB via MediaTek by 2028 - Package sizes keep climbing: 5.5x reticle today, 40x ahead Chapters: (0:00) "There Is No Chip Without the Packaging" (0:28) Intro and SpaceX IPO Day (5:15) What We're Covering: CoWOS, EMIB, Google (7:40) Simple Packaging: Wire Bonds to Flip Chip (17:07) What Makes Packaging "Advanced" (33:44) CoWOS: Three Flavors Explained (45:30) EMIB: Intel's Embedded Bridge Approach (52:47) EMIB-T and EMIB-M (57:31) CoWOS vs. EMIB Trade-offs (1:02:18) Google's 3M TPU EMIB Order Austin Lyons Vikram Sekar
Semi Doped66,895 просмотров • 4 месяцев назад
🎙️ NEW EPISODE: Qualcomm's HBC vs HBM, Dragonfly AI 250, and Winning on TCO Austin sits down with Qualcomm's Durga Malladi live in Maui to talk HBC (High Bandwidth Compute). HBC is Qualcomm's differentiated bet to win data center inference workloads. HBM shuttles data across wide buses to a separate accelerator. HBC puts compute directly on the stacked DRAM's logic die. - HBM is fast but power-hungry; HBC drops latency AND power per bit - 18x effective bandwidth on the AI 250 at the SAME 768 GB and SAME 160 kW - A trillion-parameter FP4 model on a single card - Foundry + memory-vendor relationships de-risk multi-megawatt supply at scale Chapters: 0:00 Introduction 2:49 The Memory Wall Problem 8:05 HBC Physical Architecture 9:33 HBC vs. Custom HBM 11:50 Silicon Proof Points Coming 12:08 Managing Thermals 13:34 Dragonfly AI 250 14:48 Trillion-Parameter Model on One Card 16:56 Tokenomics and TCO 18:26 Why TCO is Key 19:26 Manufacturing at Scale Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar ✈️ Japan Qualcomm Snapdragon
Semi Doped22,380 просмотров • 1 месяц назад
New episode: Austin and Vik break down Qualcomm's Investor Day - HBC, Alphawave, Modular, more • 2/3 of revenue from data center, auto, and IoT by 2029, not handsets • HBC stacks LPDDR on the logic die, not beside it: 2,000 lanes become 100,000 • Disaggregation lets latecomers slot a decode rack beside Nvidia • Alphawave is like QCOM's Mellanox; Modular (Chris Lattner) is its CUDA • C1000 CPU wins Meta, but ships ~2 yrs into a CPU shortage Chapters: 0:00 Communications? That's just the start 4:08 Inside $QCOM investor day 9:16 Can Qualcomm build a data center business? 13:09 Disaggregated inference opens the door 17:57 High Bandwidth Compute: memory on the XPU 30:29 "No advanced packaging" just moves the problem 36:20 The roadmap, Alphawave, and Modular 46:00 The C1000 CPU and the agentic shortage 50:40 Cars as token generators, the $1T robotics bet 57:32 The memory market: MOAR Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar ✈️ Bay Area
Semi Doped47,365 просмотров • 4 месяцев назад
🎙️ NEW EPISODE: GlobalFoundries Thomas Barber: CPO, Silicon Photonics, 300mm, SiGe, OCI, NRZ "The enemy to me right now is copper," says Tom Barber. Austin sits down with Tom to unpack why GF thinks it can lead the shift to optical interconnects twice over: once with its long-running silicon photonics platform, and again with the specialty Silicon Germanium process the industry needs to drive it. Key Takeaways - CPO's real win isn't speed, it's power: saving 20-25 pJ/bit frees part of a data center's fixed 50-100 MW budget to go toward compute instead of moving bits. - GlobalFoundries leads photonics revenue for an unglamorous reason: it moved to 300mm wafers early, which yields 2.25x more die per wafer than the 200mm lines rivals still run. - CPO can end up more reliable than the pluggables it's replacing, not less, because it deletes the physical plug connector, and dust at that connector is the leading cause of field failures. - The OCI MSA picks NRZ over faster PAM4, deliberately going wide and slow, because NRZ's native bit error rate is a million times lower, which simplifies the receiver and cuts power. - GlobalFoundries stacks two specialty processes into one edge: micro-mirror couplers on the photonic side, and Silicon Germanium transistors hitting 350-400 GHz on the electrical side driving them. - The real competitor for silicon photonics isn't another foundry, it's copper itself — and that market is big enough that GlobalFoundries and TSMC can both win without taking share from each other. Chapters: 0:00 GlobalFoundries in Photonics 1:21 GF's Photonics Strategy 3:02 GF's Market Leadership 5:06 300mm Wafer Advantage 6:55 Copper's Range Limits 9:18 Pluggable to CPO 13:48 CPO Reliability 19:22 OCI MSA Explained 26:20 GF's Scale Platform 29:04 Micro Mirror Technology 31:32 Photonics vs. Copper 38:25 Silicon Germanium Advantage Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar
Semi Doped28,309 просмотров • 2 месяцев назад
FEELING HOT HOT HOT! 🎙️ NEW EPISODE: OpenAI's Jalapeño! Vik and Austin react to OpenAI's Jalapeño inference chip, announced at Hot Chips by Richard Ho @cdleary. - "Dark silicon is cheaper than idle accelerators": one balanced chip beats a GPU + LPU pair? - Nine months from RTL to tapeout, a Blackwell-class chip designed with AI for EDA - NUMA-style local HBM slices per accelerator fix the "operands arrive late" problem - Broadcom ESUN scale-up: 128 chips per rack at 600 GB/s, 2,048 chips across 16 racks at 200G (still scale up!) - The "regret factor": opportunity cost of missing a future model beats the cost of generality Plus the extra spicy questions: should OpenAI sell it, how much of the RTL was AI-written, and where is Anthropic's chip? Chapters: 0:00 Hot Chips Reaction 2:32 Designing for User Experience 11:16 A Generalized Inference Chip 14:18 The Foundry-IDM Analogy 18:42 The 'Regret Factor' 21:02 The 9-Month Design Cycle 23:45 Challenging the Two-Chip Solution 35:08 Solving HBM Underutilization 36:46 The NUMA Architecture Solution 39:28 System-Level ESUN Networking 42:06 Dark Silicon vs. Idle Accelerators 49:08 A Wake-Up Call for the Industry 52:59 Where's Anthropic's Chip? Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar
Semi Doped22,762 просмотров • 2 месяцев назад
Memory is eating the world When consumers feel the heat, you know its a problem. While big-3 memory companies are enjoying sweet-sweet profit margins…the rest of the electronics market is cracking under pricing pressure. - GoPro is more like GoingUnder - Apple products 10-30% pricier - Even music synths are more expensive Is it price gouging at this point? Is this revenge for all the memory cyclicality? When will this end? Chapters: 0:00 Memory crisis hits 1:20 AI impacting consumers 3:00 AI causing inflation 6:48 Consumer demand drop? 8:49 AI demand inelastic 10:56 Long-term memory outlook 11:02 GoPro's memory woes 12:24 Apple's pricing power 18:20 Apple seeks CXMT DRAM 21:38 Shrinkflation for phones 23:16 Korea's memory investment 26:10 Micron's killing profits 33:00 Why AI needs so much DRAM 40:27 Future of AI training 44:34 Cost-optimizing inference Get more of Austin and Vik daily, free! Sign up: Connect with Vik and Austin: Vik's Paid Substack: Austin's Paid Substack: Austin Lyons Vikram Sekar $MU
Semi Doped34,810 просмотров • 4 месяцев назад
Whiplash week. Optics? Copper? Both? Mon AM: Nvidia bets $4B on optics. Mon PM: Credo posts 200% YoY growth on copper. Wed PM: Hock Tan claims 400G/lane works over copper, potentially pushing CPO past 2030. 48 hours of whiplash. Optics? Copper? The answer is both. The question is when. This week and Vikram Sekar unpack: - Nvidia locking up laser supply - Credo’s blowout quarter and the reliability thesis - Broadcom’s copper bombshell - A 4D chess theory on why Hock Tan downplays optics when Broadcom is a CPO company Chapters (00:00) - Newsletter Plugs: Groq LPUs & Broadcom’s Laser Business (03:15) - Dynamo & the Rise of Workload-Specific Hardware (08:04) - Austin’s Broadcom Laser Deep Dive (09:53) - The Week’s Whiplash: Optics Monday, Copper Wednesday (17:50) - Why Nvidia Invested $4B: Geopolitics, Supply & the HBM Playbook (24:15) - CPO Lasers & Optical Circuit Switches (26:16) - Credo Earnings: 200% YoY Growth & the Copper Bull Case (31:09) - Reliability, AECs & Oracle’s GPU Cluster Problem (35:48) - Credo’s Optics Play: Micro-LED Active Cables & the CPO Timing Risk (38:45) - Broadcom Earnings: Hock Tan’s Copper Bombshell (43:34) - Customer-Owned Tooling: Hock Tan Says “Good Luck” (44:25) - Vik’s 4D Chess Theory: Why Hock Tan Talks Up Copper (47:03) - Wrap-Up: It’s Both — The Real Question Is Timing $AVGO $CRDO $NVDA
Semi Doped33,695 просмотров • 8 месяцев назад
HBM tops out ~8 TB/s. SRAM hits ~100 TB/s. The gap is physics: only ~2,000 lanes fit along the shoreline edge of a GPU die. Qualcomm's HBC idea: stack memory ON TOP of the XPU, use the whole die face. 2,000 lanes becomes 100,000. 100x lanes = 100x bandwidth.
Semi Doped12,170 просмотров • 3 месяцев назад
New interview: Reiner Pope, co-founder/CEO of MatX A counterintuitive throughput insight: “Low latency means small batch sizes. That is just Little’s law. Memory occupancy in HBM is proportional to batch size. So you can actually fit longer contexts than you could if the latency were larger. Low latency is not just a usability win, it improves throughput.” We get into: • The hybrid SRAM + HBM bet, and why pipeline parallelism finally works • Why sparse MoE drives MatX to “the most interconnect of any announced product” • Why frontier labs are willing to bet on an AI ASIC startup • Memory-bandwidth-efficient attention, numerics, and what MatX publishes (and what it does not) • Why 95% of model-side news is noise for chip design • The biggest challenges ahead 00:00 “We left Google one week before ChatGPT” 00:24 Intro: who is MatX 01:17 Origin story: leaving Google for LLM chips 02:21 GPT-3 and the “too expensive” problem 04:25 Why buy hardware that is not a GPU 05:52 Overcoming the CUDA moat 08:46 Early investors 09:35 The name MatX 09:59 The chip: matrix multiply + hybrid SRAM/HBM 12:11 Why pipeline parallelism finally works 14:22 Reading papers and Google going dark 15:20 Research agenda: attention and numerics 17:06 Five specs and meeting customers where they are 19:24 Why frontier labs are the natural first customer 20:32 Workloads: training, prefill, decode 22:18 Little’s law and the throughput case for low latency 24:29 Interconnect and MoE topology 26:35 Inside the team: 100 people, full stack 28:32 Agentic AI: 95% noise for hardware 30:35 KV cache sizing in an agentic world 32:11 How MatX uses AI for chip design (Verilog + BlueSpec) 34:23 Go to market: proving credibility under NDA 35:12 Porting effort for frontier labs 36:34 Biggest skepticism: manufacturing at gigawatt scale 37:32 Hiring plug Vikram Sekar
Semi Doped19,842 просмотров • 6 месяцев назад
The optical networking supercycle is here! In this podcast, and Vikram Sekar go through all the tech jargon and explain what everything means. In just about 45 mins, you will know everything required to keep up with the next revolution in AI. Chapters 00:00 Introduction to AI and CPU Bottlenecks 03:00 The Rise of Silicon Photonics 06:01 Understanding Optical Networking and Data Centers 08:49 Scale Across: Connecting Data Centers 11:56 Scale Out: Optimizing Data Center Connectivity 14:53 Scale Up: The Future of GPU Connectivity 23:32 The Shift from Copper to Optical Connections 26:13 Challenges and Reliability of Lasers 30:47 Understanding Co-Packaged Optics 34:17 Market Dynamics: Demand and Supply of Lasers 40:46 Emerging Technologies: Optical Circuit Switches
Semi Doped21,842 просмотров • 8 месяцев назад
This week we bring Fintwit’s favorite game to the podcast. “Would you buy this optical company?” The game goes like this. First explain: - What the company does - How it relates to the optics industry - What is their strength/moat - What is the risk/downside Then ask: Would you buy it? Why/Why not? Not financial advice. Just a silly, fun game. Educational. Better be serious when investing money. Pod bros don’t cut it. DYDD. Chapters (00:01) - Intro (06:59) - AXT $AXTI (13:38) - Tower Semiconductor $TSEM (23:58) - GlobalFoundries $GFS (32:43) - Lumentum $LITE (39:38) - Coherent $COHR (47:09) - Fabrinet $FN (54:07) - Corning $GLW Vikram Sekar ✈️ Japan
Semi Doped13,209 просмотров • 8 месяцев назад
Context memory essentially unlocks Agentic AI Much needed for Opus 4.6's "multi-agent swarms" In this SemiDoped pod, Vikram Sekar talks to Val Bercovici from Weka about context storage. - How token warehouses save inference costs - A new networking tier? Context Storage Network! - High Bandwidth Flash for context? - Weka's Augmented Memory Grid for context storage - Where this is all headed The convo is info packed. Don't miss out on it! b/acc, context platform engineer Chapters (00:00) Introduction to Weka and AI Storage Solutions (05:18) The Evolution of Context Memory in AI (09:30) Understanding Memory Hierarchies and Their Impact (16:24) Latency Challenges in Modern Storage Solutions (21:32) The Role of Networking in AI Storage Efficiency (29:42) Dynamic Resource Utilization in AI Networks (30:04) Introducing the Context Memory Network (31:13) High Bandwidth Flash: A Game Changer (32:54) Weka’s Neural Mesh and Storage Solutions (35:01) Axon: Transforming GPU Storage into Memory (39:00) Augmented Memory Grid Explained (42:00) Pooling DRAM and CXL Innovations (46:02) Token Warehouses and Inference Economics (52:10) The Future of Storage Innovations
Semi Doped12,796 просмотров • 8 месяцев назад
Больше нет контента для загрузки