Loading video...

Video Failed to Load

Go Home

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization,...

MattBeton's profile picture

Matt Beton

2,570 subscribers

158,485 views • 1 year ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.
1:13

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

Alex Cheema

281,225 views • 11 months ago

Apple is no longer hiding it. These machines are built for local AI. Apple unveiled the new Mac Mini with M6 their first 2nm chip. Starting at $899. What you can run on the $899 Mac Mini (32GB) 👇 → Qwen 3.6 27B : fits comfortably, near-GPT-4 quality → Qwen 3 Coder 32B : full coding assistant on your desk → Llama 4 8B at full precision → DeepSeek R1 32B for reasoning tasks → 13.5x faster LLM processing than M1. 4x faster than M4. What you can run on the Mac Studio with M5 Ultra (512GB) 👇 → Llama 4 70B at high quality quantization → DeepSeek V4 with hundreds of billions of parameters → Frontier-scale models entirely in local memory → Connect four Mac Studios over Thunderbolt 5 cluster them to run trillion-parameter models locally Apple said their Mac business grew 30% last quarter largely because people are buying these to run AI models on their own desks instead of paying for cloud tokens. The future of inference is local. Apple built the hardware for it.
0:38

Apple is no longer hiding it. These machines are built for local AI. Apple unveiled the new Mac Mini with M6 their first 2nm chip. Starting at $899. What you can run on the $899 Mac Mini (32GB) 👇 → Qwen 3.6 27B : fits comfortably, near-GPT-4 quality → Qwen 3 Coder 32B : full coding assistant on your desk → Llama 4 8B at full precision → DeepSeek R1 32B for reasoning tasks → 13.5x faster LLM processing than M1. 4x faster than M4. What you can run on the Mac Studio with M5 Ultra (512GB) 👇 → Llama 4 70B at high quality quantization → DeepSeek V4 with hundreds of billions of parameters → Frontier-scale models entirely in local memory → Connect four Mac Studios over Thunderbolt 5 cluster them to run trillion-parameter models locally Apple said their Mac business grew 30% last quarter largely because people are buying these to run AI models on their own desks instead of paying for cloud tokens. The future of inference is local. Apple built the hardware for it.

Vaibhav Sisinty

517,005 views • 1 month ago

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.
0:20

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,604 views • 1 year ago

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇
0:33

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

LMSYS Org

170,690 views • 2 months ago

Introducing MTPLX V2.12: The fastest way to run local AI now even faster and with more models for lower memory macs and the new M5 Ultra. Upgrades: - 100 TPS Qwen Flash Next sustained on long coding (M5 max). - 2x Prefill speeds Qwen Flash Next: 872 --> 1695 TPS. - Requests with images now use the compiled kernels for max speeds. New Models: - Qwen Flash Next MTPLX Optimized Quality: 8 bit model built for the M5 Ultra and M3 Ultra. - Bonsai 2 27B MTPLX Optimized Speed: Works on 16gb and 24gb macs with compiled kernels and MTP running as fast as possible. 64 TPS. - MiMo V2.6 Qwen 3.5 9B MTPLX Optimized Speed: My personal favourite. Mimos magic distilled into a 9B model easily runnable at high speeds on a 16gb mac. 107 TPS. +40 bug fixes and improvements in real world coding.
0:24

Introducing MTPLX V2.12: The fastest way to run local AI now even faster and with more models for lower memory macs and the new M5 Ultra. Upgrades: - 100 TPS Qwen Flash Next sustained on long coding (M5 max). - 2x Prefill speeds Qwen Flash Next: 872 --> 1695 TPS. - Requests with images now use the compiled kernels for max speeds. New Models: - Qwen Flash Next MTPLX Optimized Quality: 8 bit model built for the M5 Ultra and M3 Ultra. - Bonsai 2 27B MTPLX Optimized Speed: Works on 16gb and 24gb macs with compiled kernels and MTP running as fast as possible. 64 TPS. - MiMo V2.6 Qwen 3.5 9B MTPLX Optimized Speed: My personal favourite. Mimos magic distilled into a 9B model easily runnable at high speeds on a 16gb mac. 107 TPS. +40 bug fixes and improvements in real world coding.

Youssof Al Toukhi

21,124 views • 10 days ago

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.
1:57

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

elvis

141,023 views • 1 year ago

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look
20:03

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look

Noisy

106,054 views • 3 months ago

What's the Big Deal with DeepSeek in AI? Here's why DeepSeek is making everyone take notice: 1. Super Smart on a Budget: DeepSeek showed you can make awesome AI without breaking the bank. Their latest model, DeepSeek-V3, was trained for only about $10 million, which is a lot less than the usual big bucks spent on AI, like the rumored $78 million for some of OpenAI's models. They did this in just two months with fewer fancy computers. 2. Open for Everyone: DeepSeek isn't keeping their tech a secret. They've made it open-source, meaning anyone can use, tweak, and learn from it. It's like they're saying, "Come join the party!" 3. Beating the Big Names: DeepSeek-V3 has done better than some top dogs from companies like OpenAI and Google in solving puzzles, math, and coding. This proves you can get great AI results without spending a fortune. 4. Challenging NVIDIA: NVIDIA's chips are usually the choice for AI because they're really powerful. But since DeepSeek did so well with less expensive chips, it might make people think twice about always going for NVIDIA's priciest options. 5. The DeepSeek Crew: The team at DeepSeek is young and smart, mostly from top Chinese schools, with brains in physics, math, and computer science. They learned AI in about six months by themselves! They use first principle thinking, which means they break down problems to the basics and build from there. This has helped them come up with cool new ways to do AI. 6. Changing AI for Good: DeepSeek is showing that AI can be cheaper and more open to everyone. They're changing how we think AI should be made and shared, which could shake up the whole AI world. So, as we watch DeepSeek, it's clear they're not just another player; they're changing the rules of the game. I predicted that this would be a make or break year for all the massive investments made in AI by American VC's. A few weeks later, DeepSeek happens! Watch the rest of my predictions in my 2025 outlook video . Link in replies #AIInnovation #DeepSeek #NVIDIA #OpenAI #TechDisruption
0:44

What's the Big Deal with DeepSeek in AI? Here's why DeepSeek is making everyone take notice: 1. Super Smart on a Budget: DeepSeek showed you can make awesome AI without breaking the bank. Their latest model, DeepSeek-V3, was trained for only about $10 million, which is a lot less than the usual big bucks spent on AI, like the rumored $78 million for some of OpenAI's models. They did this in just two months with fewer fancy computers. 2. Open for Everyone: DeepSeek isn't keeping their tech a secret. They've made it open-source, meaning anyone can use, tweak, and learn from it. It's like they're saying, "Come join the party!" 3. Beating the Big Names: DeepSeek-V3 has done better than some top dogs from companies like OpenAI and Google in solving puzzles, math, and coding. This proves you can get great AI results without spending a fortune. 4. Challenging NVIDIA: NVIDIA's chips are usually the choice for AI because they're really powerful. But since DeepSeek did so well with less expensive chips, it might make people think twice about always going for NVIDIA's priciest options. 5. The DeepSeek Crew: The team at DeepSeek is young and smart, mostly from top Chinese schools, with brains in physics, math, and computer science. They learned AI in about six months by themselves! They use first principle thinking, which means they break down problems to the basics and build from there. This has helped them come up with cool new ways to do AI. 6. Changing AI for Good: DeepSeek is showing that AI can be cheaper and more open to everyone. They're changing how we think AI should be made and shared, which could shake up the whole AI world. So, as we watch DeepSeek, it's clear they're not just another player; they're changing the rules of the game. I predicted that this would be a make or break year for all the massive investments made in AI by American VC's. A few weeks later, DeepSeek happens! Watch the rest of my predictions in my 2025 outlook video . Link in replies #AIInnovation #DeepSeek #NVIDIA #OpenAI #TechDisruption

Dr Ola Brown

83,460 views • 1 year ago

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office
0:20

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office

Noisy

22,391 views • 4 months ago

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.
2:41

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

37,459 views • 2 months ago

.Josh Wolfe: Anybody Using DeepSeek App Is 'Absolute Fool' "Anybody using the DeepSeek app is an absolute fool. If you're using DeepSeek on companies like Together Compute, one of Lux's companies, which can get rid of the CCP censorship, then it's probably okay. But remember, the open-source movement is something we deeply believe in. Most great technologists, entrepreneurs, and venture capitalists are on the side of open source. The closed-source models that have consumed tens of billions of dollars are the ones that are really going to be at risk. When you look at Hugging Face, a major repository, or Together Compute, Runway ML, and a lot of Lux's companies, they have been pioneers in open source. Now, why am I not worried about open source, even with the DeepSeek model? As long as you don't have the CCP censorship on it, the models with their open weights allow people to run on their proprietary data. This means companies like pharma or defense companies that have their own siloed, proprietary data—think about Bloomberg with their proprietary longitudinal data, or Meta with their data—are the ones who will have the edge. Even as open source takes hold, these companies will still dominate. I’m not worried about open source being the problem. I’m more concerned about people overfunding closed models with no proprietary source. A lot of capital is going to be burned there, and we’re already seeing that with people worried about OpenAI in some aspects."
1:27

.Josh Wolfe: Anybody Using DeepSeek App Is 'Absolute Fool' "Anybody using the DeepSeek app is an absolute fool. If you're using DeepSeek on companies like Together Compute, one of Lux's companies, which can get rid of the CCP censorship, then it's probably okay. But remember, the open-source movement is something we deeply believe in. Most great technologists, entrepreneurs, and venture capitalists are on the side of open source. The closed-source models that have consumed tens of billions of dollars are the ones that are really going to be at risk. When you look at Hugging Face, a major repository, or Together Compute, Runway ML, and a lot of Lux's companies, they have been pioneers in open source. Now, why am I not worried about open source, even with the DeepSeek model? As long as you don't have the CCP censorship on it, the models with their open weights allow people to run on their proprietary data. This means companies like pharma or defense companies that have their own siloed, proprietary data—think about Bloomberg with their proprietary longitudinal data, or Meta with their data—are the ones who will have the edge. Even as open source takes hold, these companies will still dominate. I’m not worried about open source being the problem. I’m more concerned about people overfunding closed models with no proprietary source. A lot of capital is going to be burned there, and we’re already seeing that with people worried about OpenAI in some aspects."

Josh Caplan

40,039 views • 1 year ago

Been giving my M3 Ultra Studio some love and tinkering with MLX. Started the afternoon at 42 tok/s. Now: 73 tok/s, and there’s still more on the table. Running Qwen3.8-Flash-Next @ 4-bit locally. Now for the crazy bit: Artificial Analysis Intelligence Index v4.3: GPT-5.6 Sol 39 (Medium) GPT-5.6 Sol 42 (High) Qwen3.8-Flash-Next 42 GPT-5.6 Sol 44 (XHigh) GPT-5.6 Sol 47 (Max) Grok 4.5 39 (High) Claude Sonnet 5 38 (Max) *Qwen score currently estimated by Artificial Analysis. A 4-bit open-weights model running at 73 tok/s on a Mac under my desk... with an estimated intelligence score matching GPT-5.6 Sol at High reasoning. Local AI is getting ridiculous. 🔥 Plus, this model does not say no to a bit of red teaming. Once I am happy with it, I will do a pr for Jun Kim's oMLX :)
0:48

Been giving my M3 Ultra Studio some love and tinkering with MLX. Started the afternoon at 42 tok/s. Now: 73 tok/s, and there’s still more on the table. Running Qwen3.8-Flash-Next @ 4-bit locally. Now for the crazy bit: Artificial Analysis Intelligence Index v4.3: GPT-5.6 Sol 39 (Medium) GPT-5.6 Sol 42 (High) Qwen3.8-Flash-Next 42 GPT-5.6 Sol 44 (XHigh) GPT-5.6 Sol 47 (Max) Grok 4.5 39 (High) Claude Sonnet 5 38 (Max) *Qwen score currently estimated by Artificial Analysis. A 4-bit open-weights model running at 73 tok/s on a Mac under my desk... with an estimated intelligence score matching GPT-5.6 Sol at High reasoning. Local AI is getting ridiculous. 🔥 Plus, this model does not say no to a bit of red teaming. Once I am happy with it, I will do a pr for Jun Kim's oMLX :)

Ash Hart

18,827 views • 25 days ago

Google just had its DeepSeek moment — and almost nobody's talking about it. Here's the story you need to know. 🧵 When DeepSeek dropped in early 2025, it didn't just impress people. It scared them. A model competing with the biggest AI players — at a fraction of the cost — through math, not hardware. Chip stocks tanked. The industry panicked. Then on March 24th, 2026, Google published a research paper called TurboQuant. Cloudflare's CEO immediately called it Google's DeepSeek moment. Memory chip stocks for Micron and Western Digital fell on the news. Why? Because TurboQuant compresses AI memory by 6x — through software alone. → No new chips needed → No retraining required → One server can now host more models than before The era of throwing hardware at AI problems is ending. The era of mathematical efficiency is here. ✅Save this post, you'll thank yourself when this reshapes every AI tool you use. 📌 Want the SOP? DM me.
8:21

Google just had its DeepSeek moment — and almost nobody's talking about it. Here's the story you need to know. 🧵 When DeepSeek dropped in early 2025, it didn't just impress people. It scared them. A model competing with the biggest AI players — at a fraction of the cost — through math, not hardware. Chip stocks tanked. The industry panicked. Then on March 24th, 2026, Google published a research paper called TurboQuant. Cloudflare's CEO immediately called it Google's DeepSeek moment. Memory chip stocks for Micron and Western Digital fell on the news. Why? Because TurboQuant compresses AI memory by 6x — through software alone. → No new chips needed → No retraining required → One server can now host more models than before The era of throwing hardware at AI problems is ending. The era of mathematical efficiency is here. ✅Save this post, you'll thank yourself when this reshapes every AI tool you use. 📌 Want the SOP? DM me.

Julian Goldie SEO

12,731 views • 6 months ago

U.S. Navy Bans DeepSeek Over 'Security Concerns' As 'Substantial' Evidence Emerges Chinese AI Ripped Off ChatGPT | ZeroHedge The U.S. Navy has instructed service members to avoid using the Chinese AI platform DeepSeek, citing "potential security and ethical concerns," according to CNBC. An email sent to "shipmates" in recent days, confirmed by CNBC on Tuesday, referenced the Navy's AI policy and emphasized the importance of refraining from using DeepSeek. The memo warned service members against using the platform "for any work-related tasks or personal use" and instructed them to "avoid downloading, installing, or using the DeepSeek model in any capacity." The warning follows the recent rise of DeepSeek’s R1 model, which has garnered significant attention worldwide, particularly within the U.S. business and technology sectors. The R1 model has demonstrated capabilities comparable to OpenAI’s models. In December, DeepSeek claimed it had successfully trained a large language model in just two months at a cost of $6 million—a figure disputed by technologists—despite U.S. restrictions on semiconductor chip exports to China. The R1, an open-source model, surged to the top of Apple’s app store rankings this week, triggering a market sell-off. Shares of AI chipmakers Nvidia and Broadcom plummeted by 17% on Monday, wiping out a combined $800 billion in market value. Nvidia has since recovered some of its losses. On Monday, DeepSeek announced a temporary restriction on user registrations, citing "large-scale malicious attacks" on its services, before later restoring normal operations. DeepSeek’s advancements have challenged the long-held belief that the U.S. was significantly ahead of China in AI development. Asked how R1 caught up to ChatGPT, AI and Crypto Czar David Sacks suggested that DeepSeek may have leveraged a technique known as "distillation" to train its model using OpenAI’s technology. “There’s a technique in AI called distillation, which you’re going to hear a lot about. It’s when one model learns from another model,” Sacks explained to Fox News. “Effectively, the student model asks the parent model millions of questions, mimicking the reasoning process and absorbing knowledge.” “They can essentially extract the knowledge out of the model,” he continued. “There’s substantial evidence that what DeepSeek did here was distill knowledge from OpenAI’s models.” “I don’t think OpenAI is too happy about this,” Sacks added. President Donald Trump has said that DeepSeek “should be a wake-up call” for U.S. tech companies. “The release of DeepSeek AI from a Chinese company should be a wake-up call for our industries that we need to be laser focused on competing,” the president told reporters ahead of a planned speech before Republican lawmakers in Florida. Read more:
1:09

U.S. Navy Bans DeepSeek Over 'Security Concerns' As 'Substantial' Evidence Emerges Chinese AI Ripped Off ChatGPT | ZeroHedge The U.S. Navy has instructed service members to avoid using the Chinese AI platform DeepSeek, citing "potential security and ethical concerns," according to CNBC. An email sent to "shipmates" in recent days, confirmed by CNBC on Tuesday, referenced the Navy's AI policy and emphasized the importance of refraining from using DeepSeek. The memo warned service members against using the platform "for any work-related tasks or personal use" and instructed them to "avoid downloading, installing, or using the DeepSeek model in any capacity." The warning follows the recent rise of DeepSeek’s R1 model, which has garnered significant attention worldwide, particularly within the U.S. business and technology sectors. The R1 model has demonstrated capabilities comparable to OpenAI’s models. In December, DeepSeek claimed it had successfully trained a large language model in just two months at a cost of $6 million—a figure disputed by technologists—despite U.S. restrictions on semiconductor chip exports to China. The R1, an open-source model, surged to the top of Apple’s app store rankings this week, triggering a market sell-off. Shares of AI chipmakers Nvidia and Broadcom plummeted by 17% on Monday, wiping out a combined $800 billion in market value. Nvidia has since recovered some of its losses. On Monday, DeepSeek announced a temporary restriction on user registrations, citing "large-scale malicious attacks" on its services, before later restoring normal operations. DeepSeek’s advancements have challenged the long-held belief that the U.S. was significantly ahead of China in AI development. Asked how R1 caught up to ChatGPT, AI and Crypto Czar David Sacks suggested that DeepSeek may have leveraged a technique known as "distillation" to train its model using OpenAI’s technology. “There’s a technique in AI called distillation, which you’re going to hear a lot about. It’s when one model learns from another model,” Sacks explained to Fox News. “Effectively, the student model asks the parent model millions of questions, mimicking the reasoning process and absorbing knowledge.” “They can essentially extract the knowledge out of the model,” he continued. “There’s substantial evidence that what DeepSeek did here was distill knowledge from OpenAI’s models.” “I don’t think OpenAI is too happy about this,” Sacks added. President Donald Trump has said that DeepSeek “should be a wake-up call” for U.S. tech companies. “The release of DeepSeek AI from a Chinese company should be a wake-up call for our industries that we need to be laser focused on competing,” the president told reporters ahead of a planned speech before Republican lawmakers in Florida. Read more:

Owen Gregorian

75,351 views • 1 year ago

** MEGA Parodius Scaling Effects Part 1 ** One of the big challenges with the Parodius Megadrive port is Stage 8's boss - The puffer-fish *Pooyan* with his full screen scaling effect. The goal is to be very close to the arcade (with extras on top ) so I thought lets tackle it head on to see how close we can get. I was also keen to jump into another scaling code rabit hole haha. Pyron pulled out all the stops and got me the source frames and reworked the BG tiles for this test - a big thankyou to him , Vector Orbitex is busy working on Stage 2 tracks so the team is working hard all round on this port. The MD has no sprite / background GFX scaling hardware , however the VDPs Vertical scroll can be updated per scanline to help vertical scaling on backgrounds, but there is a cpu cost to manage all the interupts so thats not free either. With the Horizontal scaling there is no help at all , apart from a semi-friendly packed pixel format for the cpu to work with, its not quite chunky format but better than planar format still for scaling. So its falls back to the 68k CPU to do all of the horizontal expansion which is the largest cpu cost. Basically drawing strips of either 1x, 2x, 3x or 4x wide columns at speed. So we are one week into this Boss's routine and you can see from the below video the horizontal scaling is implented ( vertical will be in the next update ) . We are scaling from 1x to 4x in the video below in 74 steps for testing . The column distributions are always a bit painfull to do - thankfully they are all worked out now. This is the third scaler I have built and the goal was with this one to make it really flexible for use in other projects also, sometimes when you optimise something to the last degree all the flexibility gets taken out of it. Currenty scaling at 12-25 FPS update here, I had some rules against some optimisations which I would use and some I wouldn't , thankfully we are a bit ahead of the Arcades animation frame rate here still and I may yet find optimisations that fit within the scope. We have vertical scaling and sprite spikes to add yet so Im hoping i can find a few more optimisations to offset things when they are implemented also. In a scale frame update we are processing close to 42000 pixels in ram before using DMA to send to VRAM . Using a 41x16 (656 tile scale buffer) - single buffered for now due to its size in VRAM. So thats nearly 21k in tiles ! I had to re-organise ram a bit to support a buffer of that size for the stage. The scaling function is written in 68k assembly , with a little C code handling the Vertical interupt code ( so the game logic can actually run & DMA updates etc ) . The DMA routines are in assembly also and customised for large chunk size ( big blocks of tiles ) which suits the scaler. I had some race conditions to sort out where the cpu was faster than DMA (sending tiles from RAM to VRAM ) and in some cases where it wasn't so it had to be balanced. We may be able to add more detail into the top and bottom of the background yet but its low priority for now until all the other bits are in !! #SGDK #SegaMegadrive #Genesis #Parodius
0:30

** MEGA Parodius Scaling Effects Part 1 ** One of the big challenges with the Parodius Megadrive port is Stage 8's boss - The puffer-fish *Pooyan* with his full screen scaling effect. The goal is to be very close to the arcade (with extras on top ) so I thought lets tackle it head on to see how close we can get. I was also keen to jump into another scaling code rabit hole haha. Pyron pulled out all the stops and got me the source frames and reworked the BG tiles for this test - a big thankyou to him , Vector Orbitex is busy working on Stage 2 tracks so the team is working hard all round on this port. The MD has no sprite / background GFX scaling hardware , however the VDPs Vertical scroll can be updated per scanline to help vertical scaling on backgrounds, but there is a cpu cost to manage all the interupts so thats not free either. With the Horizontal scaling there is no help at all , apart from a semi-friendly packed pixel format for the cpu to work with, its not quite chunky format but better than planar format still for scaling. So its falls back to the 68k CPU to do all of the horizontal expansion which is the largest cpu cost. Basically drawing strips of either 1x, 2x, 3x or 4x wide columns at speed. So we are one week into this Boss's routine and you can see from the below video the horizontal scaling is implented ( vertical will be in the next update ) . We are scaling from 1x to 4x in the video below in 74 steps for testing . The column distributions are always a bit painfull to do - thankfully they are all worked out now. This is the third scaler I have built and the goal was with this one to make it really flexible for use in other projects also, sometimes when you optimise something to the last degree all the flexibility gets taken out of it. Currenty scaling at 12-25 FPS update here, I had some rules against some optimisations which I would use and some I wouldn't , thankfully we are a bit ahead of the Arcades animation frame rate here still and I may yet find optimisations that fit within the scope. We have vertical scaling and sprite spikes to add yet so Im hoping i can find a few more optimisations to offset things when they are implemented also. In a scale frame update we are processing close to 42000 pixels in ram before using DMA to send to VRAM . Using a 41x16 (656 tile scale buffer) - single buffered for now due to its size in VRAM. So thats nearly 21k in tiles ! I had to re-organise ram a bit to support a buffer of that size for the stage. The scaling function is written in 68k assembly , with a little C code handling the Vertical interupt code ( so the game logic can actually run & DMA updates etc ) . The DMA routines are in assembly also and customised for large chunk size ( big blocks of tiles ) which suits the scaler. I had some race conditions to sort out where the cpu was faster than DMA (sending tiles from RAM to VRAM ) and in some cases where it wasn't so it had to be balanced. We may be able to add more detail into the top and bottom of the background yet but its low priority for now until all the other bits are in !! #SGDK #SegaMegadrive #Genesis #Parodius

Shannon Birt

25,803 views • 8 months ago

TWITTER ORDERED THESE EPYC CHIPS AND NEVER TOOK DELIVERY. HE BOUGHT THE ORPHANED BATCH, STACKED 512GB OF RAM AND NOW RUNS DEEPSEEK 671B UNQUANTIZED WHILE ANTHROPIC BILLS $200/MO FOR THE SAME CLASS OF MODEL 00:17 he holds the cpu up and says "these were custom made I think for twitter and then all the twitter stuff happened, so they never got put to use" amd epyc 7v13, the cancelled twitter batch -> supermicro h12ssl-nt, 8 memory channels -> 8x 64gb ddr4-3200, 512gb total -> noctua sp3 cooler -> gutted an old atx case -> the whole model loads into ram, no swap, no quantization his gaming rig has a 9950x3d and 128gb of ddr5 and only two memory channels, this board has eight, so older ddr4 on the epyc moves more data per second than the newer ram on the gaming machine, that's the whole reason cpu inference works at all here the honest part: he expects 1 to 4 tokens per second, which is slower than reading speed, you run this because it holds a 671b model at full precision, not because it's fast amd epyc 7v13 + supermicro h12ssl-nt + 512gb ddr4-3200 across 8 channels + noctua sp3 + deepseek 671b + llama 405b watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
2:51

TWITTER ORDERED THESE EPYC CHIPS AND NEVER TOOK DELIVERY. HE BOUGHT THE ORPHANED BATCH, STACKED 512GB OF RAM AND NOW RUNS DEEPSEEK 671B UNQUANTIZED WHILE ANTHROPIC BILLS $200/MO FOR THE SAME CLASS OF MODEL 00:17 he holds the cpu up and says "these were custom made I think for twitter and then all the twitter stuff happened, so they never got put to use" amd epyc 7v13, the cancelled twitter batch -> supermicro h12ssl-nt, 8 memory channels -> 8x 64gb ddr4-3200, 512gb total -> noctua sp3 cooler -> gutted an old atx case -> the whole model loads into ram, no swap, no quantization his gaming rig has a 9950x3d and 128gb of ddr5 and only two memory channels, this board has eight, so older ddr4 on the epyc moves more data per second than the newer ram on the gaming machine, that's the whole reason cpu inference works at all here the honest part: he expects 1 to 4 tokens per second, which is slower than reading speed, you run this because it holds a 671b model at full precision, not because it's fast amd epyc 7v13 + supermicro h12ssl-nt + 512gb ddr4-3200 across 8 channels + noctua sp3 + deepseek 671b + llama 405b watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

10,074 views • 1 month ago

50% of my consulting work right now is helping companies use open-source models at scale. Everyone knows how to use an open-source LLM on their computers, but it's really hard to do this at scale for thousands of users. Here is how this plays out: 1. A team builds a prototype using DeepSeek. 2. Everything looks good. It works! 3. They follow an online guide to deploy the model online. 4. They ask 10 users to try the app. 5. Latency spikes everywhere. 6. The entire system halts. 7. They blame DeepSeek and try again using a new model. The problem is always with scaling inference, not the model. Here is one recommendation I give companies: Check out Nebius Token Factory if you don't want to ever think about deploying an open-source model again. This is a managed inference platform for deploying open-source LLMs at scale. This is not for prototypes or research experiments. This is for when you have a real application with real users. Three important notes about Token Factory: • You have complete control over how inference runs. • You have predictable tail latency (P99, not averages). • No surprise costs when you scale up. You can preplan your budget. Check it out here: Here are two codes you can use to get 100 hours of GPU usage on me: ymJLFa2ARYSKEdqb AdckcZaYjm7KqYY7 Thanks to the Nebius team for their continuous partnership.
2:40

50% of my consulting work right now is helping companies use open-source models at scale. Everyone knows how to use an open-source LLM on their computers, but it's really hard to do this at scale for thousands of users. Here is how this plays out: 1. A team builds a prototype using DeepSeek. 2. Everything looks good. It works! 3. They follow an online guide to deploy the model online. 4. They ask 10 users to try the app. 5. Latency spikes everywhere. 6. The entire system halts. 7. They blame DeepSeek and try again using a new model. The problem is always with scaling inference, not the model. Here is one recommendation I give companies: Check out Nebius Token Factory if you don't want to ever think about deploying an open-source model again. This is a managed inference platform for deploying open-source LLMs at scale. This is not for prototypes or research experiments. This is for when you have a real application with real users. Three important notes about Token Factory: • You have complete control over how inference runs. • You have predictable tail latency (P99, not averages). • No surprise costs when you scale up. You can preplan your budget. Check it out here: Here are two codes you can use to get 100 hours of GPU usage on me: ymJLFa2ARYSKEdqb AdckcZaYjm7KqYY7 Thanks to the Nebius team for their continuous partnership.

Santiago

48,605 views • 8 months ago

The Bull Case for Linear Being a $100BN Company: "Linear is the clear winner. They have built an agentic product first that allows us to build 100x more software, and that means 100x more features than ever before. Humans cannot keep up with it, and humans still have to work with agents. If every human on your team's gonna build 500 features and 1,000 issues and you have 10 people on your team, you need a process and a system of record for managing all these issues with your agents. We need a new system of record for it. The team at Linear has figured it out. We have seen an explosion, 50x more agent usage than 90 days ago. This will seem cheap when Replit's at $15BN, Lovable's at $100BN, because Linear will be the one powering them all". Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts, what is your bull case for Linear Karri Saarinen Stephanie Zhan adam bain dick costolo Soleio Dylan Field Miles Clements
1:22

The Bull Case for Linear Being a $100BN Company: "Linear is the clear winner. They have built an agentic product first that allows us to build 100x more software, and that means 100x more features than ever before. Humans cannot keep up with it, and humans still have to work with agents. If every human on your team's gonna build 500 features and 1,000 issues and you have 10 people on your team, you need a process and a system of record for managing all these issues with your agents. We need a new system of record for it. The team at Linear has figured it out. We have seen an explosion, 50x more agent usage than 90 days ago. This will seem cheap when Replit's at $15BN, Lovable's at $100BN, because Linear will be the one powering them all". Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts, what is your bull case for Linear Karri Saarinen Stephanie Zhan adam bain dick costolo Soleio Dylan Field Miles Clements

Harry Stebbings

197,926 views • 1 month ago

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

131,099 views • 1 year ago