Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization,...

MattBeton's profile picture

Matt Beton

2,570 subscribers

158,485 Aufrufe • vor 11 Monaten •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.
1:13

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

Alex Cheema

281,225 Aufrufe • vor 10 Monaten

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.
0:20

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,604 Aufrufe • vor 1 Jahr

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇
0:33

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

LMSYS Org

169,652 Aufrufe • vor 1 Monat

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.
1:57

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

elvis

140,692 Aufrufe • vor 1 Jahr

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look
20:03

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look

Noisy

105,595 Aufrufe • vor 1 Monat

What's the Big Deal with DeepSeek in AI? Here's why DeepSeek is making everyone take notice: 1. Super Smart on a Budget: DeepSeek showed you can make awesome AI without breaking the bank. Their latest model, DeepSeek-V3, was trained for only about $10 million, which is a lot less than the usual big bucks spent on AI, like the rumored $78 million for some of OpenAI's models. They did this in just two months with fewer fancy computers. 2. Open for Everyone: DeepSeek isn't keeping their tech a secret. They've made it open-source, meaning anyone can use, tweak, and learn from it. It's like they're saying, "Come join the party!" 3. Beating the Big Names: DeepSeek-V3 has done better than some top dogs from companies like OpenAI and Google in solving puzzles, math, and coding. This proves you can get great AI results without spending a fortune. 4. Challenging NVIDIA: NVIDIA's chips are usually the choice for AI because they're really powerful. But since DeepSeek did so well with less expensive chips, it might make people think twice about always going for NVIDIA's priciest options. 5. The DeepSeek Crew: The team at DeepSeek is young and smart, mostly from top Chinese schools, with brains in physics, math, and computer science. They learned AI in about six months by themselves! They use first principle thinking, which means they break down problems to the basics and build from there. This has helped them come up with cool new ways to do AI. 6. Changing AI for Good: DeepSeek is showing that AI can be cheaper and more open to everyone. They're changing how we think AI should be made and shared, which could shake up the whole AI world. So, as we watch DeepSeek, it's clear they're not just another player; they're changing the rules of the game. I predicted that this would be a make or break year for all the massive investments made in AI by American VC's. A few weeks later, DeepSeek happens! Watch the rest of my predictions in my 2025 outlook video . Link in replies #AIInnovation #DeepSeek #NVIDIA #OpenAI #TechDisruption
0:44

What's the Big Deal with DeepSeek in AI? Here's why DeepSeek is making everyone take notice: 1. Super Smart on a Budget: DeepSeek showed you can make awesome AI without breaking the bank. Their latest model, DeepSeek-V3, was trained for only about $10 million, which is a lot less than the usual big bucks spent on AI, like the rumored $78 million for some of OpenAI's models. They did this in just two months with fewer fancy computers. 2. Open for Everyone: DeepSeek isn't keeping their tech a secret. They've made it open-source, meaning anyone can use, tweak, and learn from it. It's like they're saying, "Come join the party!" 3. Beating the Big Names: DeepSeek-V3 has done better than some top dogs from companies like OpenAI and Google in solving puzzles, math, and coding. This proves you can get great AI results without spending a fortune. 4. Challenging NVIDIA: NVIDIA's chips are usually the choice for AI because they're really powerful. But since DeepSeek did so well with less expensive chips, it might make people think twice about always going for NVIDIA's priciest options. 5. The DeepSeek Crew: The team at DeepSeek is young and smart, mostly from top Chinese schools, with brains in physics, math, and computer science. They learned AI in about six months by themselves! They use first principle thinking, which means they break down problems to the basics and build from there. This has helped them come up with cool new ways to do AI. 6. Changing AI for Good: DeepSeek is showing that AI can be cheaper and more open to everyone. They're changing how we think AI should be made and shared, which could shake up the whole AI world. So, as we watch DeepSeek, it's clear they're not just another player; they're changing the rules of the game. I predicted that this would be a make or break year for all the massive investments made in AI by American VC's. A few weeks later, DeepSeek happens! Watch the rest of my predictions in my 2025 outlook video . Link in replies #AIInnovation #DeepSeek #NVIDIA #OpenAI #TechDisruption

Dr Ola Brown

83,460 Aufrufe • vor 1 Jahr

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office
0:20

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office

Noisy

22,391 Aufrufe • vor 2 Monaten

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.
2:41

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

37,388 Aufrufe • vor 28 Tagen

.Josh Wolfe: Anybody Using DeepSeek App Is 'Absolute Fool' "Anybody using the DeepSeek app is an absolute fool. If you're using DeepSeek on companies like Together Compute, one of Lux's companies, which can get rid of the CCP censorship, then it's probably okay. But remember, the open-source movement is something we deeply believe in. Most great technologists, entrepreneurs, and venture capitalists are on the side of open source. The closed-source models that have consumed tens of billions of dollars are the ones that are really going to be at risk. When you look at Hugging Face, a major repository, or Together Compute, Runway ML, and a lot of Lux's companies, they have been pioneers in open source. Now, why am I not worried about open source, even with the DeepSeek model? As long as you don't have the CCP censorship on it, the models with their open weights allow people to run on their proprietary data. This means companies like pharma or defense companies that have their own siloed, proprietary data—think about Bloomberg with their proprietary longitudinal data, or Meta with their data—are the ones who will have the edge. Even as open source takes hold, these companies will still dominate. I’m not worried about open source being the problem. I’m more concerned about people overfunding closed models with no proprietary source. A lot of capital is going to be burned there, and we’re already seeing that with people worried about OpenAI in some aspects."
1:27

.Josh Wolfe: Anybody Using DeepSeek App Is 'Absolute Fool' "Anybody using the DeepSeek app is an absolute fool. If you're using DeepSeek on companies like Together Compute, one of Lux's companies, which can get rid of the CCP censorship, then it's probably okay. But remember, the open-source movement is something we deeply believe in. Most great technologists, entrepreneurs, and venture capitalists are on the side of open source. The closed-source models that have consumed tens of billions of dollars are the ones that are really going to be at risk. When you look at Hugging Face, a major repository, or Together Compute, Runway ML, and a lot of Lux's companies, they have been pioneers in open source. Now, why am I not worried about open source, even with the DeepSeek model? As long as you don't have the CCP censorship on it, the models with their open weights allow people to run on their proprietary data. This means companies like pharma or defense companies that have their own siloed, proprietary data—think about Bloomberg with their proprietary longitudinal data, or Meta with their data—are the ones who will have the edge. Even as open source takes hold, these companies will still dominate. I’m not worried about open source being the problem. I’m more concerned about people overfunding closed models with no proprietary source. A lot of capital is going to be burned there, and we’re already seeing that with people worried about OpenAI in some aspects."

Josh Caplan

40,039 Aufrufe • vor 1 Jahr

Google just had its DeepSeek moment — and almost nobody's talking about it. Here's the story you need to know. 🧵 When DeepSeek dropped in early 2025, it didn't just impress people. It scared them. A model competing with the biggest AI players — at a fraction of the cost — through math, not hardware. Chip stocks tanked. The industry panicked. Then on March 24th, 2026, Google published a research paper called TurboQuant. Cloudflare's CEO immediately called it Google's DeepSeek moment. Memory chip stocks for Micron and Western Digital fell on the news. Why? Because TurboQuant compresses AI memory by 6x — through software alone. → No new chips needed → No retraining required → One server can now host more models than before The era of throwing hardware at AI problems is ending. The era of mathematical efficiency is here. ✅Save this post, you'll thank yourself when this reshapes every AI tool you use. 📌 Want the SOP? DM me.
8:21

Google just had its DeepSeek moment — and almost nobody's talking about it. Here's the story you need to know. 🧵 When DeepSeek dropped in early 2025, it didn't just impress people. It scared them. A model competing with the biggest AI players — at a fraction of the cost — through math, not hardware. Chip stocks tanked. The industry panicked. Then on March 24th, 2026, Google published a research paper called TurboQuant. Cloudflare's CEO immediately called it Google's DeepSeek moment. Memory chip stocks for Micron and Western Digital fell on the news. Why? Because TurboQuant compresses AI memory by 6x — through software alone. → No new chips needed → No retraining required → One server can now host more models than before The era of throwing hardware at AI problems is ending. The era of mathematical efficiency is here. ✅Save this post, you'll thank yourself when this reshapes every AI tool you use. 📌 Want the SOP? DM me.

Julian Goldie SEO

12,731 Aufrufe • vor 4 Monaten

U.S. Navy Bans DeepSeek Over 'Security Concerns' As 'Substantial' Evidence Emerges Chinese AI Ripped Off ChatGPT | ZeroHedge The U.S. Navy has instructed service members to avoid using the Chinese AI platform DeepSeek, citing "potential security and ethical concerns," according to CNBC. An email sent to "shipmates" in recent days, confirmed by CNBC on Tuesday, referenced the Navy's AI policy and emphasized the importance of refraining from using DeepSeek. The memo warned service members against using the platform "for any work-related tasks or personal use" and instructed them to "avoid downloading, installing, or using the DeepSeek model in any capacity." The warning follows the recent rise of DeepSeek’s R1 model, which has garnered significant attention worldwide, particularly within the U.S. business and technology sectors. The R1 model has demonstrated capabilities comparable to OpenAI’s models. In December, DeepSeek claimed it had successfully trained a large language model in just two months at a cost of $6 million—a figure disputed by technologists—despite U.S. restrictions on semiconductor chip exports to China. The R1, an open-source model, surged to the top of Apple’s app store rankings this week, triggering a market sell-off. Shares of AI chipmakers Nvidia and Broadcom plummeted by 17% on Monday, wiping out a combined $800 billion in market value. Nvidia has since recovered some of its losses. On Monday, DeepSeek announced a temporary restriction on user registrations, citing "large-scale malicious attacks" on its services, before later restoring normal operations. DeepSeek’s advancements have challenged the long-held belief that the U.S. was significantly ahead of China in AI development. Asked how R1 caught up to ChatGPT, AI and Crypto Czar David Sacks suggested that DeepSeek may have leveraged a technique known as "distillation" to train its model using OpenAI’s technology. “There’s a technique in AI called distillation, which you’re going to hear a lot about. It’s when one model learns from another model,” Sacks explained to Fox News. “Effectively, the student model asks the parent model millions of questions, mimicking the reasoning process and absorbing knowledge.” “They can essentially extract the knowledge out of the model,” he continued. “There’s substantial evidence that what DeepSeek did here was distill knowledge from OpenAI’s models.” “I don’t think OpenAI is too happy about this,” Sacks added. President Donald Trump has said that DeepSeek “should be a wake-up call” for U.S. tech companies. “The release of DeepSeek AI from a Chinese company should be a wake-up call for our industries that we need to be laser focused on competing,” the president told reporters ahead of a planned speech before Republican lawmakers in Florida. Read more:
1:09

U.S. Navy Bans DeepSeek Over 'Security Concerns' As 'Substantial' Evidence Emerges Chinese AI Ripped Off ChatGPT | ZeroHedge The U.S. Navy has instructed service members to avoid using the Chinese AI platform DeepSeek, citing "potential security and ethical concerns," according to CNBC. An email sent to "shipmates" in recent days, confirmed by CNBC on Tuesday, referenced the Navy's AI policy and emphasized the importance of refraining from using DeepSeek. The memo warned service members against using the platform "for any work-related tasks or personal use" and instructed them to "avoid downloading, installing, or using the DeepSeek model in any capacity." The warning follows the recent rise of DeepSeek’s R1 model, which has garnered significant attention worldwide, particularly within the U.S. business and technology sectors. The R1 model has demonstrated capabilities comparable to OpenAI’s models. In December, DeepSeek claimed it had successfully trained a large language model in just two months at a cost of $6 million—a figure disputed by technologists—despite U.S. restrictions on semiconductor chip exports to China. The R1, an open-source model, surged to the top of Apple’s app store rankings this week, triggering a market sell-off. Shares of AI chipmakers Nvidia and Broadcom plummeted by 17% on Monday, wiping out a combined $800 billion in market value. Nvidia has since recovered some of its losses. On Monday, DeepSeek announced a temporary restriction on user registrations, citing "large-scale malicious attacks" on its services, before later restoring normal operations. DeepSeek’s advancements have challenged the long-held belief that the U.S. was significantly ahead of China in AI development. Asked how R1 caught up to ChatGPT, AI and Crypto Czar David Sacks suggested that DeepSeek may have leveraged a technique known as "distillation" to train its model using OpenAI’s technology. “There’s a technique in AI called distillation, which you’re going to hear a lot about. It’s when one model learns from another model,” Sacks explained to Fox News. “Effectively, the student model asks the parent model millions of questions, mimicking the reasoning process and absorbing knowledge.” “They can essentially extract the knowledge out of the model,” he continued. “There’s substantial evidence that what DeepSeek did here was distill knowledge from OpenAI’s models.” “I don’t think OpenAI is too happy about this,” Sacks added. President Donald Trump has said that DeepSeek “should be a wake-up call” for U.S. tech companies. “The release of DeepSeek AI from a Chinese company should be a wake-up call for our industries that we need to be laser focused on competing,” the president told reporters ahead of a planned speech before Republican lawmakers in Florida. Read more:

Owen Gregorian

75,351 Aufrufe • vor 1 Jahr

** MEGA Parodius Scaling Effects Part 1 ** One of the big challenges with the Parodius Megadrive port is Stage 8's boss - The puffer-fish *Pooyan* with his full screen scaling effect. The goal is to be very close to the arcade (with extras on top ) so I thought lets tackle it head on to see how close we can get. I was also keen to jump into another scaling code rabit hole haha. Pyron pulled out all the stops and got me the source frames and reworked the BG tiles for this test - a big thankyou to him , Vector Orbitex is busy working on Stage 2 tracks so the team is working hard all round on this port. The MD has no sprite / background GFX scaling hardware , however the VDPs Vertical scroll can be updated per scanline to help vertical scaling on backgrounds, but there is a cpu cost to manage all the interupts so thats not free either. With the Horizontal scaling there is no help at all , apart from a semi-friendly packed pixel format for the cpu to work with, its not quite chunky format but better than planar format still for scaling. So its falls back to the 68k CPU to do all of the horizontal expansion which is the largest cpu cost. Basically drawing strips of either 1x, 2x, 3x or 4x wide columns at speed. So we are one week into this Boss's routine and you can see from the below video the horizontal scaling is implented ( vertical will be in the next update ) . We are scaling from 1x to 4x in the video below in 74 steps for testing . The column distributions are always a bit painfull to do - thankfully they are all worked out now. This is the third scaler I have built and the goal was with this one to make it really flexible for use in other projects also, sometimes when you optimise something to the last degree all the flexibility gets taken out of it. Currenty scaling at 12-25 FPS update here, I had some rules against some optimisations which I would use and some I wouldn't , thankfully we are a bit ahead of the Arcades animation frame rate here still and I may yet find optimisations that fit within the scope. We have vertical scaling and sprite spikes to add yet so Im hoping i can find a few more optimisations to offset things when they are implemented also. In a scale frame update we are processing close to 42000 pixels in ram before using DMA to send to VRAM . Using a 41x16 (656 tile scale buffer) - single buffered for now due to its size in VRAM. So thats nearly 21k in tiles ! I had to re-organise ram a bit to support a buffer of that size for the stage. The scaling function is written in 68k assembly , with a little C code handling the Vertical interupt code ( so the game logic can actually run & DMA updates etc ) . The DMA routines are in assembly also and customised for large chunk size ( big blocks of tiles ) which suits the scaler. I had some race conditions to sort out where the cpu was faster than DMA (sending tiles from RAM to VRAM ) and in some cases where it wasn't so it had to be balanced. We may be able to add more detail into the top and bottom of the background yet but its low priority for now until all the other bits are in !! #SGDK #SegaMegadrive #Genesis #Parodius
0:30

** MEGA Parodius Scaling Effects Part 1 ** One of the big challenges with the Parodius Megadrive port is Stage 8's boss - The puffer-fish *Pooyan* with his full screen scaling effect. The goal is to be very close to the arcade (with extras on top ) so I thought lets tackle it head on to see how close we can get. I was also keen to jump into another scaling code rabit hole haha. Pyron pulled out all the stops and got me the source frames and reworked the BG tiles for this test - a big thankyou to him , Vector Orbitex is busy working on Stage 2 tracks so the team is working hard all round on this port. The MD has no sprite / background GFX scaling hardware , however the VDPs Vertical scroll can be updated per scanline to help vertical scaling on backgrounds, but there is a cpu cost to manage all the interupts so thats not free either. With the Horizontal scaling there is no help at all , apart from a semi-friendly packed pixel format for the cpu to work with, its not quite chunky format but better than planar format still for scaling. So its falls back to the 68k CPU to do all of the horizontal expansion which is the largest cpu cost. Basically drawing strips of either 1x, 2x, 3x or 4x wide columns at speed. So we are one week into this Boss's routine and you can see from the below video the horizontal scaling is implented ( vertical will be in the next update ) . We are scaling from 1x to 4x in the video below in 74 steps for testing . The column distributions are always a bit painfull to do - thankfully they are all worked out now. This is the third scaler I have built and the goal was with this one to make it really flexible for use in other projects also, sometimes when you optimise something to the last degree all the flexibility gets taken out of it. Currenty scaling at 12-25 FPS update here, I had some rules against some optimisations which I would use and some I wouldn't , thankfully we are a bit ahead of the Arcades animation frame rate here still and I may yet find optimisations that fit within the scope. We have vertical scaling and sprite spikes to add yet so Im hoping i can find a few more optimisations to offset things when they are implemented also. In a scale frame update we are processing close to 42000 pixels in ram before using DMA to send to VRAM . Using a 41x16 (656 tile scale buffer) - single buffered for now due to its size in VRAM. So thats nearly 21k in tiles ! I had to re-organise ram a bit to support a buffer of that size for the stage. The scaling function is written in 68k assembly , with a little C code handling the Vertical interupt code ( so the game logic can actually run & DMA updates etc ) . The DMA routines are in assembly also and customised for large chunk size ( big blocks of tiles ) which suits the scaler. I had some race conditions to sort out where the cpu was faster than DMA (sending tiles from RAM to VRAM ) and in some cases where it wasn't so it had to be balanced. We may be able to add more detail into the top and bottom of the background yet but its low priority for now until all the other bits are in !! #SGDK #SegaMegadrive #Genesis #Parodius

Shannon Birt

25,545 Aufrufe • vor 7 Monaten

50% of my consulting work right now is helping companies use open-source models at scale. Everyone knows how to use an open-source LLM on their computers, but it's really hard to do this at scale for thousands of users. Here is how this plays out: 1. A team builds a prototype using DeepSeek. 2. Everything looks good. It works! 3. They follow an online guide to deploy the model online. 4. They ask 10 users to try the app. 5. Latency spikes everywhere. 6. The entire system halts. 7. They blame DeepSeek and try again using a new model. The problem is always with scaling inference, not the model. Here is one recommendation I give companies: Check out Nebius Token Factory if you don't want to ever think about deploying an open-source model again. This is a managed inference platform for deploying open-source LLMs at scale. This is not for prototypes or research experiments. This is for when you have a real application with real users. Three important notes about Token Factory: • You have complete control over how inference runs. • You have predictable tail latency (P99, not averages). • No surprise costs when you scale up. You can preplan your budget. Check it out here: Here are two codes you can use to get 100 hours of GPU usage on me: ymJLFa2ARYSKEdqb AdckcZaYjm7KqYY7 Thanks to the Nebius team for their continuous partnership.
2:40

50% of my consulting work right now is helping companies use open-source models at scale. Everyone knows how to use an open-source LLM on their computers, but it's really hard to do this at scale for thousands of users. Here is how this plays out: 1. A team builds a prototype using DeepSeek. 2. Everything looks good. It works! 3. They follow an online guide to deploy the model online. 4. They ask 10 users to try the app. 5. Latency spikes everywhere. 6. The entire system halts. 7. They blame DeepSeek and try again using a new model. The problem is always with scaling inference, not the model. Here is one recommendation I give companies: Check out Nebius Token Factory if you don't want to ever think about deploying an open-source model again. This is a managed inference platform for deploying open-source LLMs at scale. This is not for prototypes or research experiments. This is for when you have a real application with real users. Three important notes about Token Factory: • You have complete control over how inference runs. • You have predictable tail latency (P99, not averages). • No surprise costs when you scale up. You can preplan your budget. Check it out here: Here are two codes you can use to get 100 hours of GPU usage on me: ymJLFa2ARYSKEdqb AdckcZaYjm7KqYY7 Thanks to the Nebius team for their continuous partnership.

Santiago

48,443 Aufrufe • vor 6 Monaten

China just made Silicon Valley's entire AI industry look like a scam. The US government spent 3 years trying to stop China from building competitive AI. But this backfired HORRIBLY. Here's what happened: Yesterday, a Chinese startup called DeepSeek released a new AI model called V4. It matches the performance of OpenAI and Anthropic's best models. At 1/7th the price. And for the first time ever, it was built on Chinese chips. NOT American ones. That last part is the one that terrifies the west. For context: Since 2022, the US has banned the export of advanced AI chips to China. The entire strategy was built on the assumption that if China can't access Nvidia's best hardware, they can't build frontier AI. But DeepSeek just proved that assumption wrong. Their V4 model was trained and runs on Huawei's Ascend chips. Huawei spent months working directly with DeepSeek to make sure V4 runs across their entire line of AI processors. Jensen Huang even predicted this on a recent podcast: "The day that DeepSeek comes out on Huawei first, that is a horrible outcome for our nation." That day was yesterday. And the numbers are crazy: DeepSeek V4 costs $3.48 per million output tokens. OpenAI's latest model GPT-5.5 costs $30. Anthropic's Claude charges $25. Same ballpark performance. 7x cheaper. Uber's CTO just admitted they burned through their ENTIRE 2026 AI budget in 4 months using Anthropic's tools. If Uber had used DeepSeek instead, that same budget would have lasted 7 YEARS. 4 months vs 7 years. Same work getting done. But the pricing isn't even the big thing here. The real story is what DeepSeek did with their technical report: They published the benchmarks where they LOSE. Every AI company cherry-picks the tests where their model wins. DeepSeek ran the full comparison against GPT-5.4 and Google's Gemini, found they trail frontier models by 3 to 6 months, and printed it anyway. They literally don't care because the price gap makes the performance gap irrelevant for 90% of use cases. So the US export controls didn't slow China down. They ACCELERATED China's independence. Because Chinese developers were FORCED to train models with limited resources, they had to figure out how to make AI radically more efficient. That constraint became their competitive advantage. Every generation of DeepSeek has gotten dramatically cheaper to train. V4 continues the trend. Meanwhile US companies are going the OPPOSITE direction: OpenAI's GPT-5.5 Pro costs $180 per million output tokens. That's 51x more expensive than DeepSeek V4 for comparable work. The Commerce Secretary confirmed this week that ZERO Nvidia advanced chip shipments have actually gone through to China despite being approved in January. So China built frontier AI anyway. Without American chips. At a fraction of the cost. And the market response tells you everything: Chinese chipmaker SMIC surged 10%. Huahong Semiconductor jumped 15%. DeepSeek's Chinese AI competitors Zhipu AI and MiniMax dropped 9% because V4 is destroying them too. DeepSeek is making Silicon Valley's pricing model look like a scam. US tech companies spent $650 billion on AI infrastructure this year. DeepSeek just showed the world you can match their output for pennies. The export controls were supposed to be America's ace card. Instead they taught China how to win without American chips, at American prices nobody can compete with. Jensen Huang was right. This is a horrible outcome. But it's the outcome America built for itself.
1:06

China just made Silicon Valley's entire AI industry look like a scam. The US government spent 3 years trying to stop China from building competitive AI. But this backfired HORRIBLY. Here's what happened: Yesterday, a Chinese startup called DeepSeek released a new AI model called V4. It matches the performance of OpenAI and Anthropic's best models. At 1/7th the price. And for the first time ever, it was built on Chinese chips. NOT American ones. That last part is the one that terrifies the west. For context: Since 2022, the US has banned the export of advanced AI chips to China. The entire strategy was built on the assumption that if China can't access Nvidia's best hardware, they can't build frontier AI. But DeepSeek just proved that assumption wrong. Their V4 model was trained and runs on Huawei's Ascend chips. Huawei spent months working directly with DeepSeek to make sure V4 runs across their entire line of AI processors. Jensen Huang even predicted this on a recent podcast: "The day that DeepSeek comes out on Huawei first, that is a horrible outcome for our nation." That day was yesterday. And the numbers are crazy: DeepSeek V4 costs $3.48 per million output tokens. OpenAI's latest model GPT-5.5 costs $30. Anthropic's Claude charges $25. Same ballpark performance. 7x cheaper. Uber's CTO just admitted they burned through their ENTIRE 2026 AI budget in 4 months using Anthropic's tools. If Uber had used DeepSeek instead, that same budget would have lasted 7 YEARS. 4 months vs 7 years. Same work getting done. But the pricing isn't even the big thing here. The real story is what DeepSeek did with their technical report: They published the benchmarks where they LOSE. Every AI company cherry-picks the tests where their model wins. DeepSeek ran the full comparison against GPT-5.4 and Google's Gemini, found they trail frontier models by 3 to 6 months, and printed it anyway. They literally don't care because the price gap makes the performance gap irrelevant for 90% of use cases. So the US export controls didn't slow China down. They ACCELERATED China's independence. Because Chinese developers were FORCED to train models with limited resources, they had to figure out how to make AI radically more efficient. That constraint became their competitive advantage. Every generation of DeepSeek has gotten dramatically cheaper to train. V4 continues the trend. Meanwhile US companies are going the OPPOSITE direction: OpenAI's GPT-5.5 Pro costs $180 per million output tokens. That's 51x more expensive than DeepSeek V4 for comparable work. The Commerce Secretary confirmed this week that ZERO Nvidia advanced chip shipments have actually gone through to China despite being approved in January. So China built frontier AI anyway. Without American chips. At a fraction of the cost. And the market response tells you everything: Chinese chipmaker SMIC surged 10%. Huahong Semiconductor jumped 15%. DeepSeek's Chinese AI competitors Zhipu AI and MiniMax dropped 9% because V4 is destroying them too. DeepSeek is making Silicon Valley's pricing model look like a scam. US tech companies spent $650 billion on AI infrastructure this year. DeepSeek just showed the world you can match their output for pennies. The export controls were supposed to be America's ace card. Instead they taught China how to win without American chips, at American prices nobody can compete with. Jensen Huang was right. This is a horrible outcome. But it's the outcome America built for itself.

Ricardo

280,891 Aufrufe • vor 3 Monaten

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

131,019 Aufrufe • vor 1 Jahr

Cloud GPU training is a scam. A single M4 MacBook does 2.9 TFLOPS. Seven friends with MacBooks match an NVIDIA A100. Alexander Hayes just open-sourced a tool that makes this work over Wi-Fi. It's called AirTrain. Here's how it works: Traditional distributed training (DDP) syncs gradients after every single step. For a 124M parameter model, that's ~500MB exchanged per step. You need 50 GB/s of sustained bandwidth. Impossible over Wi-Fi. AirTrain uses the DiLoCo algorithm. Each Mac trains independently for 500 steps, then syncs only the difference. One sync per 500 steps instead of one per step. 500x less network communication. Wi-Fi actually works. The entire sync takes ~2 seconds. Here's what makes it wild: → Zero-config discovery. Devices find each other automatically via mDNS/Bonjour. Same protocol as AirDrop. → Fault tolerant. Nodes can join and leave mid-training without killing the run. → Checkpoint relay. Train for a few hours, export a checkpoint, hand it off to someone else to continue. Like a relay race for ML training. → Built on Apple's MLX framework. Native to M1/M2/M3/M4/M5 unified memory. No host-to-device copy overhead. → Local dashboard. Real-time loss curves, peer monitoring, throughput metrics in your browser. Here's the wildest part: An M4 Max with 128GB unified memory can train a 70B parameter model without offloading. An NVIDIA RTX 4090 has 24GB VRAM. Apple Silicon gets ~245-460 GFLOPS per watt. Training on MacBooks costs almost nothing in electricity compared to cloud GPUs. And there are hundreds of millions of Apple Silicon Macs in the world. The math: Traditional DDP: 1 sync per step = 50 GB/s required AirTrain (DiLoCo): 1 sync per 500 steps = 0.1 GB/s required Wi-Fi handles 0.1 GB/s. That's it. That's the breakthrough. They even built a community platform at with live session browsing, checkpoint sharing, and a contributor leaderboard. Training a 124M parameter GPT-2? Instead of renting cloud GPUs at $3/hr, pool three MacBooks in a coffee shop and train for free. MIT licensed. Built in Python. 1 contributor. Early stage but the idea is insane. 100% Open Source. (Link in the comments)
0:44

Cloud GPU training is a scam. A single M4 MacBook does 2.9 TFLOPS. Seven friends with MacBooks match an NVIDIA A100. Alexander Hayes just open-sourced a tool that makes this work over Wi-Fi. It's called AirTrain. Here's how it works: Traditional distributed training (DDP) syncs gradients after every single step. For a 124M parameter model, that's ~500MB exchanged per step. You need 50 GB/s of sustained bandwidth. Impossible over Wi-Fi. AirTrain uses the DiLoCo algorithm. Each Mac trains independently for 500 steps, then syncs only the difference. One sync per 500 steps instead of one per step. 500x less network communication. Wi-Fi actually works. The entire sync takes ~2 seconds. Here's what makes it wild: → Zero-config discovery. Devices find each other automatically via mDNS/Bonjour. Same protocol as AirDrop. → Fault tolerant. Nodes can join and leave mid-training without killing the run. → Checkpoint relay. Train for a few hours, export a checkpoint, hand it off to someone else to continue. Like a relay race for ML training. → Built on Apple's MLX framework. Native to M1/M2/M3/M4/M5 unified memory. No host-to-device copy overhead. → Local dashboard. Real-time loss curves, peer monitoring, throughput metrics in your browser. Here's the wildest part: An M4 Max with 128GB unified memory can train a 70B parameter model without offloading. An NVIDIA RTX 4090 has 24GB VRAM. Apple Silicon gets ~245-460 GFLOPS per watt. Training on MacBooks costs almost nothing in electricity compared to cloud GPUs. And there are hundreds of millions of Apple Silicon Macs in the world. The math: Traditional DDP: 1 sync per step = 50 GB/s required AirTrain (DiLoCo): 1 sync per 500 steps = 0.1 GB/s required Wi-Fi handles 0.1 GB/s. That's it. That's the breakthrough. They even built a community platform at with live session browsing, checkpoint sharing, and a contributor leaderboard. Training a 124M parameter GPT-2? Instead of renting cloud GPUs at $3/hr, pool three MacBooks in a coffee shop and train for free. MIT licensed. Built in Python. 1 contributor. Early stage but the idea is insane. 100% Open Source. (Link in the comments)

Guri Singh

160,201 Aufrufe • vor 4 Monaten

I trained a 100 million parameter DeepSeek V3 LLM from scratch Here's what you need to know. Previously I trained traditional GPT-2 architecture which has become obsolete with recent LLM advancements. Most recent models like Llama, Mistral, DeepSeek, and GPT-4 use latest architectures. ✦ Model Configuration of my SLM DeepSeek V3 - Parameters: 109,032,032 - Embedding Dimension: 512 - Layers: 8 - Heads: 8 - Experts (MoE): 8 - Experts per token: 2 ✦ DeepSeek brings major architectural changes: - Multi Head Latent Attention - Mixture of Experts - RMS Norm - Multi Token Prediction ✦ Dataset Challenge - TinyStories is great for learning SLMs. I trained GPT-2 on it previously with good results. - But I needed a more challenging dataset. - If I use TinyStories again on DeepSeek, how would I know MHLA, MoE or MTP works better than old architecture? - The old architecture can handle it, so new DeepSeek would too without utilizing latest advancements. That's why I moved to FineWeb-Edu dataset Thanks Yuvraj Singh for the suggestion for this dataset ✦ Training Journey - Rented A100 PCIe GPU and trained the model. - Did test runs. During final run, model was 65% trained but stopped due to glitch after 4 hours. - Fixed all edge cases and ran training again with increased config parameters. - Final training: 7 hours, 20,000 epochs 𝐓𝐨𝐭𝐚𝐥 𝐆𝐏𝐔 𝐜𝐨𝐬𝐭: $17 - $9.53 for main 7-hour run - $7.42 for experiments and demos ✦ Reflection Amazing long project that taught me latest architectural advancements. I'll reimplement and revisit after a few weeks because there's too much complexity, mostly in Multi Head Latent Attention part. Need to make concepts stronger. Code Final trained Model Dataset Resources Huge shoutout to Raj Dandekar again for creating one of the most detailed video series about DeepSeek - this was my primary resource for the implementation. Playlist Blogs by Maarten Grootendorst These are excellent visual blogs to understand MoE in detail. Thanks Maarten for your amazing contributions to the community through your books and blogs Blogs on MoE Implemention of MoE from scratch by @aviTwit3 One of the most detailed blogs on implementing Mixture of Experts. Thanks Avinash for this blog - it helped me understand Mixture of Experts much better. If you're someone in the 𝐌𝐋 & 𝐋𝐋𝐌 space, would love to 𝐜𝐨𝐧𝐧𝐞𝐜𝐭 and discuss this field in general, so give a follow up for that.
6:00

I trained a 100 million parameter DeepSeek V3 LLM from scratch Here's what you need to know. Previously I trained traditional GPT-2 architecture which has become obsolete with recent LLM advancements. Most recent models like Llama, Mistral, DeepSeek, and GPT-4 use latest architectures. ✦ Model Configuration of my SLM DeepSeek V3 - Parameters: 109,032,032 - Embedding Dimension: 512 - Layers: 8 - Heads: 8 - Experts (MoE): 8 - Experts per token: 2 ✦ DeepSeek brings major architectural changes: - Multi Head Latent Attention - Mixture of Experts - RMS Norm - Multi Token Prediction ✦ Dataset Challenge - TinyStories is great for learning SLMs. I trained GPT-2 on it previously with good results. - But I needed a more challenging dataset. - If I use TinyStories again on DeepSeek, how would I know MHLA, MoE or MTP works better than old architecture? - The old architecture can handle it, so new DeepSeek would too without utilizing latest advancements. That's why I moved to FineWeb-Edu dataset Thanks Yuvraj Singh for the suggestion for this dataset ✦ Training Journey - Rented A100 PCIe GPU and trained the model. - Did test runs. During final run, model was 65% trained but stopped due to glitch after 4 hours. - Fixed all edge cases and ran training again with increased config parameters. - Final training: 7 hours, 20,000 epochs 𝐓𝐨𝐭𝐚𝐥 𝐆𝐏𝐔 𝐜𝐨𝐬𝐭: $17 - $9.53 for main 7-hour run - $7.42 for experiments and demos ✦ Reflection Amazing long project that taught me latest architectural advancements. I'll reimplement and revisit after a few weeks because there's too much complexity, mostly in Multi Head Latent Attention part. Need to make concepts stronger. Code Final trained Model Dataset Resources Huge shoutout to Raj Dandekar again for creating one of the most detailed video series about DeepSeek - this was my primary resource for the implementation. Playlist Blogs by Maarten Grootendorst These are excellent visual blogs to understand MoE in detail. Thanks Maarten for your amazing contributions to the community through your books and blogs Blogs on MoE Implemention of MoE from scratch by @aviTwit3 One of the most detailed blogs on implementing Mixture of Experts. Thanks Avinash for this blog - it helped me understand Mixture of Experts much better. If you're someone in the 𝐌𝐋 & 𝐋𝐋𝐌 space, would love to 𝐜𝐨𝐧𝐧𝐞𝐜𝐭 and discuss this field in general, so give a follow up for that.

Mayank Pratap Singh

48,100 Aufrufe • vor 1 Jahr

Distilled recap of the back-and-forth with Jensen on export controls: Dwarkesh: Wouldn’t selling Nvidia chips to China enable them to train models like Claude Mythos with cyber offensive capabilities that would be threats to American companies and national security? Jensen: First of all, Mythos was trained on fairly mundane capacity and a fairly mundane amount of it by an extraordinary company. The amount of capacity and the type of compute it was trained on is abundantly available in China. Dwarkesh: With that, could they eventually train a model like Mythos? Yes. But the question is, because we have more FLOPs, American labs are able to get to this level of capabilities first. Furthermore, even if they trained a model like this, the ability to deploy it at scale matters. If you had a cyber hacker, it's much more dangerous if they have a million of them versus a thousand of them. Jensen: Your premise is just wrong. The fact of the matter is their AI development is going just fine. The best AI researchers in the world, because they are limited in compute, also come up with extremely smart algorithms. DeepSeek is not an inconsequential advance. The day that DeepSeek comes out on Huawei first, that is a horrible outcome for our nation. Dwarkesh: Currently, you can have a model like DeepSeek that can run on any accelerator if it's open source. Why would that stop being the case in the future? Jensen: Suppose it optimizes for Huawei. Suppose it optimizes for their architecture. It would put others at a disadvantage. As AI diffuses out into the rest of the world, their standards and their tech stack will become superior to ours because their models are open. Dwarkesh: Tesla sold extremely good electric vehicles to China for a long time. iPhones are sold in China. They didn't cause some lock-in. China will still make their version of EVs, and they're dominating, or smartphones, they're dominating. Jensen: We are not a car. The fact that I can buy this car brand one day and use another car brand another day is easy. Computing is not like that. There's a reason why x86 still exists. There's a reason why Arm is so sticky. These ecosystems are hard to replace. Dwarkesh: It's just hard to imagine that there's a long-term lock-in to the Chinese ecosystem, even if they have this slightly better open-source model for a while. American labs port across accelerators constantly. Anthropic's models are run on GPUs, they're run on Trainium, they're run on TPUs. There are so many things you can do, from distilling to a model that's well fit for your chips. Jensen: China is the largest contributor to open source software in the world. China's the largest contributor to open models in the world. Today it's built on the American tech stack, Nvidia’s. Fact. All five layers of the tech stack for AI are important. The United States ought to go win all five of them. in a few years time, I'm making you the prediction that when we want American technology to be diffused around the world—out to India, out to the Middle East, out to Africa, out to Southeast Asia—on that day, I will tell you exactly about today's conversation, about how your policy ... caused the United States to concede the second largest market in the world for no good reason at all.
3:53

Distilled recap of the back-and-forth with Jensen on export controls: Dwarkesh: Wouldn’t selling Nvidia chips to China enable them to train models like Claude Mythos with cyber offensive capabilities that would be threats to American companies and national security? Jensen: First of all, Mythos was trained on fairly mundane capacity and a fairly mundane amount of it by an extraordinary company. The amount of capacity and the type of compute it was trained on is abundantly available in China. Dwarkesh: With that, could they eventually train a model like Mythos? Yes. But the question is, because we have more FLOPs, American labs are able to get to this level of capabilities first. Furthermore, even if they trained a model like this, the ability to deploy it at scale matters. If you had a cyber hacker, it's much more dangerous if they have a million of them versus a thousand of them. Jensen: Your premise is just wrong. The fact of the matter is their AI development is going just fine. The best AI researchers in the world, because they are limited in compute, also come up with extremely smart algorithms. DeepSeek is not an inconsequential advance. The day that DeepSeek comes out on Huawei first, that is a horrible outcome for our nation. Dwarkesh: Currently, you can have a model like DeepSeek that can run on any accelerator if it's open source. Why would that stop being the case in the future? Jensen: Suppose it optimizes for Huawei. Suppose it optimizes for their architecture. It would put others at a disadvantage. As AI diffuses out into the rest of the world, their standards and their tech stack will become superior to ours because their models are open. Dwarkesh: Tesla sold extremely good electric vehicles to China for a long time. iPhones are sold in China. They didn't cause some lock-in. China will still make their version of EVs, and they're dominating, or smartphones, they're dominating. Jensen: We are not a car. The fact that I can buy this car brand one day and use another car brand another day is easy. Computing is not like that. There's a reason why x86 still exists. There's a reason why Arm is so sticky. These ecosystems are hard to replace. Dwarkesh: It's just hard to imagine that there's a long-term lock-in to the Chinese ecosystem, even if they have this slightly better open-source model for a while. American labs port across accelerators constantly. Anthropic's models are run on GPUs, they're run on Trainium, they're run on TPUs. There are so many things you can do, from distilling to a model that's well fit for your chips. Jensen: China is the largest contributor to open source software in the world. China's the largest contributor to open models in the world. Today it's built on the American tech stack, Nvidia’s. Fact. All five layers of the tech stack for AI are important. The United States ought to go win all five of them. in a few years time, I'm making you the prediction that when we want American technology to be diffused around the world—out to India, out to the Middle East, out to Africa, out to Southeast Asia—on that day, I will tell you exactly about today's conversation, about how your policy ... caused the United States to concede the second largest market in the world for no good reason at all.

Dwarkesh Patel

1,252,105 Aufrufe • vor 4 Monaten

To our Wicked fans, As 2024 is coming to a close, we want to take the time to thank you for joining us on our journey with No Rest for the Wicked. This has been a huge year for all of us at Moon Studios. Launching Wicked in Early Access in April was quite the beginning. We've been listening to your feedback very closely, rapidly rolling out hotfixes, balancing, optimizing and adding quality-of-life improvements to the game. With The Crucible Update we added a whole new rogue-lite mode, setting the stage for more replayability out of the existing Early Access content. Since then we've been hard at work on “The Breach” and our next big Content and Feature updates. We know we’ve been quiet - a lot has been happening for us at Moon Studios during this year, and we will finally be able to share all about it very soon! We will be kicking off 2025 with a new announcement in January and you can also expect better and more frequent communication on our Official Channels throughout the year. Thank you for sticking with us and helping shape No Rest for the Wicked into something truly special. Your passion, feedback and support continue to further fuel our passion since the very first announcement at the The Game Awards. We couldn’t be more proud of our team at Moon Studios and everything we have in store for 2025. Here’s to an even bigger year for No Rest for the Wicked, Moon Studios and all of you! Happy holidays! 🎉🎊🎁 The Moon Studios Team ❤️
1:02

To our Wicked fans, As 2024 is coming to a close, we want to take the time to thank you for joining us on our journey with No Rest for the Wicked. This has been a huge year for all of us at Moon Studios. Launching Wicked in Early Access in April was quite the beginning. We've been listening to your feedback very closely, rapidly rolling out hotfixes, balancing, optimizing and adding quality-of-life improvements to the game. With The Crucible Update we added a whole new rogue-lite mode, setting the stage for more replayability out of the existing Early Access content. Since then we've been hard at work on “The Breach” and our next big Content and Feature updates. We know we’ve been quiet - a lot has been happening for us at Moon Studios during this year, and we will finally be able to share all about it very soon! We will be kicking off 2025 with a new announcement in January and you can also expect better and more frequent communication on our Official Channels throughout the year. Thank you for sticking with us and helping shape No Rest for the Wicked into something truly special. Your passion, feedback and support continue to further fuel our passion since the very first announcement at the The Game Awards. We couldn’t be more proud of our team at Moon Studios and everything we have in store for 2025. Here’s to an even bigger year for No Rest for the Wicked, Moon Studios and all of you! Happy holidays! 🎉🎊🎁 The Moon Studios Team ❤️

No Rest for the Wicked

73,362 Aufrufe • vor 1 Jahr