正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

245 tokens/sec on DeepSeek-V4-Flash and it’s only using 2 RTX 6000 Blackwell GPUs. tinycorp dropped a 2-GPU tinybox edition in honor of DeepSeek fully ready to expand to 4 GPUs later. The GPU middle class just got a serious upgrade.

Md Ismail Šojal 🕷️

50,117 subscribers

16,148 次观看 • 8 天前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

⚡️ We just added support for Nvidia G4 (RTX Pro 6000 Blackwell Server Edition) GPUs in Colaboratory! 🔥 With a peak rating of 960 BF16 TFLOPs (~50% more than the A100-80G) and 96 GB of VRAM (20% more than the A100-80G), it’s the most efficient high-performance GPU we’ve ever released. ⏱️Time to make GPUs go brrrr 🏎️

⚡️ We just added support for Nvidia G4 (RTX Pro 6000 Blackwell Server Edition) GPUs in Colaboratory! 🔥 With a peak rating of 960 BF16 TFLOPs (~50% more than the A100-80G) and 96 GB of VRAM (20% more than the A100-80G), it’s the most efficient high-performance GPU we’ve ever released. ⏱️Time to make GPUs go brrrr 🏎️

Colaboratory

62,384 次观看 • 5 个月前

🚨 NVIDIA just flipped the entire AI game… and this is NOT about gaming. DeepSeek-V4-Pro is now live on their build platform. 1.6 TRILLION parameters. Yes… the largest open-source model on the planet right now. And here’s the crazy part: They’re letting you run it FREE On Blackwell GPUs in the cloud. This is the same level of hardware companies like Google, Meta, and Microsoft fight billions to access. Now it’s just… available. No waitlist. No insane setup. Just raw power. We’re watching the shift happen in real time: → From closed AI → open domination → From GPU scarcity → free access → From Big Tech control → builders winning This isn’t an update. It’s a warning shot. Who’s already testing this? Link👇

🚨 NVIDIA just flipped the entire AI game… and this is NOT about gaming. DeepSeek-V4-Pro is now live on their build platform. 1.6 TRILLION parameters. Yes… the largest open-source model on the planet right now. And here’s the crazy part: They’re letting you run it FREE On Blackwell GPUs in the cloud. This is the same level of hardware companies like Google, Meta, and Microsoft fight billions to access. Now it’s just… available. No waitlist. No insane setup. Just raw power. We’re watching the shift happen in real time: → From closed AI → open domination → From GPU scarcity → free access → From Big Tech control → builders winning This isn’t an update. It’s a warning shot. Who’s already testing this? Link👇

divyansh tiwari

29,941 次观看 • 3 个月前

Codex can now run Deepseek-v4- flash! There's a catch though. Deepseek's official setup switches your entire codex over to them, so your GPT models stop showing up at all. This is exactly what Codex Router is for. It adds models to the list instead of replacing them, so sol, grok, kimi and deepseek all sit in the same picker and i just grab whichever one suits the job. Deepseek v4-flash is $0.28 per million output tokens. opus 4.8 is $25. same picker, 89x apart. Links in the comment. setup's in the video 👇

Codex can now run Deepseek-v4- flash! There's a catch though. Deepseek's official setup switches your entire codex over to them, so your GPT models stop showing up at all. This is exactly what Codex Router is for. It adds models to the list instead of replacing them, so sol, grok, kimi and deepseek all sit in the same picker and i just grab whichever one suits the job. Deepseek v4-flash is $0.28 per million output tokens. opus 4.8 is $25. same picker, 89x apart. Links in the comment. setup's in the video 👇

Ziwen

145,018 次观看 • 9 天前

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Akshay 🚀

126,122 次观看 • 9 个月前

DeepSeek V4 Flash being ranked almost equal to Opus 4.8 is actually insane to me. Against Kimi K3, Opus 5, GPT-5.6 Sol, and Qwen 3.8 Max Preview, you give up a LOT by choosing DeepSeek. This is not a 2-point difference in practice. I’ll post the Opus 4.8 comparison next because you need to see this.

DeepSeek V4 Flash being ranked almost equal to Opus 4.8 is actually insane to me. Against Kimi K3, Opus 5, GPT-5.6 Sol, and Qwen 3.8 Max Preview, you give up a LOT by choosing DeepSeek. This is not a 2-point difference in practice. I’ll post the Opus 4.8 comparison next because you need to see this.

OmedTheVibeCoder

26,990 次观看 • 8 天前

We're thrilled to unveil the new Provider Leaderboard on the Spheron GPU Marketplace! 🎮 Now, GPU providers can track their rewards and see how they stack up against others. 🏆 It’s time to up your game, add more compute power, and compete for the top spot! Get ready to level up your earnings and experience the thrill of healthy competition in the world of decentralized compute! 💪 Idle GPUs? Fill this out:-

We're thrilled to unveil the new Provider Leaderboard on the Spheron GPU Marketplace! 🎮 Now, GPU providers can track their rewards and see how they stack up against others. 🏆 It’s time to up your game, add more compute power, and compete for the top spot! Get ready to level up your earnings and experience the thrill of healthy competition in the world of decentralized compute! 💪 Idle GPUs? Fill this out:-

Spheron Network

106,041 次观看 • 2 年前

$IREN "we haven't disclosed the specific amount of GPUs" 1. 🤮 reminds me of $NBIS 2. Setting a terrible precedent here for future deals 3. Making it purposely difficult, to not let analysts properly value your 2027 revenue 4. Increasing the polarized view on IREN by the market However: "approximately 60MW of air-cooled Blackwells" 1. You typically don't talk about gross capacity in a deployment like this 2. If it would be gross capacity, the GPU hour rate at IT level would be crazy high (at PUE 1.2, $680m / 50 = 13.6m/MW) 3. At 60MW IT load, and ~14kW draw at DGX server level, we can get to ~4,286 DGX systems with 8 GPUs per. 4. Based on this we can conclude that 60MW of IT load can run approximately 34k DGX B300. 5. 34k DGX B300 at $680m/yr, would represent a GPU hour price of $2.28 Now this is the problem with not disclosing your GPU quantity. You purposely make your business model look bad, because by approach, you get to a GPU hour price that would imply a payback period of 4 years, where only the last year of the contract is 100% margin. But of course, we can also take "the glass is half full" approach. IREN has ordered 50K B300s from Dell. They have 2 purchase orders for this, 1 between Dell Canada and IE CA Leasing Ltd for 4 phases, and 1 between Dell USA and IE US Hardware 1 Inc (amended from IE US Hardware 4 Inc on April 27, 2026). The order for Canada is divided in 4 phases, and are going to Mackenzie for 80MW of gross capacity, which happens to be 4 buildings of 20MW. The order for Childress is divided in 2 phases, and are going to DC35 and DC36, (as depicted in the earnings presentation) and those are 50MW gross. The purchase price of the order for Childress was $1.2B, and for Canada it was $2.3B If we go with 50,000 B300s for a total of $3.5B then $1.2 would represent 34.285% of the 50,000 GPUs, or 17,140 B300s rounded down. For this calculation I will consider that $IREN will deploy 17,140 GPUs in 50MW gross capacity in DC35 and DC36 of block 3 in Childress.. That would imply at 1.2 PUE, IREN can run 17,140 B300s in 41.67MW IT load. Now by that ratio, they can run 24,680 GPUs in 60MW IT load — a massive difference with 34k units through the Nvidia DGX reference calculation. If common sense is applied, you can still get to 2 completely different outcomes, that show a difference of more than 9k GPUs. The GPU hour rate at 24.68k GPUs would be $3.145 per B300, as MASSIVE difference from the earlier calculated $2.28. Sure, the DGX system may be a factor here. And I'm sure that the reality is somewhere in the middle. But I personally hate this as an investor, to be unable to calculate profitability on unit economic basis. After all, contracts are signed on a $/GPU hour basis. Why hide this from your investors? Not being able to calculate payback periods, unable to calculate ROIC. And most importantly, we cannot properly assess the $NVDA deal on a contract basis. I really hope the payback period of this contract is not 4 years. I want the glass to be half full, but by starting to censor the purchases, IREN is taking a step in the wrong direction. Not a fan of this.

$IREN "we haven't disclosed the specific amount of GPUs" 1. 🤮 reminds me of $NBIS 2. Setting a terrible precedent here for future deals 3. Making it purposely difficult, to not let analysts properly value your 2027 revenue 4. Increasing the polarized view on IREN by the market However: "approximately 60MW of air-cooled Blackwells" 1. You typically don't talk about gross capacity in a deployment like this 2. If it would be gross capacity, the GPU hour rate at IT level would be crazy high (at PUE 1.2, $680m / 50 = 13.6m/MW) 3. At 60MW IT load, and ~14kW draw at DGX server level, we can get to ~4,286 DGX systems with 8 GPUs per. 4. Based on this we can conclude that 60MW of IT load can run approximately 34k DGX B300. 5. 34k DGX B300 at $680m/yr, would represent a GPU hour price of $2.28 Now this is the problem with not disclosing your GPU quantity. You purposely make your business model look bad, because by approach, you get to a GPU hour price that would imply a payback period of 4 years, where only the last year of the contract is 100% margin. But of course, we can also take "the glass is half full" approach. IREN has ordered 50K B300s from Dell. They have 2 purchase orders for this, 1 between Dell Canada and IE CA Leasing Ltd for 4 phases, and 1 between Dell USA and IE US Hardware 1 Inc (amended from IE US Hardware 4 Inc on April 27, 2026). The order for Canada is divided in 4 phases, and are going to Mackenzie for 80MW of gross capacity, which happens to be 4 buildings of 20MW. The order for Childress is divided in 2 phases, and are going to DC35 and DC36, (as depicted in the earnings presentation) and those are 50MW gross. The purchase price of the order for Childress was $1.2B, and for Canada it was $2.3B If we go with 50,000 B300s for a total of $3.5B then $1.2 would represent 34.285% of the 50,000 GPUs, or 17,140 B300s rounded down. For this calculation I will consider that $IREN will deploy 17,140 GPUs in 50MW gross capacity in DC35 and DC36 of block 3 in Childress.. That would imply at 1.2 PUE, IREN can run 17,140 B300s in 41.67MW IT load. Now by that ratio, they can run 24,680 GPUs in 60MW IT load — a massive difference with 34k units through the Nvidia DGX reference calculation. If common sense is applied, you can still get to 2 completely different outcomes, that show a difference of more than 9k GPUs. The GPU hour rate at 24.68k GPUs would be $3.145 per B300, as MASSIVE difference from the earlier calculated $2.28. Sure, the DGX system may be a factor here. And I'm sure that the reality is somewhere in the middle. But I personally hate this as an investor, to be unable to calculate profitability on unit economic basis. After all, contracts are signed on a $/GPU hour basis. Why hide this from your investors? Not being able to calculate payback periods, unable to calculate ROIC. And most importantly, we cannot properly assess the $NVDA deal on a contract basis. I really hope the payback period of this contract is not 4 years. I want the glass to be half full, but by starting to censor the purchases, IREN is taking a step in the wrong direction. Not a fan of this.

Frans Bakker

148,167 次观看 • 3 个月前

> 8 GPUs in one server rig > dude went homeless to build it > electrical bill costs more than rent now > while everyone else pays $400/month to openai > a 2 GPU desktop kills the api bill forever > rtx 4080 super + rtx 5060 ti = 32gb vram > runs qwen 3.6 with 100k context locally > no rate limits, no api keys, no data leaving the room > agents loop 400 times for free > claude opus still wins on hard reasoning > but local handles 90% of daily work > $1,200 setup pays itself off in 4 months > bookmark this and read the article below

> 8 GPUs in one server rig > dude went homeless to build it > electrical bill costs more than rent now > while everyone else pays $400/month to openai > a 2 GPU desktop kills the api bill forever > rtx 4080 super + rtx 5060 ti = 32gb vram > runs qwen 3.6 with 100k context locally > no rate limits, no api keys, no data leaving the room > agents loop 400 times for free > claude opus still wins on hard reasoning > but local handles 90% of daily work > $1,200 setup pays itself off in 4 months > bookmark this and read the article below

starmex

167,058 次观看 • 2 个月前

DeepSeek R1 is *the* best model available right now. It's at the level of o1, but you can use it for free, and it's much faster. A huge leap forward that nobody saw coming. No wonder so many people are throwing tantrums online trying to discredit the Chinese students who built this. You can use DeepSeek in Visual Studio Code right now: 1. Install the Qodo Gen AI extension 2. Select DeepSeek R1 from their list of models The Qodo team is hosting DeepSeek on their servers, so none of your data will go to China. I've been building a Tetris game using DeepSeek, and this is the most impressive model I've seen so far.

DeepSeek R1 is the best model available right now. It's at the level of o1, but you can use it for free, and it's much faster. A huge leap forward that nobody saw coming. No wonder so many people are throwing tantrums online trying to discredit the Chinese students who built this. You can use DeepSeek in Visual Studio Code right now: 1. Install the Qodo Gen AI extension 2. Select DeepSeek R1 from their list of models The Qodo team is hosting DeepSeek on their servers, so none of your data will go to China. I've been building a Tetris game using DeepSeek, and this is the most impressive model I've seen so far.

Santiago

1,224,151 次观看 • 1 年前

Loading DeFAI infrastructure ▓▓▓▓▓▓▓▓▓░ io.net, a decentralized GPU compute network has expanded to power the future of DeFAI for Injective builders. The integration aims to support the growing trend of developers building AI Agents, DeFAI, and gaming initiatives in web3. io.net provides Web3 builders with tools to train, fine-tune, and deploy ML models using decentralized resources. The collaboration combines Injective's iAgent framework with io.net’s extensive GPU network, setting the stage for more accessible and innovative development across use cases that require compute resources. Key features include: ✅ Access to over 10,000 cluster-ready GPUs and CPUs, ✅ AI-driven blockchain activities using Injective's iAgent SDK, and ✅ Potential for new on-chain financial products leveraging GPU pricing and data feeds. 👀 io.net’s DePIN network deploys on-demand, decentralized GPU resources globally, designed for low latency and high-throughput processing. Benefits include reduced barriers for AI/ML projects, enhanced integration of AI with on-chain activities, and democratized access to high-performance computing resources. Together, this marks a significant milestone in decentralized AI infrastructure, addressing key challenges and paving the way for unprecedented innovation in AI development and on-chain finance.

Loading DeFAI infrastructure ▓▓▓▓▓▓▓▓▓░ io.net, a decentralized GPU compute network has expanded to power the future of DeFAI for Injective builders. The integration aims to support the growing trend of developers building AI Agents, DeFAI, and gaming initiatives in web3. io.net provides Web3 builders with tools to train, fine-tune, and deploy ML models using decentralized resources. The collaboration combines Injective's iAgent framework with io.net’s extensive GPU network, setting the stage for more accessible and innovative development across use cases that require compute resources. Key features include: ✅ Access to over 10,000 cluster-ready GPUs and CPUs, ✅ AI-driven blockchain activities using Injective's iAgent SDK, and ✅ Potential for new on-chain financial products leveraging GPU pricing and data feeds. 👀 io.net’s DePIN network deploys on-demand, decentralized GPU resources globally, designed for low latency and high-throughput processing. Benefits include reduced barriers for AI/ML projects, enhanced integration of AI with on-chain activities, and democratized access to high-performance computing resources. Together, this marks a significant milestone in decentralized AI infrastructure, addressing key challenges and paving the way for unprecedented innovation in AI development and on-chain finance.

Injective 🥷

131,743 次观看 • 1 年前

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies

Alok

292,770 次观看 • 2 个月前

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

max fu

29,197 次观看 • 22 天前

ONE OPERATOR STACKED 300 GPUS ACROSS TWO APARTMENTS IN THE SAME BUILDING AND RUNS A $48K/MONTH AI INFERENCE FARM ON VAST AI FROM HIS LIVING ROOM 00:17 he walks past stacks of GPU boxes, "and probably another 100 GPU boxes in the second apartment, let me know in the comments if you want to see them" he rents 2 units in the same building, one as his living space with 200 GPUs in the bedroom and hallway, the second is dedicated and climate controlled just for the other 100 cards a 300 RTX 4090 setup pulls 135 kilowatts fully loaded, his power bill runs $9,800 a month at $0.10 per kwh, on vast ai the same fleet clears $48,000 in gross monthly rental income he never built this in a warehouse because residential electricity in his city is cheaper than commercial under 150 kw, the split apartment trick keeps him under that ceiling while doubling his rack space the same hardware would have cleared maybe $9,000 a month mining ethereum classic in 2022, vast ai pays 5 times that for AI inference because nobody can ship enough H100s to meet startup demand bookmark this and read the article below

ONE OPERATOR STACKED 300 GPUS ACROSS TWO APARTMENTS IN THE SAME BUILDING AND RUNS A $48K/MONTH AI INFERENCE FARM ON VAST AI FROM HIS LIVING ROOM 00:17 he walks past stacks of GPU boxes, "and probably another 100 GPU boxes in the second apartment, let me know in the comments if you want to see them" he rents 2 units in the same building, one as his living space with 200 GPUs in the bedroom and hallway, the second is dedicated and climate controlled just for the other 100 cards a 300 RTX 4090 setup pulls 135 kilowatts fully loaded, his power bill runs $9,800 a month at $0.10 per kwh, on vast ai the same fleet clears $48,000 in gross monthly rental income he never built this in a warehouse because residential electricity in his city is cheaper than commercial under 150 kw, the split apartment trick keeps him under that ceiling while doubling his rack space the same hardware would have cleared maybe $9,000 a month mining ethereum classic in 2022, vast ai pays 5 times that for AI inference because nobody can ship enough H100s to meet startup demand bookmark this and read the article below

starmex

11,545 次观看 • 1 个月前

my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 31b Q4) with only 8 GB VRAM last week I ran Gemma 4 26B A4B a mixture of experts model on my RTX 4060 and hit 25–28 tokens/sec using llama.cpp's new MTP support. smooth. snappy. but MoE has a secret: it only activates 4B parameters per token despite having 26B total. that's why it flies. so the real question started haunting me. what if I throw a full, no tricks, every parameter fires on every token, 31B DENSE model at the same machine? # Hardware: GPU: NVIDIA RTX 4060, 8 GB VRAM RAM: 16 GB CPU: Intel Core i7 H Laptop. Gaming. Modest. The model: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (model's unsloth huggingface link in the comments) This is Google DeepMind's flagship dense model in the Gemma 4 family that can run on single consumer GPU. It packs a hybrid attention architecture, supports up to 256K context natively, and is QAT (Quantization Aware Training) optimized, meaning it retains far more quality than standard post training quants at the same bit depth. This is NOT the MoE. This is 31 BILLION dense parameters, every single one of them loaded. # the flags I used: -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 8 --spec-draft-p-min 0.6 -c 6000 -v Multi Token Prediction (MTP) is still active here. Separate draft GGUF required, same as the 26B setup. # Results: → Decode: ~3 tokens/sec → Prefill: ~2 tokens/sec → Context: 6000 tokens → Hardware crying quietly in the corner: yes so is 3 tps actually usable? For real time back and forth chat? Not ideal. You're not having a fluid conversation at 3 tps. but slow ≠ useless. And this is where it gets genuinely interesting. think about how senior devs actually work in a real team. But when something is architectural, deeply complex, or needs serious reasoning? they walk down the hall and escalate to the senior. That's exactly the local AI agent architecture this unlocks: → Fast orchestrator model (Gemma 4 26B MoE at 25+ tps) handles routing, simple queries, tool calls, memory. The junior dev. → Gemma 4 31B dense is the senior, called only when the fast model genuinely hits a wall. Hard multi step reasoning. Complex code generation. Deep architectural decisions. The agentic loop stays fast. Only the hard hops touch the 31B. That's a legitimate production grade local AI architecture on a budget hardware. (requires 2 8gb gpus) other workflows where 3 tps is completely fine: - overnight batch jobs. summarize documents, extract structured data, review code. Fire it off. Sleep. wake up to results. - One shot deep reasoning - Silent code audit loops, you write and test, the 31B reviews diffs and flags issues in the background between your sprints - Any workflow where output quality > output speed A few weeks ago, nobody was running a 30B+ dense model on a single consumer GPU with 8 GB VRAM. At all. Now we're doing it on an Intel i7-H gaming laptop with a NVIDIA RTX 4060, thanks to llama.cpp + QAT quants + MTP speculative drafting. Google DeepMind said the Gemma 4 31B targets "consumer GPUs and workstations." They were not exaggerating. The hardware bar to run serious frontier class models locally keeps dropping. the tools are here. the models are here. you just have to be willing to abuse your laptop a little. what workflows would you actually run on a local 3 tps 31B dense model? genuinely curious. drop it below.

my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 31b Q4) with only 8 GB VRAM last week I ran Gemma 4 26B A4B a mixture of experts model on my RTX 4060 and hit 25–28 tokens/sec using llama.cpp's new MTP support. smooth. snappy. but MoE has a secret: it only activates 4B parameters per token despite having 26B total. that's why it flies. so the real question started haunting me. what if I throw a full, no tricks, every parameter fires on every token, 31B DENSE model at the same machine? # Hardware: GPU: NVIDIA RTX 4060, 8 GB VRAM RAM: 16 GB CPU: Intel Core i7 H Laptop. Gaming. Modest. The model: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (model's unsloth huggingface link in the comments) This is Google DeepMind's flagship dense model in the Gemma 4 family that can run on single consumer GPU. It packs a hybrid attention architecture, supports up to 256K context natively, and is QAT (Quantization Aware Training) optimized, meaning it retains far more quality than standard post training quants at the same bit depth. This is NOT the MoE. This is 31 BILLION dense parameters, every single one of them loaded. # the flags I used: -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 8 --spec-draft-p-min 0.6 -c 6000 -v Multi Token Prediction (MTP) is still active here. Separate draft GGUF required, same as the 26B setup. # Results: → Decode: ~3 tokens/sec → Prefill: ~2 tokens/sec → Context: 6000 tokens → Hardware crying quietly in the corner: yes so is 3 tps actually usable? For real time back and forth chat? Not ideal. You're not having a fluid conversation at 3 tps. but slow ≠ useless. And this is where it gets genuinely interesting. think about how senior devs actually work in a real team. But when something is architectural, deeply complex, or needs serious reasoning? they walk down the hall and escalate to the senior. That's exactly the local AI agent architecture this unlocks: → Fast orchestrator model (Gemma 4 26B MoE at 25+ tps) handles routing, simple queries, tool calls, memory. The junior dev. → Gemma 4 31B dense is the senior, called only when the fast model genuinely hits a wall. Hard multi step reasoning. Complex code generation. Deep architectural decisions. The agentic loop stays fast. Only the hard hops touch the 31B. That's a legitimate production grade local AI architecture on a budget hardware. (requires 2 8gb gpus) other workflows where 3 tps is completely fine: - overnight batch jobs. summarize documents, extract structured data, review code. Fire it off. Sleep. wake up to results. - One shot deep reasoning - Silent code audit loops, you write and test, the 31B reviews diffs and flags issues in the background between your sprints - Any workflow where output quality > output speed A few weeks ago, nobody was running a 30B+ dense model on a single consumer GPU with 8 GB VRAM. At all. Now we're doing it on an Intel i7-H gaming laptop with a NVIDIA RTX 4060, thanks to llama.cpp + QAT quants + MTP speculative drafting. Google DeepMind said the Gemma 4 31B targets "consumer GPUs and workstations." They were not exaggerating. The hardware bar to run serious frontier class models locally keeps dropping. the tools are here. the models are here. you just have to be willing to abuse your laptop a little. what workflows would you actually run on a local 3 tps 31B dense model? genuinely curious. drop it below.

Alok

63,689 次观看 • 1 个月前

Hey friends, we're excited to announce that an additional 2,000 H100s will be added San Francisco Compute's on-demand market. It's the largest* interconnected cluster, from any provider (including hyperscalers), that you can get on a per hour basis. You're not locked in with San Francisco Compute. If DeepSeek can compete with OpenAI using 2,000 H800s, you too can train a state of the art RL model without ever having to sign a long-term contract that you can't exit. You could have trained DeepSeek-v3 for $4.5m for 1.5mo on SFC or $35m if you could only buy a 1 year contract off market. This was the dream Alex & I had since our audio model company (Junelark) died because it couldn't procure enough GPUs, and it's what we've been working towards for nearly two years. Long-term contracts are a trap; they make it so only the biggest of the big can compete in AI. They force startup founders to raise at massive valuations pre-revenue, which dilutes founders and employees and sets them up to fail when they can't raise their next round. This cluster will roll out over the next few weeks as we scale our infrastructure. Soon you'll be able to access it via our managed Kubernetes service or by reaching out to set up a custom solution. We're also exploring other ways of partnering with service providers to let them offer GPU-based services, like workers and inference endpoints, without being forced into a long-term contract with a hyperscaler. You no longer need to bet your company on GPU prices to offer GPU-based services. * We think! If you know of a larger, please correct us!

Hey friends, we're excited to announce that an additional 2,000 H100s will be added San Francisco Compute's on-demand market. It's the largest* interconnected cluster, from any provider (including hyperscalers), that you can get on a per hour basis. You're not locked in with San Francisco Compute. If DeepSeek can compete with OpenAI using 2,000 H800s, you too can train a state of the art RL model without ever having to sign a long-term contract that you can't exit. You could have trained DeepSeek-v3 for $4.5m for 1.5mo on SFC or $35m if you could only buy a 1 year contract off market. This was the dream Alex & I had since our audio model company (Junelark) died because it couldn't procure enough GPUs, and it's what we've been working towards for nearly two years. Long-term contracts are a trap; they make it so only the biggest of the big can compete in AI. They force startup founders to raise at massive valuations pre-revenue, which dilutes founders and employees and sets them up to fail when they can't raise their next round. This cluster will roll out over the next few weeks as we scale our infrastructure. Soon you'll be able to access it via our managed Kubernetes service or by reaching out to set up a custom solution. We're also exploring other ways of partnering with service providers to let them offer GPU-based services, like workers and inference endpoints, without being forced into a long-term contract with a hyperscaler. You no longer need to bet your company on GPU prices to offer GPU-based services. * We think! If you know of a larger, please correct us!

evan conrad

92,590 次观看 • 1 年前

This guy built a mini AI farm out of 4 Nvidia boxes It does not look like a data center. It looks like a stack of small machines sitting next to a laptop. But each box is a DGX Spark with Grace Blackwell inside, 128GB unified memory, and enough room to run models normal gaming GPUs cannot even open. Using the launch price from the article, 4 of them is almost $12,000 of local AI compute on one desk. That sounds expensive until you compare it to cloud GPUs. A serious AI builder can burn $1,500 to $3,000 a month renting A100s and H100s for client work, fine-tunes, agents and 70B models. He basically moved that bill from the cloud into hardware he owns. 4 Nvidia boxes. 512GB unified memory. No hourly meter running in the background. No rented GPUs eating the margin every time an agent runs too long. The funny part is most people still think local AI means a slow laptop running a toy model. Meanwhile guys like this are stacking compute at home. Save this, local AI is turning into the new mining farm.

This guy built a mini AI farm out of 4 Nvidia boxes It does not look like a data center. It looks like a stack of small machines sitting next to a laptop. But each box is a DGX Spark with Grace Blackwell inside, 128GB unified memory, and enough room to run models normal gaming GPUs cannot even open. Using the launch price from the article, 4 of them is almost $12,000 of local AI compute on one desk. That sounds expensive until you compare it to cloud GPUs. A serious AI builder can burn $1,500 to $3,000 a month renting A100s and H100s for client work, fine-tunes, agents and 70B models. He basically moved that bill from the cloud into hardware he owns. 4 Nvidia boxes. 512GB unified memory. No hourly meter running in the background. No rented GPUs eating the margin every time an agent runs too long. The funny part is most people still think local AI means a slow laptop running a toy model. Meanwhile guys like this are stacking compute at home. Save this, local AI is turning into the new mining farm.

Gipp 🦅

591,167 次观看 • 2 个月前

i've been paying 200$/month for AI tools for a while now claude, chatgpt, all of it thought that was just the price of doing things properly then i read this and had to close my laptop for a second someone bought a used GPU from eBay for $700 five years old. RTX 3090. the kind of card people sell when they upgrade for gaming plugged in a free model called Qwen 3.6 that alibaba quietly dropped last month and it scored 84.1 on vision tasks claude 4.5 opus scored 77.0 the 200$/month subscription lost to a free model running on old hardware the math gets worse after 4 months you've paid off the GPU from month 5 onwards you pay $8 for electricity that's it no subscription. no per message cost. nothing leaves your computer. ever. turns out i didn't have to is anyone else just finding out about this or was i the last one to know?

i've been paying 200$/month for AI tools for a while now claude, chatgpt, all of it thought that was just the price of doing things properly then i read this and had to close my laptop for a second someone bought a used GPU from eBay for $700 five years old. RTX 3090. the kind of card people sell when they upgrade for gaming plugged in a free model called Qwen 3.6 that alibaba quietly dropped last month and it scored 84.1 on vision tasks claude 4.5 opus scored 77.0 the 200$/month subscription lost to a free model running on old hardware the math gets worse after 4 months you've paid off the GPU from month 5 onwards you pay $8 for electricity that's it no subscription. no per message cost. nothing leaves your computer. ever. turns out i didn't have to is anyone else just finding out about this or was i the last one to know?

Crypto Mavka

14,126 次观看 • 1 个月前

Testing the new Gemma 4 12B (QAT) vision and OCR capabilities locally with LM Studio. # The setup: - GPU: NVIDIA RTX 4060 (8GB VRAM) - CPU: Intel i7 - Runner: LM Studio - Config: 32k context, 38 layers offloaded, Flash Attention enabled - Speed: ~14 tokens/sec decode throughput # The test: I gave it a screenshot of Google AI Studio. Prompt: "clone this. give me a single html file" # The result: A solid one shot replication. It successfully mapped out the layout, recognized the UI text, and structured the divs correctly, with only minor differences from the original. Results available at the end of the video. Quite capable for a 12B model running on budget consumer hardware. A gpu that costs only $300. # Why the architecture under the hood is notable: Unlike traditional models that rely on heavy, separate vision and audio encoders, Gemma 4 12B uses a unified, encoder free architecture. It bypasses separate multi stage encoders. Uses a 35M parameter vision embedder to project raw 48x48 pixel patches directly to the LLM hidden dimension. Local multimodal development is becoming highly accessible on standard hardware. If you've spun up Gemma 4 12B locally, what setup are you using and what kind of throughput are you seeing?

Testing the new Gemma 4 12B (QAT) vision and OCR capabilities locally with LM Studio. # The setup: - GPU: NVIDIA RTX 4060 (8GB VRAM) - CPU: Intel i7 - Runner: LM Studio - Config: 32k context, 38 layers offloaded, Flash Attention enabled - Speed: ~14 tokens/sec decode throughput # The test: I gave it a screenshot of Google AI Studio. Prompt: "clone this. give me a single html file" # The result: A solid one shot replication. It successfully mapped out the layout, recognized the UI text, and structured the divs correctly, with only minor differences from the original. Results available at the end of the video. Quite capable for a 12B model running on budget consumer hardware. A gpu that costs only $300. # Why the architecture under the hood is notable: Unlike traditional models that rely on heavy, separate vision and audio encoders, Gemma 4 12B uses a unified, encoder free architecture. It bypasses separate multi stage encoders. Uses a 35M parameter vision embedder to project raw 48x48 pixel patches directly to the LLM hidden dimension. Local multimodal development is becoming highly accessible on standard hardware. If you've spun up Gemma 4 12B locally, what setup are you using and what kind of throughput are you seeing?

Alok

25,717 次观看 • 1 个月前