Загрузка видео...

Не удалось загрузить видео

На главную

245 tokens/sec on DeepSeek-V4-Flash and it’s only using 2 RTX 6000 Blackwell GPUs. tinycorp dropped a 2-GPU tinybox edition in honor of DeepSeek fully ready to expand to 4 GPUs later. The GPU middle class just got a serious upgrade.

0x0SojalSec's profile picture

Md Ismail Šojal 🕷️

50,117 subscribers

16,148 просмотров • 1 месяц назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

⚡️ We just added support for Nvidia G4 (RTX Pro 6000 Blackwell Server Edition) GPUs in Colaboratory! 🔥 With a peak rating of 960 BF16 TFLOPs (~50% more than the A100-80G) and 96 GB of VRAM (20% more than the A100-80G), it’s the most efficient high-performance GPU we’ve ever released. ⏱️Time to make GPUs go brrrr 🏎️
0:14

⚡️ We just added support for Nvidia G4 (RTX Pro 6000 Blackwell Server Edition) GPUs in Colaboratory! 🔥 With a peak rating of 960 BF16 TFLOPs (~50% more than the A100-80G) and 96 GB of VRAM (20% more than the A100-80G), it’s the most efficient high-performance GPU we’ve ever released. ⏱️Time to make GPUs go brrrr 🏎️

Colaboratory

62,716 просмотров • 6 месяцев назад

Qwen 27B and Deepseek v4 Flash models working together on a single Lucebox machine, thats the beauty of having a discrete GPU paired with an unified access memory. A lot of space + good FLOPS & bandwidth. Working hard to increase concurrency performance on heterogeneous setups 🏎️
0:20

Qwen 27B and Deepseek v4 Flash models working together on a single Lucebox machine, thats the beauty of having a discrete GPU paired with an unified access memory. A lot of space + good FLOPS & bandwidth. Working hard to increase concurrency performance on heterogeneous setups 🏎️

mrciffa

10,940 просмотров • 13 дней назад

DeepSeek V4 Flash 0731 first test. DeepSeek V4 Flash 0731 is ridiculously cheap compared to Kimi K3, GPT-5.6, and Opus 5 For its price, this is a surprisingly capable model. First challenge: Generate a railway station in Three.js. The result exceeded my expectations for such a low-cost model. It’s definitely one to watch if you’re building 3D projects on a tight budget.
0:15

DeepSeek V4 Flash 0731 first test. DeepSeek V4 Flash 0731 is ridiculously cheap compared to Kimi K3, GPT-5.6, and Opus 5 For its price, this is a surprisingly capable model. First challenge: Generate a railway station in Three.js. The result exceeded my expectations for such a low-cost model. It’s definitely one to watch if you’re building 3D projects on a tight budget.

FHILY👑

21,825 просмотров • 1 месяц назад

free access to 8+ top AI models 😳 APInex is offering free models with 1M context - Gemini 3.8 Flash - Muse Spark 1.3 - GLM 5.3 Flash - Gemini 3.1 Pro - DeepSeek V4 Pro - GPT-5.6 Luna - Qwen 3.8 MAX - DeepSeek V4 Flash getting started: 1. go to 2. sign up with Gmail 3. create your API key 4. base url 5. choose a free model and start using free models, one API setup 👀
0:11

free access to 8+ top AI models 😳 APInex is offering free models with 1M context - Gemini 3.8 Flash - Muse Spark 1.3 - GLM 5.3 Flash - Gemini 3.1 Pro - DeepSeek V4 Pro - GPT-5.6 Luna - Qwen 3.8 MAX - DeepSeek V4 Flash getting started: 1. go to 2. sign up with Gmail 3. create your API key 4. base url 5. choose a free model and start using free models, one API setup 👀

K2S

27,390 просмотров • 21 дней назад

this is the best trick to maximum usage limits on chatgpt codex codex's best kept secret is that your main agent doesn't have to do everything... custom agents are just files in ~/.codex/agents, and one file gives you a second worker on deepseek v4 flash > create ~/.codex/agents/deepseek-worker.toml > set model = "opencode-go/deepseek-v4-flash" with model_reasoning_effort = "max" > keep it bounded: one task packet, no scope creep, report back ```toml name = "deepseek_worker" description = "bounded implementation, testing, and cleanup on deepseek v4 flash" model = "opencode-go/deepseek-v4-flash" model_reasoning_effort = "max" ``` then @ deepseek_worker in the composer... your root agent plans while the worker ships the implementation planning on the main model, execution on the flash lane... that's the whole trick (we run this exact file, last i checked it keeps the heavy turns off the main thread)
0:18

this is the best trick to maximum usage limits on chatgpt codex codex's best kept secret is that your main agent doesn't have to do everything... custom agents are just files in ~/.codex/agents, and one file gives you a second worker on deepseek v4 flash > create ~/.codex/agents/deepseek-worker.toml > set model = "opencode-go/deepseek-v4-flash" with model_reasoning_effort = "max" > keep it bounded: one task packet, no scope creep, report back ```toml name = "deepseek_worker" description = "bounded implementation, testing, and cleanup on deepseek v4 flash" model = "opencode-go/deepseek-v4-flash" model_reasoning_effort = "max" ``` then @ deepseek_worker in the composer... your root agent plans while the worker ships the implementation planning on the main model, execution on the flash lane... that's the whole trick (we run this exact file, last i checked it keeps the heavy turns off the main thread)

Avid

49,981 просмотров • 1 месяц назад

🚨 NVIDIA just flipped the entire AI game… and this is NOT about gaming. DeepSeek-V4-Pro is now live on their build platform. 1.6 TRILLION parameters. Yes… the largest open-source model on the planet right now. And here’s the crazy part: They’re letting you run it FREE On Blackwell GPUs in the cloud. This is the same level of hardware companies like Google, Meta, and Microsoft fight billions to access. Now it’s just… available. No waitlist. No insane setup. Just raw power. We’re watching the shift happen in real time: → From closed AI → open domination → From GPU scarcity → free access → From Big Tech control → builders winning This isn’t an update. It’s a warning shot. Who’s already testing this? Link👇
0:18

🚨 NVIDIA just flipped the entire AI game… and this is NOT about gaming. DeepSeek-V4-Pro is now live on their build platform. 1.6 TRILLION parameters. Yes… the largest open-source model on the planet right now. And here’s the crazy part: They’re letting you run it FREE On Blackwell GPUs in the cloud. This is the same level of hardware companies like Google, Meta, and Microsoft fight billions to access. Now it’s just… available. No waitlist. No insane setup. Just raw power. We’re watching the shift happen in real time: → From closed AI → open domination → From GPU scarcity → free access → From Big Tech control → builders winning This isn’t an update. It’s a warning shot. Who’s already testing this? Link👇

divyansh tiwari

29,941 просмотров • 5 месяцев назад

Codex can now run Deepseek-v4- flash! There's a catch though. Deepseek's official setup switches your entire codex over to them, so your GPT models stop showing up at all. This is exactly what Codex Router is for. It adds models to the list instead of replacing them, so sol, grok, kimi and deepseek all sit in the same picker and i just grab whichever one suits the job. Deepseek v4-flash is $0.28 per million output tokens. opus 4.8 is $25. same picker, 89x apart. Links in the comment. setup's in the video 👇
0:15

Codex can now run Deepseek-v4- flash! There's a catch though. Deepseek's official setup switches your entire codex over to them, so your GPT models stop showing up at all. This is exactly what Codex Router is for. It adds models to the list instead of replacing them, so sol, grok, kimi and deepseek all sit in the same picker and i just grab whichever one suits the job. Deepseek v4-flash is $0.28 per million output tokens. opus 4.8 is $25. same picker, 89x apart. Links in the comment. setup's in the video 👇

Ziwen

145,018 просмотров • 1 месяц назад

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!
0:14

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Akshay 🚀

126,273 просмотров • 10 месяцев назад

DeepSeek V4 Flash being ranked almost equal to Opus 4.8 is actually insane to me. Against Kimi K3, Opus 5, GPT-5.6 Sol, and Qwen 3.8 Max Preview, you give up a LOT by choosing DeepSeek. This is not a 2-point difference in practice. I’ll post the Opus 4.8 comparison next because you need to see this.
0:19

DeepSeek V4 Flash being ranked almost equal to Opus 4.8 is actually insane to me. Against Kimi K3, Opus 5, GPT-5.6 Sol, and Qwen 3.8 Max Preview, you give up a LOT by choosing DeepSeek. This is not a 2-point difference in practice. I’ll post the Opus 4.8 comparison next because you need to see this.

OmedTheVibeCoder

26,990 просмотров • 1 месяц назад

We're thrilled to unveil the new Provider Leaderboard on the Spheron GPU Marketplace! 🎮 Now, GPU providers can track their rewards and see how they stack up against others. 🏆 It’s time to up your game, add more compute power, and compete for the top spot! Get ready to level up your earnings and experience the thrill of healthy competition in the world of decentralized compute! 💪 Idle GPUs? Fill this out:-
0:14

We're thrilled to unveil the new Provider Leaderboard on the Spheron GPU Marketplace! 🎮 Now, GPU providers can track their rewards and see how they stack up against others. 🏆 It’s time to up your game, add more compute power, and compete for the top spot! Get ready to level up your earnings and experience the thrill of healthy competition in the world of decentralized compute! 💪 Idle GPUs? Fill this out:-

Spheron Network

106,042 просмотров • 2 лет назад

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers
0:20

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 просмотров • 16 дней назад

$IREN "we haven't disclosed the specific amount of GPUs" 1. 🤮 reminds me of $NBIS 2. Setting a terrible precedent here for future deals 3. Making it purposely difficult, to not let analysts properly value your 2027 revenue 4. Increasing the polarized view on IREN by the market However: "approximately 60MW of air-cooled Blackwells" 1. You typically don't talk about gross capacity in a deployment like this 2. If it would be gross capacity, the GPU hour rate at IT level would be crazy high (at PUE 1.2, $680m / 50 = 13.6m/MW) 3. At 60MW IT load, and ~14kW draw at DGX server level, we can get to ~4,286 DGX systems with 8 GPUs per. 4. Based on this we can conclude that 60MW of IT load can run approximately 34k DGX B300. 5. 34k DGX B300 at $680m/yr, would represent a GPU hour price of $2.28 Now this is the problem with not disclosing your GPU quantity. You purposely make your business model look bad, because by approach, you get to a GPU hour price that would imply a payback period of 4 years, where only the last year of the contract is 100% margin. But of course, we can also take "the glass is half full" approach. IREN has ordered 50K B300s from Dell. They have 2 purchase orders for this, 1 between Dell Canada and IE CA Leasing Ltd for 4 phases, and 1 between Dell USA and IE US Hardware 1 Inc (amended from IE US Hardware 4 Inc on April 27, 2026). The order for Canada is divided in 4 phases, and are going to Mackenzie for 80MW of gross capacity, which happens to be 4 buildings of 20MW. The order for Childress is divided in 2 phases, and are going to DC35 and DC36, (as depicted in the earnings presentation) and those are 50MW gross. The purchase price of the order for Childress was $1.2B, and for Canada it was $2.3B If we go with 50,000 B300s for a total of $3.5B then $1.2 would represent 34.285% of the 50,000 GPUs, or 17,140 B300s rounded down. For this calculation I will consider that $IREN will deploy 17,140 GPUs in 50MW gross capacity in DC35 and DC36 of block 3 in Childress.. That would imply at 1.2 PUE, IREN can run 17,140 B300s in 41.67MW IT load. Now by that ratio, they can run 24,680 GPUs in 60MW IT load — a massive difference with 34k units through the Nvidia DGX reference calculation. If common sense is applied, you can still get to 2 completely different outcomes, that show a difference of more than 9k GPUs. The GPU hour rate at 24.68k GPUs would be $3.145 per B300, as MASSIVE difference from the earlier calculated $2.28. Sure, the DGX system may be a factor here. And I'm sure that the reality is somewhere in the middle. But I personally hate this as an investor, to be unable to calculate profitability on unit economic basis. After all, contracts are signed on a $/GPU hour basis. Why hide this from your investors? Not being able to calculate payback periods, unable to calculate ROIC. And most importantly, we cannot properly assess the $NVDA deal on a contract basis. I really hope the payback period of this contract is not 4 years. I want the glass to be half full, but by starting to censor the purchases, IREN is taking a step in the wrong direction. Not a fan of this.
0:14

$IREN "we haven't disclosed the specific amount of GPUs" 1. 🤮 reminds me of $NBIS 2. Setting a terrible precedent here for future deals 3. Making it purposely difficult, to not let analysts properly value your 2027 revenue 4. Increasing the polarized view on IREN by the market However: "approximately 60MW of air-cooled Blackwells" 1. You typically don't talk about gross capacity in a deployment like this 2. If it would be gross capacity, the GPU hour rate at IT level would be crazy high (at PUE 1.2, $680m / 50 = 13.6m/MW) 3. At 60MW IT load, and ~14kW draw at DGX server level, we can get to ~4,286 DGX systems with 8 GPUs per. 4. Based on this we can conclude that 60MW of IT load can run approximately 34k DGX B300. 5. 34k DGX B300 at $680m/yr, would represent a GPU hour price of $2.28 Now this is the problem with not disclosing your GPU quantity. You purposely make your business model look bad, because by approach, you get to a GPU hour price that would imply a payback period of 4 years, where only the last year of the contract is 100% margin. But of course, we can also take "the glass is half full" approach. IREN has ordered 50K B300s from Dell. They have 2 purchase orders for this, 1 between Dell Canada and IE CA Leasing Ltd for 4 phases, and 1 between Dell USA and IE US Hardware 1 Inc (amended from IE US Hardware 4 Inc on April 27, 2026). The order for Canada is divided in 4 phases, and are going to Mackenzie for 80MW of gross capacity, which happens to be 4 buildings of 20MW. The order for Childress is divided in 2 phases, and are going to DC35 and DC36, (as depicted in the earnings presentation) and those are 50MW gross. The purchase price of the order for Childress was $1.2B, and for Canada it was $2.3B If we go with 50,000 B300s for a total of $3.5B then $1.2 would represent 34.285% of the 50,000 GPUs, or 17,140 B300s rounded down. For this calculation I will consider that $IREN will deploy 17,140 GPUs in 50MW gross capacity in DC35 and DC36 of block 3 in Childress.. That would imply at 1.2 PUE, IREN can run 17,140 B300s in 41.67MW IT load. Now by that ratio, they can run 24,680 GPUs in 60MW IT load — a massive difference with 34k units through the Nvidia DGX reference calculation. If common sense is applied, you can still get to 2 completely different outcomes, that show a difference of more than 9k GPUs. The GPU hour rate at 24.68k GPUs would be $3.145 per B300, as MASSIVE difference from the earlier calculated $2.28. Sure, the DGX system may be a factor here. And I'm sure that the reality is somewhere in the middle. But I personally hate this as an investor, to be unable to calculate profitability on unit economic basis. After all, contracts are signed on a $/GPU hour basis. Why hide this from your investors? Not being able to calculate payback periods, unable to calculate ROIC. And most importantly, we cannot properly assess the $NVDA deal on a contract basis. I really hope the payback period of this contract is not 4 years. I want the glass to be half full, but by starting to censor the purchases, IREN is taking a step in the wrong direction. Not a fan of this.

Frans Bakker

148,167 просмотров • 4 месяцев назад

Got DeepSeek V4 Flash running on 2x H200s at 160–200 tok/s on JarvisLabsAI this speed is perfect for in the loop things I do with these agents. It feels like a GLM-5.2-class model with much lower hardware requirements. I’m going to daily-drive it for a bit and see how it performs, but first impressions are really good.
0:16

Got DeepSeek V4 Flash running on 2x H200s at 160–200 tok/s on JarvisLabsAI this speed is perfect for in the loop things I do with these agents. It feels like a GLM-5.2-class model with much lower hardware requirements. I’m going to daily-drive it for a bit and see how it performs, but first impressions are really good.

Atharva Ingle

21,925 просмотров • 1 месяц назад

> 8 GPUs in one server rig > dude went homeless to build it > electrical bill costs more than rent now > while everyone else pays $400/month to openai > a 2 GPU desktop kills the api bill forever > rtx 4080 super + rtx 5060 ti = 32gb vram > runs qwen 3.6 with 100k context locally > no rate limits, no api keys, no data leaving the room > agents loop 400 times for free > claude opus still wins on hard reasoning > but local handles 90% of daily work > $1,200 setup pays itself off in 4 months > bookmark this and read the article below
0:15

> 8 GPUs in one server rig > dude went homeless to build it > electrical bill costs more than rent now > while everyone else pays $400/month to openai > a 2 GPU desktop kills the api bill forever > rtx 4080 super + rtx 5060 ti = 32gb vram > runs qwen 3.6 with 100k context locally > no rate limits, no api keys, no data leaving the room > agents loop 400 times for free > claude opus still wins on hard reasoning > but local handles 90% of daily work > $1,200 setup pays itself off in 4 months > bookmark this and read the article below

starmex

167,058 просмотров • 4 месяцев назад

This is what happens when 48GB of VRAM isn't enough. Someone put 4× NVIDIA RTX A6000 inside an HP Z8 Fury G5. That gives you 192GB of VRAM across four GPUs. But here's the interesting part: You don't magically get one 192GB GPU. The memory is split across four cards, and the software has to manage how the model is distributed between them. That's why these machines are interesting for local AI. The problem isn't always compute. Sometimes you just need somewhere to put the model. Would you build this today, or go for a newer GPU with more bandwidth?
0:13

This is what happens when 48GB of VRAM isn't enough. Someone put 4× NVIDIA RTX A6000 inside an HP Z8 Fury G5. That gives you 192GB of VRAM across four GPUs. But here's the interesting part: You don't magically get one 192GB GPU. The memory is split across four cards, and the software has to manage how the model is distributed between them. That's why these machines are interesting for local AI. The problem isn't always compute. Sometimes you just need somewhere to put the model. Would you build this today, or go for a newer GPU with more bandwidth?

KyzoroX

81,436 просмотров • 23 дней назад

DeepSeek R1 is *the* best model available right now. It's at the level of o1, but you can use it for free, and it's much faster. A huge leap forward that nobody saw coming. No wonder so many people are throwing tantrums online trying to discredit the Chinese students who built this. You can use DeepSeek in Visual Studio Code right now: 1. Install the Qodo Gen AI extension 2. Select DeepSeek R1 from their list of models The Qodo team is hosting DeepSeek on their servers, so none of your data will go to China. I've been building a Tetris game using DeepSeek, and this is the most impressive model I've seen so far.
0:17

DeepSeek R1 is *the* best model available right now. It's at the level of o1, but you can use it for free, and it's much faster. A huge leap forward that nobody saw coming. No wonder so many people are throwing tantrums online trying to discredit the Chinese students who built this. You can use DeepSeek in Visual Studio Code right now: 1. Install the Qodo Gen AI extension 2. Select DeepSeek R1 from their list of models The Qodo team is hosting DeepSeek on their servers, so none of your data will go to China. I've been building a Tetris game using DeepSeek, and this is the most impressive model I've seen so far.

Santiago

1,224,409 просмотров • 1 год назад

Loading DeFAI infrastructure ▓▓▓▓▓▓▓▓▓░ io.net, a decentralized GPU compute network has expanded to power the future of DeFAI for Injective builders. The integration aims to support the growing trend of developers building AI Agents, DeFAI, and gaming initiatives in web3. io.net provides Web3 builders with tools to train, fine-tune, and deploy ML models using decentralized resources. The collaboration combines Injective's iAgent framework with io.net’s extensive GPU network, setting the stage for more accessible and innovative development across use cases that require compute resources. Key features include: ✅ Access to over 10,000 cluster-ready GPUs and CPUs, ✅ AI-driven blockchain activities using Injective's iAgent SDK, and ✅ Potential for new on-chain financial products leveraging GPU pricing and data feeds. 👀 io.net’s DePIN network deploys on-demand, decentralized GPU resources globally, designed for low latency and high-throughput processing. Benefits include reduced barriers for AI/ML projects, enhanced integration of AI with on-chain activities, and democratized access to high-performance computing resources. Together, this marks a significant milestone in decentralized AI infrastructure, addressing key challenges and paving the way for unprecedented innovation in AI development and on-chain finance.
0:11

Loading DeFAI infrastructure ▓▓▓▓▓▓▓▓▓░ io.net, a decentralized GPU compute network has expanded to power the future of DeFAI for Injective builders. The integration aims to support the growing trend of developers building AI Agents, DeFAI, and gaming initiatives in web3. io.net provides Web3 builders with tools to train, fine-tune, and deploy ML models using decentralized resources. The collaboration combines Injective's iAgent framework with io.net’s extensive GPU network, setting the stage for more accessible and innovative development across use cases that require compute resources. Key features include: ✅ Access to over 10,000 cluster-ready GPUs and CPUs, ✅ AI-driven blockchain activities using Injective's iAgent SDK, and ✅ Potential for new on-chain financial products leveraging GPU pricing and data feeds. 👀 io.net’s DePIN network deploys on-demand, decentralized GPU resources globally, designed for low latency and high-throughput processing. Benefits include reduced barriers for AI/ML projects, enhanced integration of AI with on-chain activities, and democratized access to high-performance computing resources. Together, this marks a significant milestone in decentralized AI infrastructure, addressing key challenges and paving the way for unprecedented innovation in AI development and on-chain finance.

Injective 🥷

131,743 просмотров • 1 год назад

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies
0:19

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies

Alok

292,770 просмотров • 3 месяцев назад