Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

RTX 3090 was never designed to be a server GPU. Yet someone just built a 4-GPU AI workstation around four of them. 4× RTX 3090 96GB VRAM 256GB ECC RAM AMD EPYC 7K62 2000W PSU 11 fans The interesting part isn't the performance. It's the memory. Four 24GB cards... show more

KyzoroX

1,283 subscribers

36,030 просмотров • 4 дней назад •via X (Twitter)

Наука и технологии Игры Здоровье и велнес

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Finally got my hands on the big one. Qwen3.5-122B-A10B — 122 billion parameters. Too big for any single consumer GPU. So I rented 4 of each... and then one professional card to see if brute force even matters. - 1x RTX PRO 6000 (96GB): 101.4 tok/s - 4x 5090 (128GB): 87.0 tok/s - 4x 4090 (96GB): 25.1 tok/s - 4x 3090 (96GB): 20.8 tok/s One single $8,500 card beat four RTX 5090s

Finally got my hands on the big one. Qwen3.5-122B-A10B — 122 billion parameters. Too big for any single consumer GPU. So I rented 4 of each... and then one professional card to see if brute force even matters. - 1x RTX PRO 6000 (96GB): 101.4 tok/s - 4x 5090 (128GB): 87.0 tok/s - 4x 4090 (96GB): 25.1 tok/s - 4x 3090 (96GB): 20.8 tok/s One single $8,500 card beat four RTX 5090s

stevibe

195,832 просмотров • 6 месяцев назад

$15,000 for a Mac sounds insane. Until you realize what Apple is actually selling. 256GB of unified memory. Not a huge GPU farm. Not an NVIDIA rack. Not 4 different machines. Just one box that can keep multiple large LLMs in memory and run them locally. That changes the equation for local AI. NVIDIA wins when you need raw GPU compute. Apple gets interesting when the bottleneck is memory capacity. Because once your model doesn't fit into 24GB, 32GB or 48GB of VRAM, buying a faster GPU doesn't solve the problem. You need more memory. And that's where machines like the Mac Studio start looking less crazy. The interesting question isn't: “Is a $15K Mac faster than an NVIDIA GPU?” It isn't. The real question is: “How much are you willing to pay to make huge local models actually fit?” That’s where unified memory gets interesting.

$15,000 for a Mac sounds insane. Until you realize what Apple is actually selling. 256GB of unified memory. Not a huge GPU farm. Not an NVIDIA rack. Not 4 different machines. Just one box that can keep multiple large LLMs in memory and run them locally. That changes the equation for local AI. NVIDIA wins when you need raw GPU compute. Apple gets interesting when the bottleneck is memory capacity. Because once your model doesn't fit into 24GB, 32GB or 48GB of VRAM, buying a faster GPU doesn't solve the problem. You need more memory. And that's where machines like the Mac Studio start looking less crazy. The interesting question isn't: “Is a $15K Mac faster than an NVIDIA GPU?” It isn't. The real question is: “How much are you willing to pay to make huge local models actually fit?” That’s where unified memory gets interesting.

KyzoroX

19,536 просмотров • 27 дней назад

FOUR 600W GPUS. 384GB OF GDDR7. AND DELL MADE THE INSTALL LOOK EASIER THAN BUILDING A GAMING PC. this is a Dell PowerEdge XE7740 receiving four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. each card brings: 96GB GDDR7 ECC 24,064 CUDA cores 4 PFLOPS FP4 up to 600W but the most satisfying part is the cable management. every GPU power lead is already routed, labeled and held beside its assigned slot. drop in the card. lock the retention hardware. connect one 16-pin plug. no loose bundle hanging across the airflow path and no guessing which cable belongs where. the four-card build totals 384GB of GPU memory and 16 PFLOPS FP4. the 4U XE7740 supports up to eight of these GPUs, doubling that to 768GB and 32 PFLOPS FP4. honest line: the memory remains split between separate cards, and four GPUs can still pull up to 2.4kW. Dell made a rack-scale AI server this dense without turning the next GPU swap into cable surgery. bookmark & watch today ↓

FOUR 600W GPUS. 384GB OF GDDR7. AND DELL MADE THE INSTALL LOOK EASIER THAN BUILDING A GAMING PC. this is a Dell PowerEdge XE7740 receiving four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. each card brings: 96GB GDDR7 ECC 24,064 CUDA cores 4 PFLOPS FP4 up to 600W but the most satisfying part is the cable management. every GPU power lead is already routed, labeled and held beside its assigned slot. drop in the card. lock the retention hardware. connect one 16-pin plug. no loose bundle hanging across the airflow path and no guessing which cable belongs where. the four-card build totals 384GB of GPU memory and 16 PFLOPS FP4. the 4U XE7740 supports up to eight of these GPUs, doubling that to 768GB and 32 PFLOPS FP4. honest line: the memory remains split between separate cards, and four GPUs can still pull up to 2.4kW. Dell made a rack-scale AI server this dense without turning the next GPU swap into cable surgery. bookmark & watch today ↓

Grimmer

26,693 просмотров • 1 месяц назад

THIS $1,300 CARD HAS THE SAME 32GB OF VRAM AS A $2,000+ RTX 5090 - FOR LOCAL AI, THAT'S THE SPEC THAT DECIDES EVERYTHING. that clip is the gigabyte radeon ai pro r9700 ai top, 32gb of vram, built for local ai. the pitch is simple: the same memory capacity as an rtx 5090, for roughly a third of what a 5090 actually sells for. here's why that matters, and where it doesn't. for ai, vram is the gate. a model either fits in memory or it doesn't. 32gb loads models a 16gb card can't touch, full stop. so for running local llms and image models, this card does the one thing that counts, cheap. honest line: this is not a 5090 for gaming. the 5090 is far faster at raster and ray tracing. they only match on how much vram they carry. if you want frames, buy the 5090. if you want to fit big models locally, you just found the same 32gb for a third of the money. no $2,000+ price tag, no cloud gpu rental, no model too big for 16gb. the uncomfortable part isn't that amd is cheaper. it's that for local ai you were about to overpay for gaming power you'd never use. follow RetroChainer for more on paying for the spec that matters, and bookmark this before your next gpu.

THIS $1,300 CARD HAS THE SAME 32GB OF VRAM AS A $2,000+ RTX 5090 - FOR LOCAL AI, THAT'S THE SPEC THAT DECIDES EVERYTHING. that clip is the gigabyte radeon ai pro r9700 ai top, 32gb of vram, built for local ai. the pitch is simple: the same memory capacity as an rtx 5090, for roughly a third of what a 5090 actually sells for. here's why that matters, and where it doesn't. for ai, vram is the gate. a model either fits in memory or it doesn't. 32gb loads models a 16gb card can't touch, full stop. so for running local llms and image models, this card does the one thing that counts, cheap. honest line: this is not a 5090 for gaming. the 5090 is far faster at raster and ray tracing. they only match on how much vram they carry. if you want frames, buy the 5090. if you want to fit big models locally, you just found the same 32gb for a third of the money. no $2,000+ price tag, no cloud gpu rental, no model too big for 16gb. the uncomfortable part isn't that amd is cheaper. it's that for local ai you were about to overpay for gaming power you'd never use. follow RetroChainer for more on paying for the spec that matters, and bookmark this before your next gpu.

RetroChainer

38,844 просмотров • 1 месяц назад

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

stevibe

193,034 просмотров • 4 месяцев назад

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

402,323 просмотров • 4 месяцев назад

HE STOPPED RENTING AI AND BOLTED FOUR USED RTX 3090s TO ONE BOARD watch the table: four five-year-old GPUs, 24GB of VRAM each. 96GB of local model memory sitting under one operator’s control. the build looks extreme. the subscription math is worse. then opened his statement and found $340/month going to seven AI companies. six cancellations saved $140 immediately. then he realized a single used 3090 for roughly $700 could already run Qwen 27B locally: - no token meter - no rate limits - no files leaving the machine - roughly 25–30 tokens per second you do not need the four-card monster in this clip. one card moves most daily coding, writing and document work off the cloud. the old flex was paying for every frontier model. the new flex is owning the compute they keep renting back to you.

HE STOPPED RENTING AI AND BOLTED FOUR USED RTX 3090s TO ONE BOARD watch the table: four five-year-old GPUs, 24GB of VRAM each. 96GB of local model memory sitting under one operator’s control. the build looks extreme. the subscription math is worse. then opened his statement and found $340/month going to seven AI companies. six cancellations saved $140 immediately. then he realized a single used 3090 for roughly $700 could already run Qwen 27B locally: - no token meter - no rate limits - no files leaving the machine - roughly 25–30 tokens per second you do not need the four-card monster in this clip. one card moves most daily coding, writing and document work off the cloud. the old flex was paying for every frontier model. the new flex is owning the compute they keep renting back to you.

Myttle

74,168 просмотров • 2 месяцев назад

A DEAD 2021 MINING RIG CAN NOW BECOME A 96GB AI FACTORY THAT REPLACES $4,944 A YEAR IN SUBSCRIPTIONS 00:26 he shows the full build: four RTX 3090s, separate power supplies, and enough cables to keep the entire rig running at full load. This machine was built for crypto. Now the same hardware can run local LLMs, generate images, transcribe videos, analyze files, and power AI agents. Each RTX 3090 has 24GB of VRAM. Four cards give the rig 96GB, enough to handle models that normally need an expensive workstation or rented server. Used 3090s cost around $650 to $750 each. The full stack is roughly $2,800, while a pile of AI subscriptions can quietly reach $412 every month. Seven months of subscriptions already costs more than the GPUs. After that, the rig keeps working while the software bill keeps draining another $4,944 every year. Leave it running overnight and it can organize notes, process saved videos, prepare drafts, monitor prices, and summarize documents without constant input. The crypto boom built thousands of these machines. AI is giving them a second life, turning old mining rigs into private factories for code, research, and content. Bookmark & Watch it run ↓

A DEAD 2021 MINING RIG CAN NOW BECOME A 96GB AI FACTORY THAT REPLACES $4,944 A YEAR IN SUBSCRIPTIONS 00:26 he shows the full build: four RTX 3090s, separate power supplies, and enough cables to keep the entire rig running at full load. This machine was built for crypto. Now the same hardware can run local LLMs, generate images, transcribe videos, analyze files, and power AI agents. Each RTX 3090 has 24GB of VRAM. Four cards give the rig 96GB, enough to handle models that normally need an expensive workstation or rented server. Used 3090s cost around $650 to $750 each. The full stack is roughly $2,800, while a pile of AI subscriptions can quietly reach $412 every month. Seven months of subscriptions already costs more than the GPUs. After that, the rig keeps working while the software bill keeps draining another $4,944 every year. Leave it running overnight and it can organize notes, process saved videos, prepare drafts, monitor prices, and summarize documents without constant input. The crypto boom built thousands of these machines. AI is giving them a second life, turning old mining rigs into private factories for code, research, and content. Bookmark & Watch it run ↓

slash1s

12,286 просмотров • 2 месяцев назад

EIGHT USED RTX 3090s. 192GB OF VRAM. ONE PRIVATE BOX BUILT TO RUN 70B AI MODELS WITHOUT RENTING A DATACENTER. 00:06 the team lowers all eight GPUs into the chassis at once, turning a pile of old gaming cards into the core of a serious local AI server. each 3090 contributes 24GB of memory. combined, the rack has enough VRAM for heavyweight models, large document collections, long context and multiple jobs running side by side. these cards are five years old, but AI economics work differently from gaming. memory capacity matters more than having the newest badge, which is why used 3090s remain valuable. the server can keep one model loaded while separate GPUs handle video, images, transcription and overnight agents without sending private company data into the cloud. bookmark this build before yesterday’s gaming GPUs become tomorrow’s cheapest AI infrastructure.

EIGHT USED RTX 3090s. 192GB OF VRAM. ONE PRIVATE BOX BUILT TO RUN 70B AI MODELS WITHOUT RENTING A DATACENTER. 00:06 the team lowers all eight GPUs into the chassis at once, turning a pile of old gaming cards into the core of a serious local AI server. each 3090 contributes 24GB of memory. combined, the rack has enough VRAM for heavyweight models, large document collections, long context and multiple jobs running side by side. these cards are five years old, but AI economics work differently from gaming. memory capacity matters more than having the newest badge, which is why used 3090s remain valuable. the server can keep one model loaded while separate GPUs handle video, images, transcription and overnight agents without sending private company data into the cloud. bookmark this build before yesterday’s gaming GPUs become tomorrow’s cheapest AI infrastructure.

Gipp 🦅

119,001 просмотров • 2 месяцев назад

THIS GUY WIRED GMKTEC EVO-X2 BOXES, A $40 USB-C BRICK AND AN RTX 3090 INTO A $6,800/MONTH OFFLINE AI FARM it looks like a messy rack with purple cables, mini pcs and a gpu lying on the desk. but the real point is brutal: one $1,800 gmktec evo-x2 can replace a $440/month ai stack, then the rack turns it into a local backend. the evo-x2 runs amd’s ryzen ai max+ 395 with 128gb of unified memory shared between cpu, gpu and npu. on linux, that gives up to 110gb of usable vram for local models without renting cloud compute. that is why the setup gets weird. qwen3-coder 30b, llama 3.3 70b, deepseek v3 and 200b-class models can run from boxes smaller than consoles. the rtx 3090 becomes a 24gb side node, not the whole system. the $40 anker nano ii 65w brick is the funny part. cheap power, tiny machines, one old gpu and local inference. the whole stack starts looking less like a workstation and more like a private ai grid. the subscription math is ugly. claude code max is $200/month, chatgpt pro is $200/month, cursor is $20/month and gemini adds another $20. that is $5,280/year before api usage even starts. point claude code at localhost, pull models through ollama and run agents all day: code reviews, scraping, transcription, research reports and client automations. 3 clients at $2,200/month turns this into a $6,800/month backend. bookmark this and read the article below

THIS GUY WIRED GMKTEC EVO-X2 BOXES, A $40 USB-C BRICK AND AN RTX 3090 INTO A $6,800/MONTH OFFLINE AI FARM it looks like a messy rack with purple cables, mini pcs and a gpu lying on the desk. but the real point is brutal: one $1,800 gmktec evo-x2 can replace a $440/month ai stack, then the rack turns it into a local backend. the evo-x2 runs amd’s ryzen ai max+ 395 with 128gb of unified memory shared between cpu, gpu and npu. on linux, that gives up to 110gb of usable vram for local models without renting cloud compute. that is why the setup gets weird. qwen3-coder 30b, llama 3.3 70b, deepseek v3 and 200b-class models can run from boxes smaller than consoles. the rtx 3090 becomes a 24gb side node, not the whole system. the $40 anker nano ii 65w brick is the funny part. cheap power, tiny machines, one old gpu and local inference. the whole stack starts looking less like a workstation and more like a private ai grid. the subscription math is ugly. claude code max is $200/month, chatgpt pro is $200/month, cursor is $20/month and gemini adds another $20. that is $5,280/year before api usage even starts. point claude code at localhost, pull models through ollama and run agents all day: code reviews, scraping, transcription, research reports and client automations. 3 clients at $2,200/month turns this into a $6,800/month backend. bookmark this and read the article below

Gipp 🦅

20,031 просмотров • 2 месяцев назад

watch gemma 4 12b q8 dancing on a single rtx 3090 at 33 tokens a second average. google dropped this two days ago and it's the kind of thing that quietly moves the floor. a fully multimodal model, text image and audio in one net, 256k context, apache licensed, running entirely on one consumer gpu, no one metering your tokens. what you're watching is the whole loop live: the server streaming tokens top left, the gpu pegged bottom left, the answer landing on the right. all local, all mine. a year ago this needed someone else's datacenter. today it's a card you can buy. open source isn't catching up anymore, it's setting the pace. how fast does yours run?

watch gemma 4 12b q8 dancing on a single rtx 3090 at 33 tokens a second average. google dropped this two days ago and it's the kind of thing that quietly moves the floor. a fully multimodal model, text image and audio in one net, 256k context, apache licensed, running entirely on one consumer gpu, no one metering your tokens. what you're watching is the whole loop live: the server streaming tokens top left, the gpu pegged bottom left, the answer landing on the right. all local, all mine. a year ago this needed someone else's datacenter. today it's a card you can buy. open source isn't catching up anymore, it's setting the pace. how fast does yours run?

Sudo su

36,614 просмотров • 3 месяцев назад

Revolutionizing Access to #GPU Power - One Node at a Time.⚡️ Our 10th Utility is coming out on June 20.🚨 BIG GPU Nodes Platform isn’t just another name in the crowded GPU cloud services space, we’re a game-changer. Our platform is purpose-built to bridge the gap between server owners and those hungry for raw, high-performance computing power. 🖥️ We specialize in connecting suppliers from individual server owners to enterprise-scale GPU rig operators with consumers who demand serious horsepower for AI, machine learning, data science, and other compute-heavy applications. Whether you’re offering a sleek single-GPU setup or a massive mining-style array with multiple GPUs, BIG GPU Nodes is your launchpad. We make it simple to rent out underutilized GPU resources and just as easy for users to find exactly what they need — fast, affordable, and tailored to their workload.⛓️‍💥 This is more than cloud computing. This is decentralized GPU power for the future of innovation. ♾️ This product will generate revenue and we will give back to our loyal $CNCT holders. ✨ We’re proud to have MESSIER | M87 as our trusted partner and advisor, backing us with this vision. #M87 🤝 $CNCT From the very beginning, we promised you ONE BIG ECOSYSTEM, and a seamless space where everything you need comes together. Now, that vision is becoming reality. Millions of users will soon be connected, working, creating, and thriving within it. 🌍

Revolutionizing Access to #GPU Power - One Node at a Time.⚡️ Our 10th Utility is coming out on June 20.🚨 BIG GPU Nodes Platform isn’t just another name in the crowded GPU cloud services space, we’re a game-changer. Our platform is purpose-built to bridge the gap between server owners and those hungry for raw, high-performance computing power. 🖥️ We specialize in connecting suppliers from individual server owners to enterprise-scale GPU rig operators with consumers who demand serious horsepower for AI, machine learning, data science, and other compute-heavy applications. Whether you’re offering a sleek single-GPU setup or a massive mining-style array with multiple GPUs, BIG GPU Nodes is your launchpad. We make it simple to rent out underutilized GPU resources and just as easy for users to find exactly what they need — fast, affordable, and tailored to their workload.⛓️‍💥 This is more than cloud computing. This is decentralized GPU power for the future of innovation. ♾️ This product will generate revenue and we will give back to our loyal $CNCT holders. ✨ We’re proud to have MESSIER | M87 as our trusted partner and advisor, backing us with this vision. #M87 🤝 $CNCT From the very beginning, we promised you ONE BIG ECOSYSTEM, and a seamless space where everything you need comes together. Now, that vision is becoming reality. Millions of users will soon be connected, working, creating, and thriving within it. 🌍

Fixera AI

26,682 просмотров • 1 год назад

INTEL JUST QUIETLY BUILT THE CHEAPEST PATH TO 128GB OF LOCAL AI VRAM, AND IT BEATS NVIDIA'S DGX SPARK BY 2X FOR THE SAME $4,000 00:11 four Arc Pro B70s racked in one chassis. 32GB of GDDR6 each, $949 MSRP, roughly $1,000 street per card stack four and you get 128GB of VRAM for about $4,000 total. a lab benchmarked the quad against a DGX Spark and a single RTX Pro 6000. the four B70s doubled the Spark or better they trail the Pro 6000, but that card is $10,000. so the real choice is now $4K for 128GB from Intel or $10K for 96GB from NVIDIA the article covers the DGX Spark at $3,999 as the single-desk local AI ceiling. this Intel quad matches the price, beats the memory throughput, and nobody is talking about it yet save this before Intel's VRAM-per-dollar advantage becomes the default homelab build ↓

INTEL JUST QUIETLY BUILT THE CHEAPEST PATH TO 128GB OF LOCAL AI VRAM, AND IT BEATS NVIDIA'S DGX SPARK BY 2X FOR THE SAME $4,000 00:11 four Arc Pro B70s racked in one chassis. 32GB of GDDR6 each, $949 MSRP, roughly $1,000 street per card stack four and you get 128GB of VRAM for about $4,000 total. a lab benchmarked the quad against a DGX Spark and a single RTX Pro 6000. the four B70s doubled the Spark or better they trail the Pro 6000, but that card is $10,000. so the real choice is now $4K for 128GB from Intel or $10K for 96GB from NVIDIA the article covers the DGX Spark at $3,999 as the single-desk local AI ceiling. this Intel quad matches the price, beats the memory throughput, and nobody is talking about it yet save this before Intel's VRAM-per-dollar advantage becomes the default homelab build ↓

NO1ennn

139,803 просмотров • 1 месяц назад

THE MS-A2 IS THE MINISFORUM BOX HERMES AGENT WAS NEVER MEANT TO RUN ON, AND THAT'S BY DESIGN The MS-S1 MAX earns its spot running Hermes Agent because of one thing. 128GB of unified memory, up to 96GB of it handed straight to the GPU. That's the only reason a 120B model fits and runs locally for $0 a month The MS-A2 solves a different problem. Ryzen 9 9955HX, 16 cores, 32 threads, up to 96GB of regular DDR5-5600, no unified pool Three M.2 PCIe 4.0 slots, one U.2, two 22110. Dual 10Gbps SFP+ LAN plus 2.5G. WiFi 6E. Bluetooth 5.3. A slide-out motherboard for fast upgrades. A real PCIe x16 slot that actually takes a low-profile GPU That last part is where the two machines split for good. The MS-S1 MAX's PCIe slot won't take a GPU at all, every bit of GPU power has to come from the unified chip itself. The MS-A2 trades that unified memory trick for raw expandability instead One box runs a local AI agent. The other runs a home lab that needs storage, networking, and room to grow. Minisforum built both on purpose, not as the same product wearing two names

NO1ennn

25,645 просмотров • 2 месяцев назад

48GB of VRAM rents for $500 a month, and this rig has room to double it. A vertical frame. A bare motherboard. One graphics card, exposed and humming. No rack. No server room. Just a desk. Ryzen 9 9950x. AsRock Taichi X870E. 96GB of DDR5 at 6000MT/s. A Thermalright Royal Pretor 130 Ultra on top. A 1300-watt Super Flower supply feeding all of it. One card. 48GB of VRAM. On the open market, that card alone rents for $0.58 to $0.82 an hour. Run it around the clock, and it clears $500 a month, crunching someone else's AI workloads. The motherboard and the power supply were not picked at random. Both, for one reason: a second GPU. Slide it in, and VRAM doubles to 96GB. So does the money: $1,000 a month, from a machine that fits under a desk. Most people rent that power by the hour. This desk prints it.

48GB of VRAM rents for $500 a month, and this rig has room to double it. A vertical frame. A bare motherboard. One graphics card, exposed and humming. No rack. No server room. Just a desk. Ryzen 9 9950x. AsRock Taichi X870E. 96GB of DDR5 at 6000MT/s. A Thermalright Royal Pretor 130 Ultra on top. A 1300-watt Super Flower supply feeding all of it. One card. 48GB of VRAM. On the open market, that card alone rents for $0.58 to $0.82 an hour. Run it around the clock, and it clears $500 a month, crunching someone else's AI workloads. The motherboard and the power supply were not picked at random. Both, for one reason: a second GPU. Slide it in, and VRAM doubles to 96GB. So does the money: $1,000 a month, from a machine that fits under a desk. Most people rent that power by the hour. This desk prints it.

AiMind

30,720 просмотров • 19 дней назад

Someone built a workstation that needs 4.5 kilowatts. That's more than any standard wall circuit can deliver. Renderboxes Molecule Air: eight RTX 6000 Ada, a Threadripper PRO 7975WX, 768GB of DDR5 ECC 5800, and three Corsair HX1500i supplies stacked together. 🔴 US 15A circuit → 1,800W 🔴 US 20A circuit → 2,400W 🔴 EU 16A circuit → 3,680W 🟢 This machine → 4,500W The PSU rating isn't the draw, though. Eight cards at 300W plus the CPU lands near 3kW sustained. That's three space heaters running in the room, and the cooling to remove that heat costs more power on top. Note: 384GB of VRAM total, but it's eight separate 48GB pools talking over PCIe. Not one pool of 384. This is what "local AI" looks like at the top end. A datacenter rack that happens to be in a room. So before the GPU budget — what does your panel actually deliver?

Someone built a workstation that needs 4.5 kilowatts. That's more than any standard wall circuit can deliver. Renderboxes Molecule Air: eight RTX 6000 Ada, a Threadripper PRO 7975WX, 768GB of DDR5 ECC 5800, and three Corsair HX1500i supplies stacked together. 🔴 US 15A circuit → 1,800W 🔴 US 20A circuit → 2,400W 🔴 EU 16A circuit → 3,680W 🟢 This machine → 4,500W The PSU rating isn't the draw, though. Eight cards at 300W plus the CPU lands near 3kW sustained. That's three space heaters running in the room, and the cooling to remove that heat costs more power on top. Note: 384GB of VRAM total, but it's eight separate 48GB pools talking over PCIe. Not one pool of 384. This is what "local AI" looks like at the top end. A datacenter rack that happens to be in a room. So before the GPU budget — what does your panel actually deliver?

KyzoroX

21,491 просмотров • 9 дней назад

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

left curve dev

166,478 просмотров • 4 месяцев назад

Nvidia announces the new RTX Spark, a new platform powered by the NX1 CPU, and shows off Spark laptops running 007 First Light and Forza 6. The CPU has 20 ARM based cores and a Blackwell RTX GPU with 6144 CUDA Cores. This is the same core count as a 5070, but with 128GB of unified LPDDR5X RAM memory sitting in the same package as the CPU and GPU. The entire Nvidia software stack is available, particularly CUDA, vital for AI. Nvidia's new laptops will likely be ideal for running local LLMs be cause the unified memory means you can load models up to 120-180B parameters (quantized). These laptops are expected to ship later this year and could become strong competitors to high-end MacBooks and even Mac Studios for local AI workloads, thanks to CUDA support and unified memory. Price is unannounced.

Nvidia announces the new RTX Spark, a new platform powered by the NX1 CPU, and shows off Spark laptops running 007 First Light and Forza 6. The CPU has 20 ARM based cores and a Blackwell RTX GPU with 6144 CUDA Cores. This is the same core count as a 5070, but with 128GB of unified LPDDR5X RAM memory sitting in the same package as the CPU and GPU. The entire Nvidia software stack is available, particularly CUDA, vital for AI. Nvidia's new laptops will likely be ideal for running local LLMs be cause the unified memory means you can load models up to 120-180B parameters (quantized). These laptops are expected to ship later this year and could become strong competitors to high-end MacBooks and even Mac Studios for local AI workloads, thanks to CUDA support and unified memory. Price is unannounced.

Grummz

30,573 просмотров • 3 месяцев назад

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 просмотров • 2 лет назад