Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Cloud GPU training is a scam. A single M4 MacBook does 2.9 TFLOPS. Seven friends with MacBooks match an NVIDIA A100. Alexander Hayes just open-sourced a tool that makes this work over Wi-Fi. It's called AirTrain. Here's how it works: Traditional distributed training (DDP) syncs gradients after every single...

heygurisingh's profile picture

Guri Singh

53,747 subscribers

160,201 Aufrufe • vor 2 Monaten •via X (Twitter)

BildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰
0:32

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 Aufrufe • vor 1 Jahr

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.
0:20

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,487 Aufrufe • vor 1 Jahr

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:
0:20

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:

Awni Hannun

42,761 Aufrufe • vor 11 Monaten

Jensen Huang said Grok 5 will be 7 Trillion parameter model. On time to train, with the training window fixed at 1 month, the new Vera-Rubin GPU system of Nvidia needs 1/4 the number of systems compared with Blackwell to train the same frontier model. “factory throughput” improves by about 10x over Blackwell, and Blackwell itself was about 10x over Hopper. i.e. overall Rubin is roughly 100x Hopper in factory throughput per watt, which matters hugely, because a 1 GW, $50B data center is power-limited and revenue scales with throughput per watt. --- From Nvidia YT channel
1:24

Jensen Huang said Grok 5 will be 7 Trillion parameter model. On time to train, with the training window fixed at 1 month, the new Vera-Rubin GPU system of Nvidia needs 1/4 the number of systems compared with Blackwell to train the same frontier model. “factory throughput” improves by about 10x over Blackwell, and Blackwell itself was about 10x over Hopper. i.e. overall Rubin is roughly 100x Hopper in factory throughput per watt, which matters hugely, because a 1 GW, $50B data center is power-limited and revenue scales with throughput per watt. --- From Nvidia YT channel

Rohan Paul

165,770 Aufrufe • vor 6 Monaten

I ran a 35-billion parameter AI agent on a $600 Mac mini. Specs: M4 Mac-Mini 16GB RAM The model doesn't fit in RAM. It pages from the SSD at 30 tokens/second. On NVIDIA, the same paging gives you 1.6 tok/s. Apple Silicon gives you 30. That's 18.6x faster. No cloud. No API keys. $0/month. Here's what it can do 🧵
1:39

I ran a 35-billion parameter AI agent on a $600 Mac mini. Specs: M4 Mac-Mini 16GB RAM The model doesn't fit in RAM. It pages from the SSD at 30 tokens/second. On NVIDIA, the same paging gives you 1.6 tok/s. Apple Silicon gives you 30. That's 18.6x faster. No cloud. No API keys. $0/month. Here's what it can do 🧵

thestreamingdev()

727,259 Aufrufe • vor 3 Monaten

I trained models across MacBooks using Apple's AirDrop protocol. grove is a distributed training library for Apple Silicon. Devices discover each other over AWDL, a direct radio link. If there's a shared WiFi network it upgrades to that for speed, otherwise everything goes over the direct link. No router, no cloud, no setup. grove start -n 4 grove join
0:34

I trained models across MacBooks using Apple's AirDrop protocol. grove is a distributed training library for Apple Silicon. Devices discover each other over AWDL, a direct radio link. If there's a shared WiFi network it upgrades to that for speed, otherwise everything goes over the direct link. No router, no cloud, no setup. grove start -n 4 grove join

Swarnim Jain

667,109 Aufrufe • vor 3 Monaten

JENSEN HUANG UNVEILED A $3,000 SUPERCOMPUTER THAT REPLACES $1,900/MONTH IN CLOUD GPUS WITH $2 IN ELECTRICITY. IT FITS ON YOUR DESK a developer was paying $1,900 a month renting a100 and h100 cloud gpus. he bought the dgx spark, set it up in an afternoon, and his monthly bill dropped from $1,900 to $2 in electricity the box runs 128gb of unified memory and 1 petaflop of ai compute. a 4090 gives you 24gb of vram and taps out at 30b models. the spark loads 70b at full precision and stretches to 200b without breaking a sweat break-even hit in 6 weeks. after that the $1,890 he used to wire to a rental company every month stayed in his business. first year that's $22,000 back install ollama, change one line in your code, point it at localhost instead of the cloud. nothing else changes. nothing leaves your machine and nothing costs money per request cloud gpu costs aren't getting cheaper and rate limits keep getting tighter. the people who pulled their ai workloads onto a box on their desk in 2026 are going to look very far ahead in 2028 bookmark this and read the article below
0:33

JENSEN HUANG UNVEILED A $3,000 SUPERCOMPUTER THAT REPLACES $1,900/MONTH IN CLOUD GPUS WITH $2 IN ELECTRICITY. IT FITS ON YOUR DESK a developer was paying $1,900 a month renting a100 and h100 cloud gpus. he bought the dgx spark, set it up in an afternoon, and his monthly bill dropped from $1,900 to $2 in electricity the box runs 128gb of unified memory and 1 petaflop of ai compute. a 4090 gives you 24gb of vram and taps out at 30b models. the spark loads 70b at full precision and stretches to 200b without breaking a sweat break-even hit in 6 weeks. after that the $1,890 he used to wire to a rental company every month stayed in his business. first year that's $22,000 back install ollama, change one line in your code, point it at localhost instead of the cloud. nothing else changes. nothing leaves your machine and nothing costs money per request cloud gpu costs aren't getting cheaper and rate limits keep getting tighter. the people who pulled their ai workloads onto a box on their desk in 2026 are going to look very far ahead in 2028 bookmark this and read the article below

starmex

141,376 Aufrufe • vor 1 Monat

What Lisa Su actually held on stage: A mini PC the size of a lunchbox running Qwen3-235B locally, with no cloud and no discrete GPU Inside: the Ryzen AI Max+ 395, 128GB unified memory, 110GB usable as VRAM on Linux The first x86 chip that handles 200+ billion parameters on a single die AMD claims it beats the RTX 5080 by several times on memory-bound models — because the 5080 simply cannot fit them $1,400 to $2,500 once. cloud bills run $200 to $400 a month It pays for itself in a few months, then costs nothing per request This is not a faster GPU. it is the first real argument that your AI does not belong in someone else's data center
0:31

What Lisa Su actually held on stage: A mini PC the size of a lunchbox running Qwen3-235B locally, with no cloud and no discrete GPU Inside: the Ryzen AI Max+ 395, 128GB unified memory, 110GB usable as VRAM on Linux The first x86 chip that handles 200+ billion parameters on a single die AMD claims it beats the RTX 5080 by several times on memory-bound models — because the 5080 simply cannot fit them $1,400 to $2,500 once. cloud bills run $200 to $400 a month It pays for itself in a few months, then costs nothing per request This is not a faster GPU. it is the first real argument that your AI does not belong in someone else's data center

plutos

1,232,746 Aufrufe • vor 25 Tagen

I got Llama 3 running in my browser using only my GPU with my Wi-Fi switched OFF completely client-side WebGPU is a new feature in browsers where JS can use the GPU of the device and apparently you can run LLMs on it too, and it's fast! My end goal for 🧠 is a locally run client side 100% private conversation where nothing gets sent to, processed in or stored in the cloud This will be helpful because I prefer making it web based instead of coding for 4 different native platforms (iOS, Android, MacOS, Windows) Two challenges still: 1) the user has to download the LLM model first and the smallest model is still ~3GB, but they only have to download it once 2) not all devices have fast GPUs yet for LLMs, but it's starting to become a default in most devices e.g.s smartphones
1:37

I got Llama 3 running in my browser using only my GPU with my Wi-Fi switched OFF completely client-side WebGPU is a new feature in browsers where JS can use the GPU of the device and apparently you can run LLMs on it too, and it's fast! My end goal for 🧠 is a locally run client side 100% private conversation where nothing gets sent to, processed in or stored in the cloud This will be helpful because I prefer making it web based instead of coding for 4 different native platforms (iOS, Android, MacOS, Windows) Two challenges still: 1) the user has to download the LLM model first and the smallest model is still ~3GB, but they only have to download it once 2) not all devices have fast GPUs yet for LLMs, but it's starting to become a default in most devices e.g.s smartphones

@levelsio

261,352 Aufrufe • vor 2 Jahren

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.
1:24

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,403 Aufrufe • vor 6 Monaten

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below
1:06

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below

starmex

93,086 Aufrufe • vor 26 Tagen

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓
1:21

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,051 Aufrufe • vor 26 Tagen

This is actually insane. 🔥 NVIDIA built a single AI that replaces decades of animation technology. 15,000 frames per second. 2 milliseconds of delay. For robots and games. It's called MotionBricks. One neural network trained on 350,000 motion clips. You tell it: go here, pick up this sword, do it in zombie style. It generates everything else foot placement, balance, transitions, follow-through in real time. No animation graphs. No hand-crafted transitions. No per-task training. One model does it all. In the demo, a character navigates a space, picks up a sword, vaults over a bench, sits down, and switches between zombie, injured, and skipping styles. Every single frame generated live by the AI. And here's the part that matters most. This is now the core motion engine inside NVIDIA's GR00T the stack powering humanoid robots used in research globally. Same AI that animates game characters is now teaching robots how to move their entire body. vc: NVIDIA Robotics
0:36

This is actually insane. 🔥 NVIDIA built a single AI that replaces decades of animation technology. 15,000 frames per second. 2 milliseconds of delay. For robots and games. It's called MotionBricks. One neural network trained on 350,000 motion clips. You tell it: go here, pick up this sword, do it in zombie style. It generates everything else foot placement, balance, transitions, follow-through in real time. No animation graphs. No hand-crafted transitions. No per-task training. One model does it all. In the demo, a character navigates a space, picks up a sword, vaults over a bench, sits down, and switches between zombie, injured, and skipping styles. Every single frame generated live by the AI. And here's the part that matters most. This is now the core motion engine inside NVIDIA's GR00T the stack powering humanoid robots used in research globally. Same AI that animates game characters is now teaching robots how to move their entire body. vc: NVIDIA Robotics

Vaibhav Sisinty

260,012 Aufrufe • vor 24 Tagen

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables
21:59

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

47,399 Aufrufe • vor 17 Tagen

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year
1:16

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year

NO1ennn

48,054 Aufrufe • vor 11 Tagen

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.
0:25

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

38,953 Aufrufe • vor 2 Tagen

people burn $1,900 a month on cloud, this $2,999 NVIDIA box runs 5 AI models at once and costs $10 to power 128GB of unified memory, stack two together and you double it, enough to run models a $2,000 graphics card cannot even open in the demo it ran Qwen3 Vision, Nemotron, a voice model, text to speech and speech to text all at once, and only used half the box here is the math that makes people switch: > cloud GPUs were costing $1,900 every single month, billed forever > the box is $2,999 one time, then about $10 of electricity a month > it paid for itself in two months, $22,000 stayed in the business in year one > zero data ever leaves your desk, no rental, no rate limits most people are still renting what this owns outright the full breakdown of the box and the exact math is in the article below
1:04

people burn $1,900 a month on cloud, this $2,999 NVIDIA box runs 5 AI models at once and costs $10 to power 128GB of unified memory, stack two together and you double it, enough to run models a $2,000 graphics card cannot even open in the demo it ran Qwen3 Vision, Nemotron, a voice model, text to speech and speech to text all at once, and only used half the box here is the math that makes people switch: > cloud GPUs were costing $1,900 every single month, billed forever > the box is $2,999 one time, then about $10 of electricity a month > it paid for itself in two months, $22,000 stayed in the business in year one > zero data ever leaves your desk, no rental, no rate limits most people are still renting what this owns outright the full breakdown of the box and the exact math is in the article below

John Doe

40,605 Aufrufe • vor 1 Monat

The bottleneck in AI has quietly shifted. - It's not the models. They are capable. - It's not the frameworks. They are mature. - It's not even the data, in many cases. When you want to train a model today, the first question isn't "what architecture should I use?" Instead, it's: "Where am I going to get infrastructure that actually works?" Not just GPUs but the entire stack: compute, deployment, scaling, storage. The traditional path is major cloud providers or specialized GPU clouds. Both have the same problem: they're built for enterprises with committed workloads, minimum spend requirements, contract negotiations, and involve quota approvals that take days. Even the "on-demand" options require you to piece together training, deployment, and scaling across different services. By the time you're actually training, hours, if not days, have passed. And there's a subtler cost: part of your brain is always managing infrastructure instead of thinking about the actual problem. I've been using Runpod for a while now, and it's the closest I've found to infrastructure that just disappears. I pay for the serverless solution by the second, and stop when I'm done. This sounds like it should be the default across all providers, but it isn't. For instance, when I'm prototyping, I don't need an H100. Instead, I need the flexibility to use cheaper GPUs that are actually available, where I can iterate fast and not worry about cost. An A40 at a few cents per hour is perfect for this. Then, when the approach is validated, I scale up. This matches how good engineering actually works. Running distributed training across multiple nodes for multi-GPU training usually requires significant infra work. RunPod abstracts most of this away. A lot of the advantage in AI comes from iteration speed. Infra that adds days of latency to that loop is a real cost, even if it's hard to measure. But good infra gets out of your way. It's available when you need it, invisible when you don't. In the video below, I have shown a simple model training workflow trained using PyTorch in Jupyter Lab. It runs in a dedicated PyTorch Pod hosted on Runpod, and I worked with the team to put this together for you. Find a link to start using Runpod in the replies!
0:23

The bottleneck in AI has quietly shifted. - It's not the models. They are capable. - It's not the frameworks. They are mature. - It's not even the data, in many cases. When you want to train a model today, the first question isn't "what architecture should I use?" Instead, it's: "Where am I going to get infrastructure that actually works?" Not just GPUs but the entire stack: compute, deployment, scaling, storage. The traditional path is major cloud providers or specialized GPU clouds. Both have the same problem: they're built for enterprises with committed workloads, minimum spend requirements, contract negotiations, and involve quota approvals that take days. Even the "on-demand" options require you to piece together training, deployment, and scaling across different services. By the time you're actually training, hours, if not days, have passed. And there's a subtler cost: part of your brain is always managing infrastructure instead of thinking about the actual problem. I've been using Runpod for a while now, and it's the closest I've found to infrastructure that just disappears. I pay for the serverless solution by the second, and stop when I'm done. This sounds like it should be the default across all providers, but it isn't. For instance, when I'm prototyping, I don't need an H100. Instead, I need the flexibility to use cheaper GPUs that are actually available, where I can iterate fast and not worry about cost. An A40 at a few cents per hour is perfect for this. Then, when the approach is validated, I scale up. This matches how good engineering actually works. Running distributed training across multiple nodes for multi-GPU training usually requires significant infra work. RunPod abstracts most of this away. A lot of the advantage in AI comes from iteration speed. Infra that adds days of latency to that loop is a real cost, even if it's hard to measure. But good infra gets out of your way. It's available when you need it, invisible when you don't. In the video below, I have shown a simple model training workflow trained using PyTorch in Jupyter Lab. It runs in a dedicated PyTorch Pod hosted on Runpod, and I worked with the team to put this together for you. Find a link to start using Runpod in the replies!

Avi Chawla

13,696 Aufrufe • vor 6 Monaten

Ever wondered what training a former GB athlete does in his 40's as he prepares for the challenge of a lifetime? Part One:
2:18

Ever wondered what training a former GB athlete does in his 40's as he prepares for the challenge of a lifetime? Part One:

Jimmy Watkins

30,655 Aufrufe • vor 2 Jahren

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours
22:35

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours

Noisy

79,665 Aufrufe • vor 21 Tagen