Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

There's no point in doing decentralized training without efficient communication. >> DiLoCo (H=15) ships ~480mb/merge with 163 syncs. >> SparseLoCo (H=15) ships ~5.5–17mb/merge at 0.78–3.12% density with 163 syncs Top-K Compression + 2 bit comms ~28–89× smaller per sync than DiLoCo. Subnet 3 :: Luis el grande If you...

opentensor's profile picture

Openτensor Foundaτion

171,776 subscribers

17,767 görüntüleme • 9 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Hash Rate - Ep 145: Templar (3), Basilica (39), Grail (81) 🧙 Guest: Distributed State of templar + basilica + grail 00:00 Introduction to Covenant and Its Subnets 02:01 Understanding Templar: Decentralized Training 05:31 The Role of Grail and Basilica 10:38 Transitioning from Research to Product 14:37 Competitive Landscape and Model Development 21:58 Target Audience and Market Potential 25:51 The Effect of $TAO Flow 32:52 Incentive Mechanisms in Basilica 39:24 Vision for Templar: Democratizing AI Training 45:20 Future Vision
53:04

Hash Rate - Ep 145: Templar (3), Basilica (39), Grail (81) 🧙 Guest: Distributed State of templar + basilica + grail 00:00 Introduction to Covenant and Its Subnets 02:01 Understanding Templar: Decentralized Training 05:31 The Role of Grail and Basilica 10:38 Transitioning from Research to Product 14:37 Competitive Landscape and Model Development 21:58 Target Audience and Market Potential 25:51 The Effect of $TAO Flow 32:52 Incentive Mechanisms in Basilica 39:24 Vision for Templar: Democratizing AI Training 45:20 Future Vision

Mark Jeffrey

27,995 görüntüleme • 6 ay önce

Hash Rate - Ep 109: Templar $TAO sn3 - Decentralized AI Training 🧙 Guest: Sam Dare Distributed State of templar 00:00 Introductions 02:51 dTAO Launch: Is it working? 09:05 What is Templar? 11:50 Decentralized Training Networks Explained 15:02 The Mechanics of Training with Templar 26:58 DeepSeek's Impact on AI Training 31:28 Monetization for AI Models 36:51 What Makes A Successful Subnet? 44:09 Sam's Journey to Templar
54:10

Hash Rate - Ep 109: Templar $TAO sn3 - Decentralized AI Training 🧙 Guest: Sam Dare Distributed State of templar 00:00 Introductions 02:51 dTAO Launch: Is it working? 09:05 What is Templar? 11:50 Decentralized Training Networks Explained 15:02 The Mechanics of Training with Templar 26:58 DeepSeek's Impact on AI Training 31:28 Monetization for AI Models 36:51 What Makes A Successful Subnet? 44:09 Sam's Journey to Templar

Mark Jeffrey

18,048 görüntüleme • 1 yıl önce

Introducing OpenDiLoCo, an open-source implementation and scaling of DeepMind’s Distributed Low-Communication (DiLoCo) method, enabling globally distributed AI model training.
0:21

Introducing OpenDiLoCo, an open-source implementation and scaling of DeepMind’s Distributed Low-Communication (DiLoCo) method, enabling globally distributed AI model training.

Prime Intellect

260,002 görüntüleme • 1 yıl önce

Covenant Labs just did a 90-minute AMA breaking down their 3 Bittensor subnets. templar. basilica. grail. Pre-training, compute, and post-training under one roof. Most people missed it. Here's everything they said. Covenant is building what they call the "end to end intelligence continuum." Three subnets. Three layers of the AI stack. All permissionless. Templar (SN3) handles decentralized pre-training. Basilica (SN39) handles compute. Grail (SN81) handles RL post-training. Distributed State, the lead, put it bluntly. Decentralized training is "humanity's last dance." Not about beating OpenAI head to head. About creating optionality. About making it cheap enough for anyone to train models. The gap between academia and frontier labs is growing exponentially. Researchers can't afford to experiment. The actual training run costs 5% of the reported budget. The other 95% is experimentation. If Covenant cracks cheap training, that entire surface area opens up. On Templar specifically: • Hit 39% emission on Bittensor. Highest since Apex was the only subnet on the network • Covenant-72B trained permissionlessly with 70+ contributors on commodity internet • 1.1 trillion tokens processed. No centralized data center • Performance competitive with LLaMA-2-70B On Grail, something flew under the radar. They built Pulse. A weight synchronization method that compresses model updates by 100x. • In RL post-training, only ~1% of weights update per step • Pulse exploits that sparsity. Lossless compression • Prime Intellect's comparable system took 14 minutes to sync a 30B model • Pulse makes decentralized RL training actually feasible at scale • Already used by Cursor The lead researcher on Grail said they've trained on math, code, and GPU kernels. Got 40-60% improvement on benchmarks. Working toward agentic training with 100K+ token context and 30B+ parameter models. On Basilica, the compute subnet: The team was blunt. Just reselling GPU hours is a 5-10% margin game. Traditional compute providers already do that. Their play is value-added services. • "GPU as code." No dashboard. No UI. Agents interact via SDK • Custom scheduler that places workloads across heterogeneous hardware • Verification checks for GPU, CPU, bandwidth, memory, storage, and OS security • Partnerships with providers like Mass Compute for 10-20% below market pricing • Miners compete on useful infrastructure, not just GPU hours Sam then went on a rant about the miner burn debate. His take: Bittensor had to grow up. dTAO introduced investors. The old "miners are God" philosophy doesn't hold. • Subnet owners have a duty to protect token value • Miners are a resource optimization exercise, not a cost reduction exercise • 100% miner emissions on compute subnets = immediate sell pressure • The 41% miner allocation is arbitrary. Different business models need different splits • Fish (who started burns) agreed. Burns usually mean the validation isn't mature enough The bigger point. You can't police burns. Subnets just send to their own keys instead of the burn address. Subnet 28 does exactly that. Sam's position: judge subnets on outcomes, not process. Const has changed the protocol 9-10 times in 2 years. That iteration speed is Bittensor's actual moat. The whole Covenant thesis is playing out in real time. TAO is up 100%+ in a month. Jensen Huang name-dropped the network. Grayscale has an ETF filing. But the real story is three subnets quietly building every layer of decentralized AI.
1:39:04

Covenant Labs just did a 90-minute AMA breaking down their 3 Bittensor subnets. templar. basilica. grail. Pre-training, compute, and post-training under one roof. Most people missed it. Here's everything they said. Covenant is building what they call the "end to end intelligence continuum." Three subnets. Three layers of the AI stack. All permissionless. Templar (SN3) handles decentralized pre-training. Basilica (SN39) handles compute. Grail (SN81) handles RL post-training. Distributed State, the lead, put it bluntly. Decentralized training is "humanity's last dance." Not about beating OpenAI head to head. About creating optionality. About making it cheap enough for anyone to train models. The gap between academia and frontier labs is growing exponentially. Researchers can't afford to experiment. The actual training run costs 5% of the reported budget. The other 95% is experimentation. If Covenant cracks cheap training, that entire surface area opens up. On Templar specifically: • Hit 39% emission on Bittensor. Highest since Apex was the only subnet on the network • Covenant-72B trained permissionlessly with 70+ contributors on commodity internet • 1.1 trillion tokens processed. No centralized data center • Performance competitive with LLaMA-2-70B On Grail, something flew under the radar. They built Pulse. A weight synchronization method that compresses model updates by 100x. • In RL post-training, only ~1% of weights update per step • Pulse exploits that sparsity. Lossless compression • Prime Intellect's comparable system took 14 minutes to sync a 30B model • Pulse makes decentralized RL training actually feasible at scale • Already used by Cursor The lead researcher on Grail said they've trained on math, code, and GPU kernels. Got 40-60% improvement on benchmarks. Working toward agentic training with 100K+ token context and 30B+ parameter models. On Basilica, the compute subnet: The team was blunt. Just reselling GPU hours is a 5-10% margin game. Traditional compute providers already do that. Their play is value-added services. • "GPU as code." No dashboard. No UI. Agents interact via SDK • Custom scheduler that places workloads across heterogeneous hardware • Verification checks for GPU, CPU, bandwidth, memory, storage, and OS security • Partnerships with providers like Mass Compute for 10-20% below market pricing • Miners compete on useful infrastructure, not just GPU hours Sam then went on a rant about the miner burn debate. His take: Bittensor had to grow up. dTAO introduced investors. The old "miners are God" philosophy doesn't hold. • Subnet owners have a duty to protect token value • Miners are a resource optimization exercise, not a cost reduction exercise • 100% miner emissions on compute subnets = immediate sell pressure • The 41% miner allocation is arbitrary. Different business models need different splits • Fish (who started burns) agreed. Burns usually mean the validation isn't mature enough The bigger point. You can't police burns. Subnets just send to their own keys instead of the burn address. Subnet 28 does exactly that. Sam's position: judge subnets on outcomes, not process. Const has changed the protocol 9-10 times in 2 years. That iteration speed is Bittensor's actual moat. The whole Covenant thesis is playing out in real time. TAO is up 100%+ in a month. Jensen Huang name-dropped the network. Grayscale has an ETF filing. But the real story is three subnets quietly building every layer of decentralized AI.

Jesus Martinez

26,522 görüntüleme • 2 ay önce

The Holy Grail of Bittensor: Permissionless, trustless, Incentivized >> Distributed Training. Dive into how two subnets in Bittensor are pioneering this frontier in a world-first effort. Full episode on YT+ Spotify:
1:06

The Holy Grail of Bittensor: Permissionless, trustless, Incentivized >> Distributed Training. Dive into how two subnets in Bittensor are pioneering this frontier in a world-first effort. Full episode on YT+ Spotify:

Openτensor Foundaτion

65,455 görüntüleme • 1 yıl önce

Launching a new concept where a research paper is reviewed jointly with the author ! Being a firm believer in decentralised training, it is a pleasure to have Arthur Douillard @ ICLR25, Senior Research Scientist Google DeepMind, discussing his work on "Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch" ! Links below ! Timestamps 0:01:16 - What is Diloco ? 0:03:58 - SOTA in decentralized training 0:05:41 - Diloco algorithm explained 0:13:36 - Limitation of Diloco 0:18:23 - What is Streaming Diloco ? 0:19:43 - Streaming Diloco applied on different "axis" ? 0:23:30 - Streaming Diloco algorithm explained 0:30:34 - Overlap communication & computation 0:37:30 - Compute utilization across range bandwidth (part 1) 0:42:34 - Compute utilization across range bandwidth (part 2) 0:45:30 - Evaluation loss + potential risks when scaling the method 0:48:57 - How much we can overlap steps ? (part 1) 0:53:21 - Compressing the outer gradients 0:58:18 - Hypothesis on why FP4 compression works so well 1:00:37 - How much we can overlap steps ? (part 2) 1:04:42 - What's the next step ?
1:07:37

Launching a new concept where a research paper is reviewed jointly with the author ! Being a firm believer in decentralised training, it is a pleasure to have Arthur Douillard @ ICLR25, Senior Research Scientist Google DeepMind, discussing his work on "Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch" ! Links below ! Timestamps 0:01:16 - What is Diloco ? 0:03:58 - SOTA in decentralized training 0:05:41 - Diloco algorithm explained 0:13:36 - Limitation of Diloco 0:18:23 - What is Streaming Diloco ? 0:19:43 - Streaming Diloco applied on different "axis" ? 0:23:30 - Streaming Diloco algorithm explained 0:30:34 - Overlap communication & computation 0:37:30 - Compute utilization across range bandwidth (part 1) 0:42:34 - Compute utilization across range bandwidth (part 2) 0:45:30 - Evaluation loss + potential risks when scaling the method 0:48:57 - How much we can overlap steps ? (part 1) 0:53:21 - Compressing the outer gradients 0:58:18 - Hypothesis on why FP4 compression works so well 1:00:37 - How much we can overlap steps ? (part 2) 1:04:42 - What's the next step ?

Ferdinand Mom

13,219 görüntüleme • 1 yıl önce

New research from the team officially drops tomorrow: scaling decentralized training from 72B to frontier models. SparseLoCo solved gradient sync. Heterogenous SparseLoCo solves activation transmission between pipeline stages. What's it mean? The internet can be your datacenter. Inter-datacenter training that connects datacenters with consumer GPUs, university clusters, and distributed infrastructure worldwide. Join us for a TGIF deep dive with Distributed State and the research team.
0:27

New research from the team officially drops tomorrow: scaling decentralized training from 72B to frontier models. SparseLoCo solved gradient sync. Heterogenous SparseLoCo solves activation transmission between pipeline stages. What's it mean? The internet can be your datacenter. Inter-datacenter training that connects datacenters with consumer GPUs, university clusters, and distributed infrastructure worldwide. Join us for a TGIF deep dive with Distributed State and the research team.

templar

10,047 görüntüleme • 5 ay önce

The largest decentralised LLM pre-training run in history. SN3 templar trained Covenant-72B across 70+ contributors on open internet infrastructure. Now it’s being discussed by Chamath Palihapitiya with NVIDIA CEO Jensen Huang. Distributed, open-weight model training on Bittensor is getting started.
4:13

The largest decentralised LLM pre-training run in history. SN3 templar trained Covenant-72B across 70+ contributors on open internet infrastructure. Now it’s being discussed by Chamath Palihapitiya with NVIDIA CEO Jensen Huang. Distributed, open-weight model training on Bittensor is getting started.

Openτensor Foundaτion

120,670 görüntüleme • 2 ay önce

Decentralized RL as fast as centralized RL for LLMs. Bittensor SN81 grail has shattered the bandwidth barrier with PULSE (Patch Updates via Lossless Sparse Encoding). "By identifying the 99% weight sparsity inherent in Adam-bounded updates, we've achieved a 100x reduction in weight synchronization - dropping 14GB transfers to just ~140MB " Erfan Miahi + const :: covenant
1:26

Decentralized RL as fast as centralized RL for LLMs. Bittensor SN81 grail has shattered the bandwidth barrier with PULSE (Patch Updates via Lossless Sparse Encoding). "By identifying the 99% weight sparsity inherent in Adam-bounded updates, we've achieved a 100x reduction in weight synchronization - dropping 14GB transfers to just ~140MB " Erfan Miahi + const :: covenant

Openτensor Foundaτion

13,068 görüntüleme • 2 ay önce

For anyone trying to understand Bittensor from first principles, this lecture is a useful place to start. Presented by Bittensor co-founder const. Learn Bittensor > Start with Bitcoin, distributed systems, incentives, > How Bitcoin leads to Bittensor Subnets coordinating AI infrastructure. Topics: // Start - Bitcoin as more than a digital currency // Risks of AI centralization + closed systems // "The incentive computer" // How Bittensor subnets work (mining, validating) // How distributed AI infrastructure could scale globally // Impact on students, builders & future founders Recorded at the National University of Singapore Computer Science Club. NUS Computing Chapters - Bitcoin, AI, and Bittensor - Bitcoin history and decentralization - AI changes how engineers work - The danger of centralized AI power - Why most crypto visions fail - Bitcoin as the world’s largest compute network - Bitcoin as a market for compute - The idea of an “incentive computer” - Bitcoin compared to Bittensor - Classroom example of decentralized scoring - A simple subnet example - SN62 :: Ridges AI | SN62 SWE agents - SN3 templar :: Distributed AI Training - SN52 lium.io :: GPU rentals on Bittensor 128 subnets, some examples Why this matters for the future of work Q&A Subnet examples mentioned @ SN64 - Serverless + TEE Compute :: Chutes SN8 - Prop firm Vanta Trading SN52 - AutoML :: Gradients SN62 - SWE agents :: Ridges AI | SN62 SN51 - Compute / GPU rental lium.io SN4 - TEE compute for enterprise :: Targon SN3 - 72B Distributed Training run :: templar SN41 - Prediction markets :: Almanac SN44 - Computer Vision Score - Subnet 44 SN68 - Drug discovery :: METANOVA SN18 - Weather Forecasting Zeus | SN 18 SN50 - Bitcoin prediction data :: Synthdata SN61 - Quantum computing :: qBitTensor Labs SN14 - Bitcoin mining pool :: TaoHash SN34 - Perp Dex :: 0xMarkets SN17 - 3D model generation :: 404 SN33 - Data analytics :: ReadyAI SN19 - [Since relaunched] RPC infrastructure :
53:45

For anyone trying to understand Bittensor from first principles, this lecture is a useful place to start. Presented by Bittensor co-founder const. Learn Bittensor > Start with Bitcoin, distributed systems, incentives, > How Bitcoin leads to Bittensor Subnets coordinating AI infrastructure. Topics: // Start - Bitcoin as more than a digital currency // Risks of AI centralization + closed systems // "The incentive computer" // How Bittensor subnets work (mining, validating) // How distributed AI infrastructure could scale globally // Impact on students, builders & future founders Recorded at the National University of Singapore Computer Science Club. NUS Computing Chapters - Bitcoin, AI, and Bittensor - Bitcoin history and decentralization - AI changes how engineers work - The danger of centralized AI power - Why most crypto visions fail - Bitcoin as the world’s largest compute network - Bitcoin as a market for compute - The idea of an “incentive computer” - Bitcoin compared to Bittensor - Classroom example of decentralized scoring - A simple subnet example - SN62 :: Ridges AI | SN62 SWE agents - SN3 templar :: Distributed AI Training - SN52 lium.io :: GPU rentals on Bittensor 128 subnets, some examples Why this matters for the future of work Q&A Subnet examples mentioned @ SN64 - Serverless + TEE Compute :: Chutes SN8 - Prop firm Vanta Trading SN52 - AutoML :: Gradients SN62 - SWE agents :: Ridges AI | SN62 SN51 - Compute / GPU rental lium.io SN4 - TEE compute for enterprise :: Targon SN3 - 72B Distributed Training run :: templar SN41 - Prediction markets :: Almanac SN44 - Computer Vision Score - Subnet 44 SN68 - Drug discovery :: METANOVA SN18 - Weather Forecasting Zeus | SN 18 SN50 - Bitcoin prediction data :: Synthdata SN61 - Quantum computing :: qBitTensor Labs SN14 - Bitcoin mining pool :: TaoHash SN34 - Perp Dex :: 0xMarkets SN17 - 3D model generation :: 404 SN33 - Data analytics :: ReadyAI SN19 - [Since relaunched] RPC infrastructure :

Openτensor Foundaτion

1,172,229 görüntüleme • 2 ay önce

we're turning University of Waterloo into a supercomputer! arceus is a cross-device distributed compute network for training large models, using model/tensor/pipeline parallelism. you can train anything, from deep neural networks to language models on the network. oss & deployment soon!
0:33

we're turning University of Waterloo into a supercomputer! arceus is a cross-device distributed compute network for training large models, using model/tensor/pipeline parallelism. you can train anything, from deep neural networks to language models on the network. oss & deployment soon!

rajan agarwal

35,534 görüntüleme • 1 yıl önce

Preview: New Subnet Analyzer Tool by WOMBO in our latest episode with Keith Singery, zion | w.ai and const Topics in this episode: - Cellular automata in AI and the potential of Rule 30 - Model compression for affordable AI on consumer hardware: SN39 - The trillion-row dataset initiative in SN36 - Validator diversity and decentralized ecosystem growth
1:23

Preview: New Subnet Analyzer Tool by WOMBO in our latest episode with Keith Singery, zion | w.ai and const Topics in this episode: - Cellular automata in AI and the potential of Rule 30 - Model compression for affordable AI on consumer hardware: SN39 - The trillion-row dataset initiative in SN36 - Validator diversity and decentralized ecosystem growth

Openτensor Foundaτion

60,351 görüntüleme • 1 yıl önce

AI History is being made Distributed Training on Bittensor Made possible with Macrocosmos + Manifold + const
2:33

AI History is being made Distributed Training on Bittensor Made possible with Macrocosmos + Manifold + const

Openτensor Foundaτion

596,002 görüntüleme • 1 yıl önce

“Some of these research problems we’ve been going after really are holy grail predictive problems for drug design - and we’ve made some incredible breakthroughs there, which have really stunned our chemists.” On Sequoia Capital’s Training Data podcast, our Chief AI Officer Max Jaderberg talks about the scale of AI drug discovery and its potential to transform how we design new medicines. Listen to the full podcast at the link below:
0:29

“Some of these research problems we’ve been going after really are holy grail predictive problems for drug design - and we’ve made some incredible breakthroughs there, which have really stunned our chemists.” On Sequoia Capital’s Training Data podcast, our Chief AI Officer Max Jaderberg talks about the scale of AI drug discovery and its potential to transform how we design new medicines. Listen to the full podcast at the link below:

Isomorphic Labs

12,294 görüntüleme • 1 yıl önce

Which Servant would you choose in the Holy Grail War? #AnimeJapan2026
0:39

Which Servant would you choose in the Holy Grail War? #AnimeJapan2026

ANN Events

19,825 görüntüleme • 2 ay önce

The Mission :: const Worlds First Permissionless, Incentivized, Open Source, Decentralized Training on Subnet 3 :: templar Novelty Search >> E037 Full episode:
3:00

The Mission :: const Worlds First Permissionless, Incentivized, Open Source, Decentralized Training on Subnet 3 :: templar Novelty Search >> E037 Full episode:

Openτensor Foundaτion

30,302 görüntüleme • 1 yıl önce

Sat down with Bittensor co-founder Jacob Steeves (Const) for a deep dive into the state of Bittensor in 2026. We covered how Subnet 3 trained a 72B parameter model in a fully permissionless decentralized way, how subnets create markets for digital commodities like gradients and inference, the dynamic TAO tokenomics that lock emissions into subnet liquidity pools, and why open ownership of AI matters more than ever as centralized labs head toward trillion-dollar IPOs. 0:00 Intro: Bittensor Co-Founder Jacob Steeves 1:13 Jacob's Background, From Deep Learning to Crypto 4:45 Founding Bittensor: The Monetary Computer 7:58 How Subnets Work as Markets for Digital Commodities 11:18 Subnet 3: Decentralized Model Training 15:28 Permissionless Training at Internet Scale 18:52 Why This Matters: Open Source AI Ownership 25:53 Subnet Tour: Optimization, Inference, and Compute 29:55 Affine: Beating Qwen 30B via Market Incentives 34:26 TAO Tokenomics and Dynamic TAO 51:54 How Investors Participate: Just Buy TAO 54:05 Open Ownership vs Fiat AI
1:00:37

Sat down with Bittensor co-founder Jacob Steeves (Const) for a deep dive into the state of Bittensor in 2026. We covered how Subnet 3 trained a 72B parameter model in a fully permissionless decentralized way, how subnets create markets for digital commodities like gradients and inference, the dynamic TAO tokenomics that lock emissions into subnet liquidity pools, and why open ownership of AI matters more than ever as centralized labs head toward trillion-dollar IPOs. 0:00 Intro: Bittensor Co-Founder Jacob Steeves 1:13 Jacob's Background, From Deep Learning to Crypto 4:45 Founding Bittensor: The Monetary Computer 7:58 How Subnets Work as Markets for Digital Commodities 11:18 Subnet 3: Decentralized Model Training 15:28 Permissionless Training at Internet Scale 18:52 Why This Matters: Open Source AI Ownership 25:53 Subnet Tour: Optimization, Inference, and Compute 29:55 Affine: Beating Qwen 30B via Market Incentives 34:26 TAO Tokenomics and Dynamic TAO 51:54 How Investors Participate: Just Buy TAO 54:05 Open Ownership vs Fiat AI

VirtualBacon

41,859 görüntüleme • 2 ay önce

Cloud GPU training is a scam. A single M4 MacBook does 2.9 TFLOPS. Seven friends with MacBooks match an NVIDIA A100. Alexander Hayes just open-sourced a tool that makes this work over Wi-Fi. It's called AirTrain. Here's how it works: Traditional distributed training (DDP) syncs gradients after every single step. For a 124M parameter model, that's ~500MB exchanged per step. You need 50 GB/s of sustained bandwidth. Impossible over Wi-Fi. AirTrain uses the DiLoCo algorithm. Each Mac trains independently for 500 steps, then syncs only the difference. One sync per 500 steps instead of one per step. 500x less network communication. Wi-Fi actually works. The entire sync takes ~2 seconds. Here's what makes it wild: → Zero-config discovery. Devices find each other automatically via mDNS/Bonjour. Same protocol as AirDrop. → Fault tolerant. Nodes can join and leave mid-training without killing the run. → Checkpoint relay. Train for a few hours, export a checkpoint, hand it off to someone else to continue. Like a relay race for ML training. → Built on Apple's MLX framework. Native to M1/M2/M3/M4/M5 unified memory. No host-to-device copy overhead. → Local dashboard. Real-time loss curves, peer monitoring, throughput metrics in your browser. Here's the wildest part: An M4 Max with 128GB unified memory can train a 70B parameter model without offloading. An NVIDIA RTX 4090 has 24GB VRAM. Apple Silicon gets ~245-460 GFLOPS per watt. Training on MacBooks costs almost nothing in electricity compared to cloud GPUs. And there are hundreds of millions of Apple Silicon Macs in the world. The math: Traditional DDP: 1 sync per step = 50 GB/s required AirTrain (DiLoCo): 1 sync per 500 steps = 0.1 GB/s required Wi-Fi handles 0.1 GB/s. That's it. That's the breakthrough. They even built a community platform at with live session browsing, checkpoint sharing, and a contributor leaderboard. Training a 124M parameter GPT-2? Instead of renting cloud GPUs at $3/hr, pool three MacBooks in a coffee shop and train for free. MIT licensed. Built in Python. 1 contributor. Early stage but the idea is insane. 100% Open Source. (Link in the comments)
0:44

Cloud GPU training is a scam. A single M4 MacBook does 2.9 TFLOPS. Seven friends with MacBooks match an NVIDIA A100. Alexander Hayes just open-sourced a tool that makes this work over Wi-Fi. It's called AirTrain. Here's how it works: Traditional distributed training (DDP) syncs gradients after every single step. For a 124M parameter model, that's ~500MB exchanged per step. You need 50 GB/s of sustained bandwidth. Impossible over Wi-Fi. AirTrain uses the DiLoCo algorithm. Each Mac trains independently for 500 steps, then syncs only the difference. One sync per 500 steps instead of one per step. 500x less network communication. Wi-Fi actually works. The entire sync takes ~2 seconds. Here's what makes it wild: → Zero-config discovery. Devices find each other automatically via mDNS/Bonjour. Same protocol as AirDrop. → Fault tolerant. Nodes can join and leave mid-training without killing the run. → Checkpoint relay. Train for a few hours, export a checkpoint, hand it off to someone else to continue. Like a relay race for ML training. → Built on Apple's MLX framework. Native to M1/M2/M3/M4/M5 unified memory. No host-to-device copy overhead. → Local dashboard. Real-time loss curves, peer monitoring, throughput metrics in your browser. Here's the wildest part: An M4 Max with 128GB unified memory can train a 70B parameter model without offloading. An NVIDIA RTX 4090 has 24GB VRAM. Apple Silicon gets ~245-460 GFLOPS per watt. Training on MacBooks costs almost nothing in electricity compared to cloud GPUs. And there are hundreds of millions of Apple Silicon Macs in the world. The math: Traditional DDP: 1 sync per step = 50 GB/s required AirTrain (DiLoCo): 1 sync per 500 steps = 0.1 GB/s required Wi-Fi handles 0.1 GB/s. That's it. That's the breakthrough. They even built a community platform at with live session browsing, checkpoint sharing, and a contributor leaderboard. Training a 124M parameter GPT-2? Instead of renting cloud GPUs at $3/hr, pool three MacBooks in a coffee shop and train for free. MIT licensed. Built in Python. 1 contributor. Early stage but the idea is insane. 100% Open Source. (Link in the comments)

Guri Singh

160,201 görüntüleme • 1 ay önce

This is @jason's statement regarding the SN3 Templar drama on Bittensor today
1:32

This is @jason's statement regarding the SN3 Templar drama on Bittensor today

Jesus Martinez

15,008 görüntüleme • 2 ay önce