Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

The largest decentralised LLM pre-training run in history. SN3 templar trained Covenant-72B across 70+ contributors on open internet infrastructure. Now it’s being discussed by Chamath Palihapitiya with NVIDIA CEO Jensen Huang. Distributed, open-weight model training on Bittensor is getting started.

120,976 görüntüleme • 4 ay önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

For anyone trying to understand Bittensor from first principles, this lecture is a useful place to start. Presented by Bittensor co-founder const. Learn Bittensor > Start with Bitcoin, distributed systems, incentives, > How Bitcoin leads to Bittensor Subnets coordinating AI infrastructure. Topics: // Start - Bitcoin as more than a digital currency // Risks of AI centralization + closed systems // "The incentive computer" // How Bittensor subnets work (mining, validating) // How distributed AI infrastructure could scale globally // Impact on students, builders & future founders Recorded at the National University of Singapore Computer Science Club. NUS Computing Chapters - Bitcoin, AI, and Bittensor - Bitcoin history and decentralization - AI changes how engineers work - The danger of centralized AI power - Why most crypto visions fail - Bitcoin as the world’s largest compute network - Bitcoin as a market for compute - The idea of an “incentive computer” - Bitcoin compared to Bittensor - Classroom example of decentralized scoring - A simple subnet example - SN62 :: Ridges AI | SN62 SWE agents - SN3 templar :: Distributed AI Training - SN52 lium.io :: GPU rentals on Bittensor 128 subnets, some examples Why this matters for the future of work Q&A Subnet examples mentioned @ SN64 - Serverless + TEE Compute :: Chutes SN8 - Prop firm Vanta Trading SN52 - AutoML :: Gradients SN62 - SWE agents :: Ridges AI | SN62 SN51 - Compute / GPU rental lium.io SN4 - TEE compute for enterprise :: Targon SN3 - 72B Distributed Training run :: templar SN41 - Prediction markets :: Almanac SN44 - Computer Vision Score - Subnet 44 SN68 - Drug discovery :: METANOVA SN18 - Weather Forecasting Zeus | SN 18 SN50 - Bitcoin prediction data :: Synthdata SN61 - Quantum computing :: qBitTensor Labs SN14 - Bitcoin mining pool :: TaoHash SN34 - Perp Dex :: 0xMarkets SN17 - 3D model generation :: 404 SN33 - Data analytics :: ReadyAI SN19 - [Since relaunched] RPC infrastructure :

Openτensor Foundaτion

1,173,117 görüntüleme • 4 ay önce

Covenant Labs just did a 90-minute AMA breaking down their 3 Bittensor subnets. templar. basilica. grail. Pre-training, compute, and post-training under one roof. Most people missed it. Here's everything they said. Covenant is building what they call the "end to end intelligence continuum." Three subnets. Three layers of the AI stack. All permissionless. Templar (SN3) handles decentralized pre-training. Basilica (SN39) handles compute. Grail (SN81) handles RL post-training. Sam Dare, the lead, put it bluntly. Decentralized training is "humanity's last dance." Not about beating OpenAI head to head. About creating optionality. About making it cheap enough for anyone to train models. The gap between academia and frontier labs is growing exponentially. Researchers can't afford to experiment. The actual training run costs 5% of the reported budget. The other 95% is experimentation. If Covenant cracks cheap training, that entire surface area opens up. On Templar specifically: • Hit 39% emission on Bittensor. Highest since Apex was the only subnet on the network • Covenant-72B trained permissionlessly with 70+ contributors on commodity internet • 1.1 trillion tokens processed. No centralized data center • Performance competitive with LLaMA-2-70B On Grail, something flew under the radar. They built Pulse. A weight synchronization method that compresses model updates by 100x. • In RL post-training, only ~1% of weights update per step • Pulse exploits that sparsity. Lossless compression • Prime Intellect's comparable system took 14 minutes to sync a 30B model • Pulse makes decentralized RL training actually feasible at scale • Already used by Cursor The lead researcher on Grail said they've trained on math, code, and GPU kernels. Got 40-60% improvement on benchmarks. Working toward agentic training with 100K+ token context and 30B+ parameter models. On Basilica, the compute subnet: The team was blunt. Just reselling GPU hours is a 5-10% margin game. Traditional compute providers already do that. Their play is value-added services. • "GPU as code." No dashboard. No UI. Agents interact via SDK • Custom scheduler that places workloads across heterogeneous hardware • Verification checks for GPU, CPU, bandwidth, memory, storage, and OS security • Partnerships with providers like Mass Compute for 10-20% below market pricing • Miners compete on useful infrastructure, not just GPU hours Sam then went on a rant about the miner burn debate. His take: Bittensor had to grow up. dTAO introduced investors. The old "miners are God" philosophy doesn't hold. • Subnet owners have a duty to protect token value • Miners are a resource optimization exercise, not a cost reduction exercise • 100% miner emissions on compute subnets = immediate sell pressure • The 41% miner allocation is arbitrary. Different business models need different splits • Fish (who started burns) agreed. Burns usually mean the validation isn't mature enough The bigger point. You can't police burns. Subnets just send to their own keys instead of the burn address. Subnet 28 does exactly that. Sam's position: judge subnets on outcomes, not process. Const has changed the protocol 9-10 times in 2 years. That iteration speed is Bittensor's actual moat. The whole Covenant thesis is playing out in real time. TAO is up 100%+ in a month. Jensen Huang name-dropped the network. Grayscale has an ETF filing. But the real story is three subnets quietly building every layer of decentralized AI.

Jesus Martinez

26,642 görüntüleme • 3 ay önce

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Vaibhav (VB) Srivastav

80,474 görüntüleme • 1 yıl önce

A $10 MILLION DUMP JUST TESTED WHETHER BITTENSOR IS REAL OR NOT. THE NETWORK GAVE ITS ANSWER. Covenant AI walked. The biggest builder team in the ecosystem. The team behind the 72B parameter model that Jensen Huang praised on the All In podcast. They accused leadership of centralization. Said one person controls too much. Said changes were made without process. 37,000 TAO sold. Price crashed 25%. $650 million wiped. And then something happened that most people missed because they were too busy panic selling. The network kept running. Every subnet stayed active. 100+ subnets still live. AI training continued. New models started. Builders kept building. The team that built Covenant 72B left. But the model was trained in a decentralized way across 70+ independent contributors using home GPUs. The milestone belongs to the network, not the team. Grayscale didn't sell. They increased exposure. ETF conversations didn't stop. Jacob Steves proposed locked stake for governance transparency within days. This was BitTensor's first public crisis. The kind that either kills a project or proves it can't be killed. The network answered. It's still standing. $256 right now. If it reclaims $280, this was the healthiest shakeout in the project's history. If it doesn't, the bear case deepens. Either way, you now know something you didn't know last week. BitTensor can take a direct hit from its biggest contributor leaving and keep running without interruption. That's not nothing. That might be everything.

Altcoin Buzz

23,317 görüntüleme • 3 ay önce