Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Successfully deployed Deepseek R1 Distilled 70B (AWQ) across 8x NVIDIA RTX 3080 10G GPUs, achieving 60 tokens/s with full tensor parallelism via PCIe. Total hardware cost: $6,400 This demonstrates that consumer GPUs can deliver substantial ML inference capabilities at a fraction of the cost of datacenter hardware. For perspective,...

tensorblock_aoi's profile picture

TensorBlock

1,601 subscribers

16,219 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & TeknolojiHaberler & PolitikaFinans
LIVE

Anya Rossi• Live Now

Private livecam show

6 Yorum

TensorBlock profil fotoğrafı
TensorBlock1 yıl önce

For more detailed discussion and setup configurations, check out our Reddit post here:

The Information profil fotoğrafı
The Information1 yıl önce

Elon Musk claims to have finished a 100,000-strong H100 cluster in four months. How likely is that?

Anti-Federalist Caesar profil fotoğrafı
Anti-Federalist Caesar1 yıl önce

@nvidia cc @__tinygrad__

אלף profil fotoğrafı
אלף1 yıl önce

@nvidia Awesome!

Novel Engineer profil fotoğrafı
Novel Engineer1 yıl önce

Why not 3090 - twice the vram

TensorBlock profil fotoğrafı
TensorBlock1 yıl önce

Thanks for the feedback! While 3090s with NVLink would offer higher bandwidth, this experiment focuses on validating distributed inference with consumer GPUs - exploring possibilities for widespread LLM deployment. Excited to share more findings soon.

Benzer Videolar

🚨 I DON'T KNOW WHAT PEOPLE ARE WAITING FOR! This guy just deployed a full GPU compute farm in a data center Not renting cloud instances, not paying per-token, just racks of consumer GPUs on warehouse shelving doing AI inference 24/7 The setup is wild: open-frame rigs with rows of white fans, PCIe riser cables, server-grade PSUs, and a PDU for power distribution. All mounted on red industrial shelving with raised-floor cooling underneath. It looks like a crypto mining op, because it probably was one Here's what's actually happening: > Ex-mining rigs are being redeployed for AI inference at scale > Each rig runs 6-8 GPUs on PCIe risers connected to a single motherboard > Fans pull cold air from the raised floor through the open frames > A PDU on each rack handles power distribution across all rigs > The whole thing runs in a colocation facility with cheap power rates The economics make sense: cloud GPU pricing runs $2-4/hr per card. Own the hardware, colocate it for $0.08-0.12/kWh, and you're printing compute at a fraction of the cost The crypto-to-AI pipeline is real. Same hardware, same facilities, same power contracts, completely different revenue model Bookmark this!
0:53

🚨 I DON'T KNOW WHAT PEOPLE ARE WAITING FOR! This guy just deployed a full GPU compute farm in a data center Not renting cloud instances, not paying per-token, just racks of consumer GPUs on warehouse shelving doing AI inference 24/7 The setup is wild: open-frame rigs with rows of white fans, PCIe riser cables, server-grade PSUs, and a PDU for power distribution. All mounted on red industrial shelving with raised-floor cooling underneath. It looks like a crypto mining op, because it probably was one Here's what's actually happening: > Ex-mining rigs are being redeployed for AI inference at scale > Each rig runs 6-8 GPUs on PCIe risers connected to a single motherboard > Fans pull cold air from the raised floor through the open frames > A PDU on each rack handles power distribution across all rigs > The whole thing runs in a colocation facility with cheap power rates The economics make sense: cloud GPU pricing runs $2-4/hr per card. Own the hardware, colocate it for $0.08-0.12/kWh, and you're printing compute at a fraction of the cost The crypto-to-AI pipeline is real. Same hardware, same facilities, same power contracts, completely different revenue model Bookmark this!

cristal

16,305 görüntüleme • 2 ay önce

"Compute is the currency of the future." "Compute is the new OIL." Great, so how do I invest? You can play with the NVIDIA stock, or spin up an AI-Ready Data Center. Challenges of Spinning up AI-Ready Compute: Cost: Even a small cluster of H100 machines is upward to $10M GPU Scarcity: Unallocated or “underutilized” H100, A100, and RTX4090 GPUs are almost non-existent. Very Specific Physical & Technical Requirements: A data center with the right infrastructure is also very rare. There are only 10 in North America that deploy thousands of GPUs, in the open market. Domain Expertise: Teams of experts with the years of experience required to operate, deploy, and manage AI compute at scale are as rare as the GPUs they deploy. Exabits checks all of these boxes and we are the premier Compute Tokenization Platform. We are backed by REAL COMPUTE used by REAL CUSTOMERS.
0:30

"Compute is the currency of the future." "Compute is the new OIL." Great, so how do I invest? You can play with the NVIDIA stock, or spin up an AI-Ready Data Center. Challenges of Spinning up AI-Ready Compute: Cost: Even a small cluster of H100 machines is upward to $10M GPU Scarcity: Unallocated or “underutilized” H100, A100, and RTX4090 GPUs are almost non-existent. Very Specific Physical & Technical Requirements: A data center with the right infrastructure is also very rare. There are only 10 in North America that deploy thousands of GPUs, in the open market. Domain Expertise: Teams of experts with the years of experience required to operate, deploy, and manage AI compute at scale are as rare as the GPUs they deploy. Exabits checks all of these boxes and we are the premier Compute Tokenization Platform. We are backed by REAL COMPUTE used by REAL CUSTOMERS.

Exabits

13,896 görüntüleme • 2 yıl önce

What if we could connect all the dark compute across the globe to build the world's biggest AI data center? Most of the compute in the world is dark: phones, laptops, Tesla's, PS5's, TV's. These devices have powerful GPUs but are mostly sitting idle. Today, EXO Labs is announcing the first step in activating all of this dark compute for AI workloads: evML, a distributed computing protocol that enforces honest behavior through hardware security and spot-checks with just 5% performance overhead. Version 0.0.1 of evML is compatible with Apple, Samsung and LG devices. Our preliminary report on evML is now available (link below).
0:24

What if we could connect all the dark compute across the globe to build the world's biggest AI data center? Most of the compute in the world is dark: phones, laptops, Tesla's, PS5's, TV's. These devices have powerful GPUs but are mostly sitting idle. Today, EXO Labs is announcing the first step in activating all of this dark compute for AI workloads: evML, a distributed computing protocol that enforces honest behavior through hardware security and spot-checks with just 5% performance overhead. Version 0.0.1 of evML is compatible with Apple, Samsung and LG devices. Our preliminary report on evML is now available (link below).

EXO Labs

137,958 görüntüleme • 1 yıl önce

Groq is serving the fastest responses I've ever seen. We're talking almost 500 T/s! I did some research on how they're able to do it. Turns out they developed their own hardware that utilize LPUs instead of GPUs. Here's the skinny: Groq created a novel processing unit known as the Tensor Streaming Processor (TSP) which they categorize as a Linear Processor Unit (LPU). Unlike traditional GPUs that are parallel processors with hundreds of cores designed for graphics rendering, LPUs are architected to deliver deterministic performance for AI computations. The LPU's architecture is a departure from the SIMD (Single Instruction, Multiple Data) model used by GPUs and favor a more streamlined approach that eliminate the need for complex scheduling hardware. This design allows every clock cycle to be utilized effectively, ensuring consistent latency and throughput. For developers, this means that performance can be precisely predicted and optimized which is critical in real-time AI applications. Energy efficiency is another area where LPUs shine. By reducing the overhead of managing multiple threads and avoiding the underutilization of cores, LPUs can deliver more computations per watt. Groq's innovative chip design allows multiple TSPs to be linked together without the traditional bottlenecks found in GPU clusters making them extremely scalable. This enables linear scaling of performance as more LPUs are added simplifying the hardware requirements for large-scale AI models and making it easier for developers to scale their applications without rearchitecting their systems. So what does this all mean? LPUs could provide a massive improvement compared to GPUs for serving AI applications in the future! If anything it will be great to have alternative high performing hardware since A100s and H100s are so in demand
0:48

Groq is serving the fastest responses I've ever seen. We're talking almost 500 T/s! I did some research on how they're able to do it. Turns out they developed their own hardware that utilize LPUs instead of GPUs. Here's the skinny: Groq created a novel processing unit known as the Tensor Streaming Processor (TSP) which they categorize as a Linear Processor Unit (LPU). Unlike traditional GPUs that are parallel processors with hundreds of cores designed for graphics rendering, LPUs are architected to deliver deterministic performance for AI computations. The LPU's architecture is a departure from the SIMD (Single Instruction, Multiple Data) model used by GPUs and favor a more streamlined approach that eliminate the need for complex scheduling hardware. This design allows every clock cycle to be utilized effectively, ensuring consistent latency and throughput. For developers, this means that performance can be precisely predicted and optimized which is critical in real-time AI applications. Energy efficiency is another area where LPUs shine. By reducing the overhead of managing multiple threads and avoiding the underutilization of cores, LPUs can deliver more computations per watt. Groq's innovative chip design allows multiple TSPs to be linked together without the traditional bottlenecks found in GPU clusters making them extremely scalable. This enables linear scaling of performance as more LPUs are added simplifying the hardware requirements for large-scale AI models and making it easier for developers to scale their applications without rearchitecting their systems. So what does this all mean? LPUs could provide a massive improvement compared to GPUs for serving AI applications in the future! If anything it will be great to have alternative high performing hardware since A100s and H100s are so in demand

Jay Scambler

318,228 görüntüleme • 2 yıl önce

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:
0:44

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:

Satya Nadella

908,065 görüntüleme • 9 ay önce

BREAKING: SambaNova Announces First Close of $1 Billion Series E at $11B Valuation Led by General Atlantic, with participation from Seligman Ventures, T. Rowe Price, & Capital Group SambaNova CEO Rodrigo Liang (Rodrigo Liang) says AI is entering a new era of "premium inference" "The world of inference is running into an economics problem." Enter: new SambaNova "SM50" "A cloud-scale RDU that allows us to deliver premium inference at significantly higher performance than you can get with GPUs, at a fraction of the cost & a fraction of the power." Why does that matter? "When you're running faster with the high-accuracy models, you can charge more at a lower cost, allowing our service providers to generate more margin." Liang says the shift is just beginning: "The world's moving into inference." And after today's funding announcement: "We're using this capital to accelerate our supply chain and make sure we can deliver the racks to customers worldwide." Full interview from Paris coming soon.
2:14

BREAKING: SambaNova Announces First Close of $1 Billion Series E at $11B Valuation Led by General Atlantic, with participation from Seligman Ventures, T. Rowe Price, & Capital Group SambaNova CEO Rodrigo Liang (Rodrigo Liang) says AI is entering a new era of "premium inference" "The world of inference is running into an economics problem." Enter: new SambaNova "SM50" "A cloud-scale RDU that allows us to deliver premium inference at significantly higher performance than you can get with GPUs, at a fraction of the cost & a fraction of the power." Why does that matter? "When you're running faster with the high-accuracy models, you can charge more at a lower cost, allowing our service providers to generate more margin." Liang says the shift is just beginning: "The world's moving into inference." And after today's funding announcement: "We're using this capital to accelerate our supply chain and make sure we can deliver the racks to customers worldwide." Full interview from Paris coming soon.

Molly O’Shea

14,162 görüntüleme • 1 ay önce

If intelligence is the log of compute… it starts with a lot of compute! And that’s why we’re scaling our GPU fleet faster than anyone else. Just last year, we added over 2 gigawatts of new capacity – roughly the output of 2 nuclear power plants. And today we’re going further, announcing the world's most powerful AI datacenter, located in southeastern Wisconsin. Fairwater is a seamless cluster of hundreds of thousands of NVIDIA GB200s, connected by enough fiber to circle the Earth 4.5 times. It will deliver 10x the performance of the world’s fastest supercomputer today, enabling AI training and inference workloads at a level never before seen. For AI training workloads, you need compute at exponential scale. That’s why we designed the datacenter, GPU fleet, and network together as one integrated system. This ensures a single job can run from day 1 at exponential scale across thousands of GPUs. Fairwater uses a liquid-cooled closed-loop system for cooling GPUs that requires zero water for operations after construction. And we’re matching all of the energy that is consumed with renewable sources. And of course, it is just one of several similar sites we’re lighting up across our 70+ regions. We have multiple identical Fairwater datacenters under construction in other locations across the US, in addition to our AI infrastructure already deployed in over 100 datacenters around the world, powering model training, test-time compute, RL tuning, and real-time inference at global scale. Too often during times like this, people go with the current and only later wonder, how did we get here? With Fairwater, we're charting a new path: doing the hard engineering work, bringing compute, network, and storage into one highly scaled cluster, and designing closed-loop energy systems to meet real-world computing needs. And partnering with local communities to ensure it's thoughtfully done in a way that is sustainable, creates new jobs, and expands opportunity. We are thrilled to see this take hold in Wisconsin, and we are just getting started.
0:21

If intelligence is the log of compute… it starts with a lot of compute! And that’s why we’re scaling our GPU fleet faster than anyone else. Just last year, we added over 2 gigawatts of new capacity – roughly the output of 2 nuclear power plants. And today we’re going further, announcing the world's most powerful AI datacenter, located in southeastern Wisconsin. Fairwater is a seamless cluster of hundreds of thousands of NVIDIA GB200s, connected by enough fiber to circle the Earth 4.5 times. It will deliver 10x the performance of the world’s fastest supercomputer today, enabling AI training and inference workloads at a level never before seen. For AI training workloads, you need compute at exponential scale. That’s why we designed the datacenter, GPU fleet, and network together as one integrated system. This ensures a single job can run from day 1 at exponential scale across thousands of GPUs. Fairwater uses a liquid-cooled closed-loop system for cooling GPUs that requires zero water for operations after construction. And we’re matching all of the energy that is consumed with renewable sources. And of course, it is just one of several similar sites we’re lighting up across our 70+ regions. We have multiple identical Fairwater datacenters under construction in other locations across the US, in addition to our AI infrastructure already deployed in over 100 datacenters around the world, powering model training, test-time compute, RL tuning, and real-time inference at global scale. Too often during times like this, people go with the current and only later wonder, how did we get here? With Fairwater, we're charting a new path: doing the hard engineering work, bringing compute, network, and storage into one highly scaled cluster, and designing closed-loop energy systems to meet real-world computing needs. And partnering with local communities to ensure it's thoughtfully done in a way that is sustainable, creates new jobs, and expands opportunity. We are thrilled to see this take hold in Wisconsin, and we are just getting started.

Satya Nadella

2,024,356 görüntüleme • 11 ay önce

🎙️ Want a full breakdown on why cost per token is a key metric for AI infrastructure TCO? We’re operating in a power-constrained world. Demand for intelligence is scaling faster than infrastructure can keep up—so metrics like cost per GPU hour or FLOPs per dollar don’t tell the full story. When it comes to TCO, Cost per Token is the metric that captures end-to-end performance across GPUs, CPUs, networking, software, and ecosystem making it the key driver of real profitability and scalability in AI. If you’re not measuring efficiency at the token level, you’re missing what actually drives cost. Watch the deep-dive conversation with Dr. Gerro Prinsloo , Nader Khalil and Carter Abdallah, live on the NVIDIA YouTube channel.
1:05

🎙️ Want a full breakdown on why cost per token is a key metric for AI infrastructure TCO? We’re operating in a power-constrained world. Demand for intelligence is scaling faster than infrastructure can keep up—so metrics like cost per GPU hour or FLOPs per dollar don’t tell the full story. When it comes to TCO, Cost per Token is the metric that captures end-to-end performance across GPUs, CPUs, networking, software, and ecosystem making it the key driver of real profitability and scalability in AI. If you’re not measuring efficiency at the token level, you’re missing what actually drives cost. Watch the deep-dive conversation with Dr. Gerro Prinsloo , Nader Khalil and Carter Abdallah, live on the NVIDIA YouTube channel.

NVIDIA

21,970 görüntüleme • 4 ay önce

Introducing Phoenix Deep Research, a vertical AI agent for in-depth, multi-step research and real-time knowledge. Phoenix Deep Research is blazingly fast with average response output time of 30 to 90 seconds, much faster than industry averages and is built on top of an accelerated Deepseek R1 utilizing SkyNet High Performance Cluster consisting of the newest Huawei Ascend and Nvidia A100, and A10 GPUs. Phoenix DR combines the most cutting-edge open source LLM, industry-specific agentic workflows, real-time AI search, and novel decentralized compute infrastructure into a differentiated and versatile tool previously unseen in the decentralized AI space. Phoenix DR will be open for Beta whitelist and testing early next week for community users and partners. Stay tuned. #Phoenix $PHB #DeepResearch
1:42

Introducing Phoenix Deep Research, a vertical AI agent for in-depth, multi-step research and real-time knowledge. Phoenix Deep Research is blazingly fast with average response output time of 30 to 90 seconds, much faster than industry averages and is built on top of an accelerated Deepseek R1 utilizing SkyNet High Performance Cluster consisting of the newest Huawei Ascend and Nvidia A100, and A10 GPUs. Phoenix DR combines the most cutting-edge open source LLM, industry-specific agentic workflows, real-time AI search, and novel decentralized compute infrastructure into a differentiated and versatile tool previously unseen in the decentralized AI space. Phoenix DR will be open for Beta whitelist and testing early next week for community users and partners. Stay tuned. #Phoenix $PHB #DeepResearch

Phoenix AI Labs

21,303 görüntüleme • 1 yıl önce

Building AI infrastructure at yotta-scale requires more than raw performance; it demands an open, modular rack design that can evolve across product generations, combining leadership compute engines with high-speed networking to connect thousands of accelerators into a single, unified system. Take a look at the Helios. Our rack-scale AI platform powered by MI455X GPUs and EPYC “Venice” CPUs and designed for trillion-parameter training.
0:39

Building AI infrastructure at yotta-scale requires more than raw performance; it demands an open, modular rack design that can evolve across product generations, combining leadership compute engines with high-speed networking to connect thousands of accelerators into a single, unified system. Take a look at the Helios. Our rack-scale AI platform powered by MI455X GPUs and EPYC “Venice” CPUs and designed for trillion-parameter training.

AMD

20,832 görüntüleme • 7 ay önce

$AMD| $META is using $GOOGL to negotiate 🧵 The Ironwood pod is 5.1–10x more expensive annually ($148.3 million ÷ $14.87–$29.04 million) and 5.1–10x more expensive monthly ($12.36 million ÷ $1.24–$2.42 million) than renting 15 MI450 racks for equivalent compute. The rapidly evolving landscape of artificial intelligence infrastructure presents a complex interplay of technological innovation, market dynamics, and strategic maneuvering among major players. Recent leaked information suggesting that Meta Platforms ($META) might work with Google's Tensor Processing Unit (TPU) in 2027 has sparked speculation about its true intent. This leak is likely a strategic move by Meta to negotiate more favorable terms with AMD , leveraging the competitive dynamics of the AI hardware market to optimize its substantial investment in AI infrastructure. By examining the key elements of this scenario Meta's investment strategy, the comparative advantages of AMD's MI450 and Google's Ironwood TPU, and the broader market context; we can discern the potential beneficiaries and the strategic implications of this information. Meta's aggressive pursuit of AI capabilities is underscored by its planned expenditure of $66-72 billion on AI infrastructure in 2025, with expectations to escalate significantly in 2026. This investment is part of a broader strategy to build "titan clusters" like Prometheus, which are projected to reach 1 gigawatt of compute power by 2026. Such a scale of investment reflects Meta's recognition of the critical role that AI will play in its future growth, particularly in enhancing its social media platforms and developing new AI-driven applications. However, the financial burden of this infrastructure buildout necessitates a careful consideration of cost-effectiveness and scalability, which brings us to the leaked information about potential collaboration with Google's Ironwood TPU. Google's Ironwood TPU, introduced as the seventh-generation ASIC optimized for TensorFlow-based inference, represents a high-cost, cloud-locked solution priced at $445 million per pod (9,216 chips) over three years. This model, while offering significant performance gains and power efficiency, is tailored for pod-scale deployment and integrated with Google's cloud services, limiting flexibility and increasing costs for customers. In contrast, AMD's MI450 GPU, priced at $30,000–$40,000 per unit, provides a modular, open ROCm ecosystem that delivers comparable compute capacity at a fraction of the cost. Renting 15 MI450 racks could achieve similar 42+ exaFLOPS inference compute at 5–10x lower cost than renting a single Ironwood pod, underscoring AMD's competitive edge in terms of total cost of ownership (TCO). The leaked information about Meta's potential TPU deployment in 2027, therefore, can be interpreted as a negotiating tactic rather than a definitive shift in strategy. By signaling interest in Google's solution, Meta may be attempting to pressure AMD into offering more favorable terms/prices for 5-10GW. This tactic aligns with Meta's broader goal to finance most of its AI spend internally while exploring partnerships that can reduce costs and enhance flexibility. The post's emphasis on MI450's TCO advantage and its partnerships with major players like OpenAI, Microsoft, and Meta itself suggests that AMD is a critical component of Meta's AI infrastructure strategy. The threat of working with Google's TPU could prompt AMD to reassess its pricing, provide additional support, or offer incentives to retain Meta as a customer, thereby securing or expanding its market share. From a logical standpoint, Meta stands to benefit the most from this strategy. As a major buyer in a high-stakes market projected to surpass $1 trillion in annual spending by 2030, Meta's negotiating power is significant. The leaked information could lead to substantial cost savings on its $66-72 billion investment, enhancing its financial flexibility and allowing for further investment in AI capabilities. Moreover, this tactic reinforces Meta's position as a leader in the AI infrastructure race, potentially attracting more external financing for its data center projects and strengthening its competitive stance against other hyperscalers like Amazon and Microsoft. AMD could also benefit from this scenario. The negotiation pressure might lead to small short-term concessions, but it could also solidify long-term partnerships with Meta, ensuring continued demand for MI450 and other AI hardware solutions. Initially Meta's 42% allocation to AMD MI300X and its partnerships with Oracle, Dell, and HP indicates a deep integration of AMD's technology into Meta's infrastructure, which could be leveraged to maintain this relationship. For AMD, retaining Meta as a large key customer is crucial to capturing a larger share of the rapidly growing data center infrastructure market, driven by the insatiable demand for AI compute power. Google, on the other hand, faces a more limited benefit from this leaked information. While securing Meta as a customer would reinforce its position in the AI hardware market, the high cost and ecosystem lock-in of the Ironwood TPU might deter Meta from fully committing to this solution. The leaked information could prompt Google to reconsider its pricing or ecosystem strategy to remain competitive, but the immediate impact is likely to be minimal compared to the potential gains for Meta and AMD. Investors and market analysts also stand to benefit from this information, as it provides insights into the competitive dynamics of the AI hardware market. Adjustments in portfolios based on anticipated shifts in market share and profitability could lead to opportunities for those who correctly anticipate outcomes. The negotiation dynamic might introduce volatility, but it also highlights the strategic importance of cost-effective solutions in the AI infrastructure space. Lastly, the leaked information about Meta potentially working with Google's TPU in 2027 is likely a strategic move to negotiate with AMD, leveraging the competitive landscape to optimize its AI infrastructure investment. Meta, as the primary negotiator, stands to gain the most by securing better terms from AMD, reducing costs, and enhancing its financial flexibility. AMD, while initially at risk, could benefit from retaining a key customer and solidifying its market position. Google faces limited immediate benefits but may need to adapt its strategy to remain competitive. This scenario underscores the complex interplay of technology, market dynamics, and strategic maneuvering in the AI hardware market, where cost-effectiveness and scalability are paramount. As the data center infrastructure market continues to grow, the outcomes of such negotiations will shape the future of AI development and deployment.
1:12

$AMD| $META is using $GOOGL to negotiate 🧵 The Ironwood pod is 5.1–10x more expensive annually ($148.3 million ÷ $14.87–$29.04 million) and 5.1–10x more expensive monthly ($12.36 million ÷ $1.24–$2.42 million) than renting 15 MI450 racks for equivalent compute. The rapidly evolving landscape of artificial intelligence infrastructure presents a complex interplay of technological innovation, market dynamics, and strategic maneuvering among major players. Recent leaked information suggesting that Meta Platforms ($META) might work with Google's Tensor Processing Unit (TPU) in 2027 has sparked speculation about its true intent. This leak is likely a strategic move by Meta to negotiate more favorable terms with AMD , leveraging the competitive dynamics of the AI hardware market to optimize its substantial investment in AI infrastructure. By examining the key elements of this scenario Meta's investment strategy, the comparative advantages of AMD's MI450 and Google's Ironwood TPU, and the broader market context; we can discern the potential beneficiaries and the strategic implications of this information. Meta's aggressive pursuit of AI capabilities is underscored by its planned expenditure of $66-72 billion on AI infrastructure in 2025, with expectations to escalate significantly in 2026. This investment is part of a broader strategy to build "titan clusters" like Prometheus, which are projected to reach 1 gigawatt of compute power by 2026. Such a scale of investment reflects Meta's recognition of the critical role that AI will play in its future growth, particularly in enhancing its social media platforms and developing new AI-driven applications. However, the financial burden of this infrastructure buildout necessitates a careful consideration of cost-effectiveness and scalability, which brings us to the leaked information about potential collaboration with Google's Ironwood TPU. Google's Ironwood TPU, introduced as the seventh-generation ASIC optimized for TensorFlow-based inference, represents a high-cost, cloud-locked solution priced at $445 million per pod (9,216 chips) over three years. This model, while offering significant performance gains and power efficiency, is tailored for pod-scale deployment and integrated with Google's cloud services, limiting flexibility and increasing costs for customers. In contrast, AMD's MI450 GPU, priced at $30,000–$40,000 per unit, provides a modular, open ROCm ecosystem that delivers comparable compute capacity at a fraction of the cost. Renting 15 MI450 racks could achieve similar 42+ exaFLOPS inference compute at 5–10x lower cost than renting a single Ironwood pod, underscoring AMD's competitive edge in terms of total cost of ownership (TCO). The leaked information about Meta's potential TPU deployment in 2027, therefore, can be interpreted as a negotiating tactic rather than a definitive shift in strategy. By signaling interest in Google's solution, Meta may be attempting to pressure AMD into offering more favorable terms/prices for 5-10GW. This tactic aligns with Meta's broader goal to finance most of its AI spend internally while exploring partnerships that can reduce costs and enhance flexibility. The post's emphasis on MI450's TCO advantage and its partnerships with major players like OpenAI, Microsoft, and Meta itself suggests that AMD is a critical component of Meta's AI infrastructure strategy. The threat of working with Google's TPU could prompt AMD to reassess its pricing, provide additional support, or offer incentives to retain Meta as a customer, thereby securing or expanding its market share. From a logical standpoint, Meta stands to benefit the most from this strategy. As a major buyer in a high-stakes market projected to surpass $1 trillion in annual spending by 2030, Meta's negotiating power is significant. The leaked information could lead to substantial cost savings on its $66-72 billion investment, enhancing its financial flexibility and allowing for further investment in AI capabilities. Moreover, this tactic reinforces Meta's position as a leader in the AI infrastructure race, potentially attracting more external financing for its data center projects and strengthening its competitive stance against other hyperscalers like Amazon and Microsoft. AMD could also benefit from this scenario. The negotiation pressure might lead to small short-term concessions, but it could also solidify long-term partnerships with Meta, ensuring continued demand for MI450 and other AI hardware solutions. Initially Meta's 42% allocation to AMD MI300X and its partnerships with Oracle, Dell, and HP indicates a deep integration of AMD's technology into Meta's infrastructure, which could be leveraged to maintain this relationship. For AMD, retaining Meta as a large key customer is crucial to capturing a larger share of the rapidly growing data center infrastructure market, driven by the insatiable demand for AI compute power. Google, on the other hand, faces a more limited benefit from this leaked information. While securing Meta as a customer would reinforce its position in the AI hardware market, the high cost and ecosystem lock-in of the Ironwood TPU might deter Meta from fully committing to this solution. The leaked information could prompt Google to reconsider its pricing or ecosystem strategy to remain competitive, but the immediate impact is likely to be minimal compared to the potential gains for Meta and AMD. Investors and market analysts also stand to benefit from this information, as it provides insights into the competitive dynamics of the AI hardware market. Adjustments in portfolios based on anticipated shifts in market share and profitability could lead to opportunities for those who correctly anticipate outcomes. The negotiation dynamic might introduce volatility, but it also highlights the strategic importance of cost-effective solutions in the AI infrastructure space. Lastly, the leaked information about Meta potentially working with Google's TPU in 2027 is likely a strategic move to negotiate with AMD, leveraging the competitive landscape to optimize its AI infrastructure investment. Meta, as the primary negotiator, stands to gain the most by securing better terms from AMD, reducing costs, and enhancing its financial flexibility. AMD, while initially at risk, could benefit from retaining a key customer and solidifying its market position. Google faces limited immediate benefits but may need to adapt its strategy to remain competitive. This scenario underscores the complex interplay of technology, market dynamics, and strategic maneuvering in the AI hardware market, where cost-effectiveness and scalability are paramount. As the data center infrastructure market continues to grow, the outcomes of such negotiations will shape the future of AI development and deployment.

Mike

182,273 görüntüleme • 9 ay önce

NVIDIA JUST SHOWED HOW A $4,699 BOX CAN KILL A $200 MONTHLY AI BILL AND RUN 200B MODELS FROM YOUR DESK. 00:02 NVIDIA delivers the DGX Spark to developers, researchers, and AI teams, turning what used to require rented datacenter GPUs into a box small enough to hold in one hand. inside is 128GB of unified memory, 4TB of storage, and up to 1 PFLOP of AI compute, enough to run 70B models comfortably and push into the 200B class locally. that means no per-token API bill, no files leaving your machine, and no waiting for someone else’s server every time an agent reads documents, writes code, or runs an automation. two DGX Sparks can even be linked together for models around 400B parameters, while a single unit costs roughly $4,000 to $4,700 instead of renting multi-GPU cloud infrastructure every month. the full local AI ladder starts at $0 with the computer you already own, but NVIDIA just showed what the final rung looks like: a personal AI supercomputer sitting directly on your desk.
0:24

NVIDIA JUST SHOWED HOW A $4,699 BOX CAN KILL A $200 MONTHLY AI BILL AND RUN 200B MODELS FROM YOUR DESK. 00:02 NVIDIA delivers the DGX Spark to developers, researchers, and AI teams, turning what used to require rented datacenter GPUs into a box small enough to hold in one hand. inside is 128GB of unified memory, 4TB of storage, and up to 1 PFLOP of AI compute, enough to run 70B models comfortably and push into the 200B class locally. that means no per-token API bill, no files leaving your machine, and no waiting for someone else’s server every time an agent reads documents, writes code, or runs an automation. two DGX Sparks can even be linked together for models around 400B parameters, while a single unit costs roughly $4,000 to $4,700 instead of renting multi-GPU cloud infrastructure every month. the full local AI ladder starts at $0 with the computer you already own, but NVIDIA just showed what the final rung looks like: a personal AI supercomputer sitting directly on your desk.

Gipp 🦅

11,334 görüntüleme • 1 ay önce

A DEAD 2021 MINING RIG CAN NOW BECOME A 96GB AI FACTORY THAT REPLACES $4,944 A YEAR IN SUBSCRIPTIONS 00:26 he shows the full build: four RTX 3090s, separate power supplies, and enough cables to keep the entire rig running at full load. This machine was built for crypto. Now the same hardware can run local LLMs, generate images, transcribe videos, analyze files, and power AI agents. Each RTX 3090 has 24GB of VRAM. Four cards give the rig 96GB, enough to handle models that normally need an expensive workstation or rented server. Used 3090s cost around $650 to $750 each. The full stack is roughly $2,800, while a pile of AI subscriptions can quietly reach $412 every month. Seven months of subscriptions already costs more than the GPUs. After that, the rig keeps working while the software bill keeps draining another $4,944 every year. Leave it running overnight and it can organize notes, process saved videos, prepare drafts, monitor prices, and summarize documents without constant input. The crypto boom built thousands of these machines. AI is giving them a second life, turning old mining rigs into private factories for code, research, and content. Bookmark & Watch it run ↓
0:33

A DEAD 2021 MINING RIG CAN NOW BECOME A 96GB AI FACTORY THAT REPLACES $4,944 A YEAR IN SUBSCRIPTIONS 00:26 he shows the full build: four RTX 3090s, separate power supplies, and enough cables to keep the entire rig running at full load. This machine was built for crypto. Now the same hardware can run local LLMs, generate images, transcribe videos, analyze files, and power AI agents. Each RTX 3090 has 24GB of VRAM. Four cards give the rig 96GB, enough to handle models that normally need an expensive workstation or rented server. Used 3090s cost around $650 to $750 each. The full stack is roughly $2,800, while a pile of AI subscriptions can quietly reach $412 every month. Seven months of subscriptions already costs more than the GPUs. After that, the rig keeps working while the software bill keeps draining another $4,944 every year. Leave it running overnight and it can organize notes, process saved videos, prepare drafts, monitor prices, and summarize documents without constant input. The crypto boom built thousands of these machines. AI is giving them a second life, turning old mining rigs into private factories for code, research, and content. Bookmark & Watch it run ↓

slash1s

12,286 görüntüleme • 2 ay önce

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰
0:32

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 görüntüleme • 1 yıl önce

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.
0:20

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,604 görüntüleme • 1 yıl önce

Unsloth AI and NVIDIA are Revolutionizing Local LLM Fine-Tuning: From RTX Desktops to DGX Spark Fine-tune popular AI models faster with Unsloth on NVIDIA RTX AI PCs such as GeForce RTX desktops and laptops to RTX PRO workstations and the new DGX Spark to build personalized assistants for coding, creative work, and complex agentic workflows. The landscape of modern AI is shifting. We are moving away from a total reliance on massive, generalized cloud models and entering the era of local, agentic AI. Whether it is tuning a chatbot to handle hyper-specific product support or building a personal assistant that manages intricate schedules, the potential for generative AI on local hardware is boundless. However, developers face a persistent bottleneck: How do you get a Small Language Model (SLM) to punch above its weight class and respond with high accuracy for specialized tasks? The answer is Fine-Tuning, and the tool of choice is Unsloth. Unsloth provides an easy and high-speed method to customize models. Optimized for efficient, low-memory training on NVIDIA GPUs, Unsloth scales effortlessly from GeForce RTX desktops and laptop all the way to the DGX Spark, the world’s smallest AI supercomputer...... Full analysis: NVIDIA NVIDIA AI NVIDIA AIDev NVIDIAnewsroom Unsloth AI Unsloth
0:40

Unsloth AI and NVIDIA are Revolutionizing Local LLM Fine-Tuning: From RTX Desktops to DGX Spark Fine-tune popular AI models faster with Unsloth on NVIDIA RTX AI PCs such as GeForce RTX desktops and laptops to RTX PRO workstations and the new DGX Spark to build personalized assistants for coding, creative work, and complex agentic workflows. The landscape of modern AI is shifting. We are moving away from a total reliance on massive, generalized cloud models and entering the era of local, agentic AI. Whether it is tuning a chatbot to handle hyper-specific product support or building a personal assistant that manages intricate schedules, the potential for generative AI on local hardware is boundless. However, developers face a persistent bottleneck: How do you get a Small Language Model (SLM) to punch above its weight class and respond with high accuracy for specialized tasks? The answer is Fine-Tuning, and the tool of choice is Unsloth. Unsloth provides an easy and high-speed method to customize models. Optimized for efficient, low-memory training on NVIDIA GPUs, Unsloth scales effortlessly from GeForce RTX desktops and laptop all the way to the DGX Spark, the world’s smallest AI supercomputer...... Full analysis: NVIDIA NVIDIA AI NVIDIA AIDev NVIDIAnewsroom Unsloth AI Unsloth

Marktechpost AI Dev News ⚡

31,551 görüntüleme • 8 ay önce

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.
3:14

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 görüntüleme • 3 ay önce

📢 NEURALAI X 🤩 NeuralAI is extraordinarily excited to reveal that we have started working closely with Akash Network, the world's premier decentralized compute marketplace! ✔️ The offerings of align perfectly with the vision and roadmap of NeuralAI, making this collaboration extremely significant for NeuralAI. For a start, NeuralAI will run inference on Akash based on our 3D #AI model and prepare TAO miners for reward-yielding subnets. NeuralAI will also leverage Akash's powerful GPUs to enhance our #AI #model training. Currently, provides us with 56 powerful A6000 GPUs, scalable on demand. ✔️ NeuralAI also gets highly competitive pricing from Akash Network, significantly lower than any other compute provider in the world. The lower pricing is possible partially thanks to the crypto-driven incentives within Akash Network, making it an excellent opportunity for NeuralAI to leverage. The synergy with Akash Network is a big leap forward, fueling our mission of becoming the #1 AI-driven 3D asset generator subnet on Bittensor. ✔️ We look forward to our close work with , we are certain with their support we will be able to secure a unique edge in the market! Stay tuned for exciting updates and developments.
0:29

📢 NEURALAI X 🤩 NeuralAI is extraordinarily excited to reveal that we have started working closely with Akash Network, the world's premier decentralized compute marketplace! ✔️ The offerings of align perfectly with the vision and roadmap of NeuralAI, making this collaboration extremely significant for NeuralAI. For a start, NeuralAI will run inference on Akash based on our 3D #AI model and prepare TAO miners for reward-yielding subnets. NeuralAI will also leverage Akash's powerful GPUs to enhance our #AI #model training. Currently, provides us with 56 powerful A6000 GPUs, scalable on demand. ✔️ NeuralAI also gets highly competitive pricing from Akash Network, significantly lower than any other compute provider in the world. The lower pricing is possible partially thanks to the crypto-driven incentives within Akash Network, making it an excellent opportunity for NeuralAI to leverage. The synergy with Akash Network is a big leap forward, fueling our mission of becoming the #1 AI-driven 3D asset generator subnet on Bittensor. ✔️ We look forward to our close work with , we are certain with their support we will be able to secure a unique edge in the market! Stay tuned for exciting updates and developments.

NeuralAI

241,287 görüntüleme • 2 yıl önce

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

131,019 görüntüleme • 1 yıl önce