Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Transformer architecture can be difficult to conceptualize. This clip makes it considerably more accessible. At the International Semiconductor Industry Group (ISIG) Executive Summit, Marvell President of the Data Center Group Sandeep Bharathi walks through how large language models process information using a restaurant analogy, from the moment an order... show more

Marvell Technology

15,252 subscribers

11,026 просмотров • 1 месяц назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Chris Camillo explains how Bloom Energy is solving AI data center energy usage problems… “ $BE is one of the only companies in the world that has a method to essentially bring almost an infinite amount of energy to these data centers." And we won't get into all the technology, but they are now working with hyperscalers and they are a resource for energy that will allow a data center to get ramped up quicker. "So, even though it costs a little bit more money today to put an energy cell from Bloom, an energy box from Bloom, which are energy cells, it essentially takes things like natural gas, but instead of burning the natural gas, there is a chemical process that turns that natural gas into energy for the data center."

Chris Camillo explains how Bloom Energy is solving AI data center energy usage problems… “ $BE is one of the only companies in the world that has a method to essentially bring almost an infinite amount of energy to these data centers." And we won't get into all the technology, but they are now working with hyperscalers and they are a resource for energy that will allow a data center to get ramped up quicker. "So, even though it costs a little bit more money today to put an energy cell from Bloom, an energy box from Bloom, which are energy cells, it essentially takes things like natural gas, but instead of burning the natural gas, there is a chemical process that turns that natural gas into energy for the data center."

The Iced Coffee Hour

274,893 просмотров • 2 месяцев назад

A massive 350,000 square foot data center in Massachusetts is nicknamed “The Dungeon” by residents The data center promised it would provide lots of new jobs to the community. Only about 12 people work there…. The data center is causing constant visible pollution and a lot of noise to residents It’s using about 120,000 gallons of water per day and using as much power as 40,000 American homes Residents were lied to Get this, the data center is buying nearby homes and properties to expand….

A massive 350,000 square foot data center in Massachusetts is nicknamed “The Dungeon” by residents The data center promised it would provide lots of new jobs to the community. Only about 12 people work there…. The data center is causing constant visible pollution and a lot of noise to residents It’s using about 120,000 gallons of water per day and using as much power as 40,000 American homes Residents were lied to Get this, the data center is buying nearby homes and properties to expand….

Wall Street Apes

105,569 просмотров • 17 дней назад

In the next 2-3 years, the skills you need to build an ETL process will fundamentally differ from anything we've done in the last decade. The EXTRACT and LOAD steps of an ETL pipeline have been solved forever. They are boring, and I suspect they will continue to be boring. But the TRANSFORM step is where the money is at (and where innovation will continue happening!) An ETL pipeline that can use Large Language Models effectively to transform data is a whole new ballgame. I recorded a video to talk about this. And more importantly, I talk about how you can build ETL pipelines to transform UNSTRUCTURED data (the kind of data that's really hard to handle).

In the next 2-3 years, the skills you need to build an ETL process will fundamentally differ from anything we've done in the last decade. The EXTRACT and LOAD steps of an ETL pipeline have been solved forever. They are boring, and I suspect they will continue to be boring. But the TRANSFORM step is where the money is at (and where innovation will continue happening!) An ETL pipeline that can use Large Language Models effectively to transform data is a whole new ballgame. I recorded a video to talk about this. And more importantly, I talk about how you can build ETL pipelines to transform UNSTRUCTURED data (the kind of data that's really hard to handle).

Santiago

56,412 просмотров • 1 год назад

Taiwan sits at the center of one of the world's biggest geopolitical disputes. 🇹🇼🇺🇸🇨🇳 Taiwan is home to the world's most advanced semiconductor industry, producing cutting-edge chips that power smartphones, AI, data centers, cars, and advanced military equipment. China says Taiwan is part of its territory and has not ruled out force to bring it under control.😯😯 The United States supports Taiwan's ability to defend itself while trying to avoid direct conflict with China!! So how did this dispute begin, and why does it matter to the world?🤔 The story goes back to the Chinese Civil War... Taiwan is more than an island. It sits at the center of global technology. Subscribe to Donald J Trump Posts TruthSocial for more global affairs videos like these.

Taiwan sits at the center of one of the world's biggest geopolitical disputes. 🇹🇼🇺🇸🇨🇳 Taiwan is home to the world's most advanced semiconductor industry, producing cutting-edge chips that power smartphones, AI, data centers, cars, and advanced military equipment. China says Taiwan is part of its territory and has not ruled out force to bring it under control.😯😯 The United States supports Taiwan's ability to defend itself while trying to avoid direct conflict with China!! So how did this dispute begin, and why does it matter to the world?🤔 The story goes back to the Chinese Civil War... Taiwan is more than an island. It sits at the center of global technology. Subscribe to Donald J Trump Posts TruthSocial for more global affairs videos like these.

Donald J Trump Posts TruthSocial

154,159 просмотров • 4 дней назад

$MU $00060 $00530 $TSM Evolution of the AI Memory Hierarchy: HBM, DRAM, & SRAM Overview and Market Analysis The video outlines the strategic shift of modern computing toward a memory-centric architecture driven by the demands of generative AI. It explains that the industry is moving away from a traditional commodity model to a tightly coupled hierarchy of specialized technologies, where High Bandwidth Memory (HBM) and advanced packaging act as primary bottlenecks for performance. Leading manufacturers like SK hynix, Samsung, and Micron are navigating complex transitions to HBM4 and CXL-based expansion to overcome data movement constraints. Beyond the memory producers, the ecosystem relies on integration enablers like TSMC for packaging and specialized equipment providers for high-precision manufacturing. Ultimately, the analysis suggests that memory intensity and power efficiency have become the decisive factors in the success of gigawatt-scale data centers.

$MU $00060 $00530 $TSM Evolution of the AI Memory Hierarchy: HBM, DRAM, & SRAM Overview and Market Analysis The video outlines the strategic shift of modern computing toward a memory-centric architecture driven by the demands of generative AI. It explains that the industry is moving away from a traditional commodity model to a tightly coupled hierarchy of specialized technologies, where High Bandwidth Memory (HBM) and advanced packaging act as primary bottlenecks for performance. Leading manufacturers like SK hynix, Samsung, and Micron are navigating complex transitions to HBM4 and CXL-based expansion to overcome data movement constraints. Beyond the memory producers, the ecosystem relies on integration enablers like TSMC for packaging and specialized equipment providers for high-precision manufacturing. Ultimately, the analysis suggests that memory intensity and power efficiency have become the decisive factors in the success of gigawatt-scale data centers.

TheValueist

23,901 просмотров • 6 месяцев назад

A Heavy Lift for a Powerful Future! $IREN's multi-disciplinary engineering teams successfully installed the Childress Phase 4 substation transformer over the weekend, bringing the company one step closer to energizing an additional 150MW of data center capacity. David Shaw, Chief Operating Officer, commented: "With unprecedented growth in the data center industry and increasingly constrained supply chains, careful and proactive planning is more critical than ever. The ahead-of-schedule installation of this 138/34.5 kV transformer underscores the strategic importance of IREN’s internal procurement team and strength of our supplier partnerships."

A Heavy Lift for a Powerful Future! $IREN's multi-disciplinary engineering teams successfully installed the Childress Phase 4 substation transformer over the weekend, bringing the company one step closer to energizing an additional 150MW of data center capacity. David Shaw, Chief Operating Officer, commented: "With unprecedented growth in the data center industry and increasingly constrained supply chains, careful and proactive planning is more critical than ever. The ahead-of-schedule installation of this 138/34.5 kV transformer underscores the strategic importance of IREN’s internal procurement team and strength of our supplier partnerships."

IREN

60,064 просмотров • 1 год назад

MiniMax is building with next-gen AGI, offering foundation AI models and revolutionizing content creation with AI-powered video, music, chat products. With Alibaba's one-stop multimodal data platform, MiniMax can save the complexity of handling multimodal data work with the AI-ready data foundation, and boost streamlined innovation on it. Feifei Li, SVP and President of International Business at Alibaba Cloud Intelligence Group, shares how the right tech stack can elevate AI from potential to powerful. 🚀

MiniMax is building with next-gen AGI, offering foundation AI models and revolutionizing content creation with AI-powered video, music, chat products. With Alibaba's one-stop multimodal data platform, MiniMax can save the complexity of handling multimodal data work with the AI-ready data foundation, and boost streamlined innovation on it. Feifei Li, SVP and President of International Business at Alibaba Cloud Intelligence Group, shares how the right tech stack can elevate AI from potential to powerful. 🚀

Alibaba Group

305,831 просмотров • 7 месяцев назад

$NVDA CEO Jensen Huang saying $MRVL can become “the next trillion-dollar company” doesn’t sound crazy when you realize it sits directly inside one of the biggest bottlenecks of the AI buildout: networking. Marvell is trying to become the connective tissue of the AI factory with exposure to custom ASICs, optics, switching, photonics and data movement layer every large-scale AI system needs.

$NVDA CEO Jensen Huang saying $MRVL can become “the next trillion-dollar company” doesn’t sound crazy when you realize it sits directly inside one of the biggest bottlenecks of the AI buildout: networking. Marvell is trying to become the connective tissue of the AI factory with exposure to custom ASICs, optics, switching, photonics and data movement layer every large-scale AI system needs.

Shay Boloor

533,096 просмотров • 1 месяц назад

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

Fireside Alpha

38,370 просмотров • 14 дней назад

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Andrew Ng

253,812 просмотров • 1 год назад

Threadguy explains why Apple will capture ALL of the value from AI “Models have become a commodity, we'll be able to just use the best model at any given time” "The more data AI has access to, the better it is. Everyone already has an iPhone, the next iPhone is just going to launch with some OpenClaw esque assistant utilising the best model plugged into it and its probably going to be the best AI experience you’ve ever had”

Threadguy explains why Apple will capture ALL of the value from AI “Models have become a commodity, we'll be able to just use the best model at any given time” "The more data AI has access to, the better it is. Everyone already has an iPhone, the next iPhone is just going to launch with some OpenClaw esque assistant utilising the best model plugged into it and its probably going to be the best AI experience you’ve ever had”

Jack

24,179 просмотров • 4 месяцев назад

Enrollment is now open for the Data Engineering Professional Certificate! Data engineers are the architects of modern organizations, ensuring data is reliable, accessible, and ready for analytics and machine learning. This professional certificate is tailored to equip you with the critical skills, through frameworks and hands-on practice, to excel in this role. Taught by industry expert Joe Reis, co-author of the best-selling book "Fundamentals of Data Engineering," along with 17 guest instructors from the data field, you will gain expertise to start and further your career in the high-demand field of data engineering. Key focus areas: 🗂️ Data Engineering Lifecycle: Learn the important stages of building an efficient data pipeline that creates business value. 📥 Data Ingestion: Learn how to efficiently gather data from various sources. 💾 Data Storage: Master the techniques for storing data securely and cost-effectively. 🔄 Data Transformation: Understand how to clean, organize, and prepare data for analysis and machine learning. 🏗️ Data Architecture Design: Build robust architectures that support scalable, efficient data workflows. 📊 Serving Data: Ensure that data is available to stakeholders when and where they need it to drive business decisions. Enroll now!

Enrollment is now open for the Data Engineering Professional Certificate! Data engineers are the architects of modern organizations, ensuring data is reliable, accessible, and ready for analytics and machine learning. This professional certificate is tailored to equip you with the critical skills, through frameworks and hands-on practice, to excel in this role. Taught by industry expert Joe Reis, co-author of the best-selling book "Fundamentals of Data Engineering," along with 17 guest instructors from the data field, you will gain expertise to start and further your career in the high-demand field of data engineering. Key focus areas: 🗂️ Data Engineering Lifecycle: Learn the important stages of building an efficient data pipeline that creates business value. 📥 Data Ingestion: Learn how to efficiently gather data from various sources. 💾 Data Storage: Master the techniques for storing data securely and cost-effectively. 🔄 Data Transformation: Understand how to clean, organize, and prepare data for analysis and machine learning. 🏗️ Data Architecture Design: Build robust architectures that support scalable, efficient data workflows. 📊 Serving Data: Ensure that data is available to stakeholders when and where they need it to drive business decisions. Enroll now!

DeepLearning.AI

20,833 просмотров • 1 год назад

IEEE recently recognized the 40th anniversary of the FPGA at an event at AMD in San Jose. When it made its debut four decades ago, the FPGA introduced a new kind of flexibility in hardware that has reshaped the semiconductor industry. The IEEE event honored AMD and the engineers, partners, and customers who made it possible. It also paid tribute to the decades of innovation and collaboration that continue to power innovation across so many applications, from communications, automotive, and industrial systems, to aerospace, data centers, and physical AI systems at the edge.

IEEE recently recognized the 40th anniversary of the FPGA at an event at AMD in San Jose. When it made its debut four decades ago, the FPGA introduced a new kind of flexibility in hardware that has reshaped the semiconductor industry. The IEEE event honored AMD and the engineers, partners, and customers who made it possible. It also paid tribute to the decades of innovation and collaboration that continue to power innovation across so many applications, from communications, automotive, and industrial systems, to aerospace, data centers, and physical AI systems at the edge.

AMD Embedded

11,158 просмотров • 3 месяцев назад

Professor Jones, whose work sits at the center of the climate change modeling apparatus, stated plainly that he would not share his data even if the World Meteorological Organization requested it. His reasoning was that 25 years of work had gone into the research, and he saw no obligation to hand it over to people whose aim was to find something wrong with it. That statement is worth reading slowly, because it is a precise inversion of how science is supposed to work. The whole point of publishing a scientific claim is that other researchers can examine the underlying data, attempt to replicate the result, and either confirm or challenge the conclusion. That is not a hostile process. That is the process. Replication is the mechanism by which a scientific claim earns the right to be called scientific. When a researcher withholds the data behind conclusions that are being used to drive policy affecting hundreds of millions of people, the enterprise has left the domain of science and entered the domain of politics. The 25 years invested in the work is not a justification for withholding it. If anything, the scale of that investment makes independent scrutiny more important, not less. You do not get to claim scientific authority while simultaneously blocking the one process that confers it.

Professor Jones, whose work sits at the center of the climate change modeling apparatus, stated plainly that he would not share his data even if the World Meteorological Organization requested it. His reasoning was that 25 years of work had gone into the research, and he saw no obligation to hand it over to people whose aim was to find something wrong with it. That statement is worth reading slowly, because it is a precise inversion of how science is supposed to work. The whole point of publishing a scientific claim is that other researchers can examine the underlying data, attempt to replicate the result, and either confirm or challenge the conclusion. That is not a hostile process. That is the process. Replication is the mechanism by which a scientific claim earns the right to be called scientific. When a researcher withholds the data behind conclusions that are being used to drive policy affecting hundreds of millions of people, the enterprise has left the domain of science and entered the domain of politics. The 25 years invested in the work is not a justification for withholding it. If anything, the scale of that investment makes independent scrutiny more important, not less. You do not get to claim scientific authority while simultaneously blocking the one process that confers it.

Randall Carlson

25,234 просмотров • 1 месяц назад

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

Karl Mehta

25,716 просмотров • 17 дней назад

In this year, where we are commemorating the 70th Anniversary of the Freedom Charter, we must rededicate ourselves to achieving the vision of the Freedom Charter and the pursuit of the National Democratic Revolution. As set out in Through the Eye of a Needle, to become a leader of the ANC is neither an entitlement nor should it be an easy process attached merely to status. It is an immeasurable honour to become a leader of our movement. #FreedomCharter70 #ANCRenewal

In this year, where we are commemorating the 70th Anniversary of the Freedom Charter, we must rededicate ourselves to achieving the vision of the Freedom Charter and the pursuit of the National Democratic Revolution. As set out in Through the Eye of a Needle, to become a leader of the ANC is neither an entitlement nor should it be an easy process attached merely to status. It is an immeasurable honour to become a leader of our movement. #FreedomCharter70 #ANCRenewal

Cyril Ramaphosa 🇿🇦

20,703 просмотров • 1 год назад

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 просмотров • 1 месяц назад

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 просмотров • 1 месяц назад

China is using AI-driven humanoid robots to improve intelligent automation in the manufacturing industry. This humanoid robot named NAVIAI, from the Zhejiang Humanoid Robot Innovation Center, is currently undergoing training at the Geely Auto Lynk & Co factory. It can evenly apply sealant to designated areas during the glass sealing process of the vehicle body. In the charging detection process at the end of the assembly line, it can adapt to slight deviations during the plugging and unplugging process.

China is using AI-driven humanoid robots to improve intelligent automation in the manufacturing industry. This humanoid robot named NAVIAI, from the Zhejiang Humanoid Robot Innovation Center, is currently undergoing training at the Geely Auto Lynk & Co factory. It can evenly apply sealant to designated areas during the glass sealing process of the vehicle body. In the charging detection process at the end of the assembly line, it can adapt to slight deviations during the plugging and unplugging process.

CyberRobo

19,596 просмотров • 1 год назад