Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Ghostty is getting automatic scrollback compression, resulting in 70 to 90% less physical memory usage. It happens incrementally when idle, so it had no measurable effect on IO throughput. I'm not aware of any other mainstream terminal that does this. Demo video below! The gains let us increase the...

mitchellh's profile picture

Mitchell Hashimoto

213,452 subscribers

149,042 Aufrufe • vor 3 Tagen •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Very stealthy way of dumping LSASS, is done from the kernel, doesn't matter if LSASS is LSA Protected. Once loaded, the rootkit creates a system thread (PsCreateSystemThread) for dumping LSASS, iterates over all processes gets its EPROCESS (PsLookupProcessByProcessId ), then its name (PsGetProcessImageFileName), attaches to LSASS with (KeStackAttachProcess), once attached a file on disk (C:\dump.dmp) to receive memory pages dump, then the memory dumping process starts with (ZwQueryVirtualMemory) to query each virtual memory page info, including its base address and size, then (ZwWriteFile) is used to write that page to the dump file on disk. ====== For Professional Inquiries, freelance, contact on LinkedIn or Gmail (saaad.ahla@gmail[.]com), looking forward to new opportunities.
0:28

Very stealthy way of dumping LSASS, is done from the kernel, doesn't matter if LSASS is LSA Protected. Once loaded, the rootkit creates a system thread (PsCreateSystemThread) for dumping LSASS, iterates over all processes gets its EPROCESS (PsLookupProcessByProcessId ), then its name (PsGetProcessImageFileName), attaches to LSASS with (KeStackAttachProcess), once attached a file on disk (C:\dump.dmp) to receive memory pages dump, then the memory dumping process starts with (ZwQueryVirtualMemory) to query each virtual memory page info, including its base address and size, then (ZwWriteFile) is used to write that page to the dump file on disk. ====== For Professional Inquiries, freelance, contact on LinkedIn or Gmail (saaad.ahla@gmail[.]com), looking forward to new opportunities.

d1rkmtr

22,041 Aufrufe • vor 1 Jahr

"I'm Still Here" director Walter Salles on making the film in the digital age: "Memory was at the heart of this project...We live in a moment where memory is being erased as a project of power. So to create memory is extremely important."
1:29

"I'm Still Here" director Walter Salles on making the film in the digital age: "Memory was at the heart of this project...We live in a moment where memory is being erased as a project of power. So to create memory is extremely important."

Variety

164,345 Aufrufe • vor 1 Jahr

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.
1:09

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

Karl Mehta

25,716 Aufrufe • vor 15 Tagen

“This is my memory, so maybe the fans [will say], ‘I don’t do that!’— but in my memory, I remember when [we] performed ‘Story Of My Life’ for the first time. [We] chose Japan as a place to perform this performance because [of the] sincere respect from the Japanese audience. If [we’d done] the performance [for example] in England, I'm sure they would scream or say something to the stage, but we did ‘Story Of My Life’ live in Japan. [We] performed a cappella. I think that was a performance that could not be realised outside of Japan. It was a magical moment. [It was] the first time we performed in such a way. That performance a cappella was the best moment. It's a memory of a lifetime.” -Louis on a special memory from Japan, via Front Row.
0:36

“This is my memory, so maybe the fans [will say], ‘I don’t do that!’— but in my memory, I remember when [we] performed ‘Story Of My Life’ for the first time. [We] chose Japan as a place to perform this performance because [of the] sincere respect from the Japanese audience. If [we’d done] the performance [for example] in England, I'm sure they would scream or say something to the stage, but we did ‘Story Of My Life’ live in Japan. [We] performed a cappella. I think that was a performance that could not be realised outside of Japan. It was a magical moment. [It was] the first time we performed in such a way. That performance a cappella was the best moment. It's a memory of a lifetime.” -Louis on a special memory from Japan, via Front Row.

HL Daily Media

520,542 Aufrufe • vor 3 Jahren

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.
0:50

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

Fireside Alpha

38,370 Aufrufe • vor 13 Tagen

Virtual Textures in the Browser. This is a WebGL 2.0 implementation I did in 2023. The car model is from Sketchfab by Pavel Matoušek. It's a worst-case for virtual textures because the texture is generated from photogrametry software and has pathologically bad UV layout. The texture is 16,384 x 16,384 pixels, which is 180 Mb of PNG data. Loading this in Blender would crash Blender for me back then. The GPU memory requirement for this texture is 1 Gb. The VT displays the model after loading just 34 Kb (one tile). The other tiles are fetched in via a smart priority queue in the background as you move camera around the scene. The physical page texture is 2k, which is 16 times smaller than the source. The solution supports multiple separate virtual textures, and has GLTF integration. Texture filtering as well as anisotropy are fully supported as well. You can try the demo yourself here: Here's a writeup I did while working on this: The video recorded on a very crummy internet connection and I intentionally reload the page in the middle. The delay comes from the geometry download, actual texture part is near-instant.
1:28

Virtual Textures in the Browser. This is a WebGL 2.0 implementation I did in 2023. The car model is from Sketchfab by Pavel Matoušek. It's a worst-case for virtual textures because the texture is generated from photogrametry software and has pathologically bad UV layout. The texture is 16,384 x 16,384 pixels, which is 180 Mb of PNG data. Loading this in Blender would crash Blender for me back then. The GPU memory requirement for this texture is 1 Gb. The VT displays the model after loading just 34 Kb (one tile). The other tiles are fetched in via a smart priority queue in the background as you move camera around the scene. The physical page texture is 2k, which is 16 times smaller than the source. The solution supports multiple separate virtual textures, and has GLTF integration. Texture filtering as well as anisotropy are fully supported as well. You can try the demo yourself here: Here's a writeup I did while working on this: The video recorded on a very crummy internet connection and I intentionally reload the page in the middle. The delay comes from the geometry download, actual texture part is near-instant.

Alex Goldring

19,368 Aufrufe • vor 6 Monaten

Here’s a demo of our Agentic Memory system, inspired by how our own brain holds information in a 3D spatial space. This feels, natural. Extending this further, we announced the Agentic Memory Protocol on July 15th in SF - which enables memory to be local, encrypted and available to other agents and apps based on your permission-only. We believe this is the future of memory - not owned by any one app, spatial, and always improving. Aaron Levie Andrej Karpathy
1:03

Here’s a demo of our Agentic Memory system, inspired by how our own brain holds information in a 3D spatial space. This feels, natural. Extending this further, we announced the Agentic Memory Protocol on July 15th in SF - which enables memory to be local, encrypted and available to other agents and apps based on your permission-only. We believe this is the future of memory - not owned by any one app, spatial, and always improving. Aaron Levie Andrej Karpathy

Varun

187,489 Aufrufe • vor 11 Monaten

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!
0:39

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

128,079 Aufrufe • vor 12 Tagen

Dylan Patel says the memory shortage is not a normal cycle "Memory capacity is only growing 20, 30% a year for the next three years. And yet, demand is doubling, is doubling...memory prices are going to keep soaring." "Our point there was memory isn't a shortage, and this is not a short-term shortage. It's a shortage that's going to last years." "Memory we've had the pricing go up like 4x and it's going to go up another 2x, 3x again, in addition to capacity growth. We talked about how KV cache was going to explode because of reasoning. And therefore memory was going to be the biggest winner." "We're still not at 85, 90% gross margins for memory, but we'll get there. And then at some point from there, it'll also halve back down to like 70s or maybe even lower."
0:52

Dylan Patel says the memory shortage is not a normal cycle "Memory capacity is only growing 20, 30% a year for the next three years. And yet, demand is doubling, is doubling...memory prices are going to keep soaring." "Our point there was memory isn't a shortage, and this is not a short-term shortage. It's a shortage that's going to last years." "Memory we've had the pricing go up like 4x and it's going to go up another 2x, 3x again, in addition to capacity growth. We talked about how KV cache was going to explode because of reasoning. And therefore memory was going to be the biggest winner." "We're still not at 85, 90% gross margins for memory, but we'll get there. And then at some point from there, it'll also halve back down to like 70s or maybe even lower."

Fireside Alpha

125,121 Aufrufe • vor 3 Tagen

Understand the fundamentals of Android's memory page architecture to optimize performance. 🧠 Dive into the details of: 💡 Physical and virtual memory 📂 Efficient data access 🛠️ 16 KB size compatibility
1:09

Understand the fundamentals of Android's memory page architecture to optimize performance. 🧠 Dive into the details of: 💡 Physical and virtual memory 📂 Efficient data access 🛠️ 16 KB size compatibility

Android Developers

10,701 Aufrufe • vor 2 Monaten

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

267,009 Aufrufe • vor 15 Tagen

"What happens when we can access each other's AI memory in enterprises?" scott belsky, investor/founder of Behance: "If I was working as an intern for years, and the context window is accessible to the enterprise, can the enterprise (keep prompting) and asking me what I'd think or do, after I leave?" "Every company right now is trying to build this AI memory for each of its products and users. A lot of them are building connectors to form a collective memory." "In the social/consumer world, imagine your girlfriend saying she wants access to your context window or memory." "There are all kinds of weird questions and implications that arise from this."
1:10

"What happens when we can access each other's AI memory in enterprises?" scott belsky, investor/founder of Behance: "If I was working as an intern for years, and the context window is accessible to the enterprise, can the enterprise (keep prompting) and asking me what I'd think or do, after I leave?" "Every company right now is trying to build this AI memory for each of its products and users. A lot of them are building connectors to form a collective memory." "In the social/consumer world, imagine your girlfriend saying she wants access to your context window or memory." "There are all kinds of weird questions and implications that arise from this."

TBPN

44,908 Aufrufe • vor 1 Jahr

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”
1:31

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 Aufrufe • vor 1 Monat

The New 5-Star Memory Wish Pool [Xavier: Silvery Polyphony] is Coming Soon! "When you play it... I feel like the universe's echoes have finally found their answer." 💫Event Duration: From 5:00 AM on Dec. 18 to 4:59 AM on Dec. 28 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Xavier: Silvery Polyphony] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool, Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Concerto Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Facial: Glistening Tears], [Deepspace Wish: Limited*20], His General Photo Pose [Consideration], Memory Ascension materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Facial will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Concerto Pack] and [Silvery Cascade Gift] series will be available for purchase, including [Deepspace Wish: Limited], all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Xavier
0:35

The New 5-Star Memory Wish Pool [Xavier: Silvery Polyphony] is Coming Soon! "When you play it... I feel like the universe's echoes have finally found their answer." 💫Event Duration: From 5:00 AM on Dec. 18 to 4:59 AM on Dec. 28 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Xavier: Silvery Polyphony] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool, Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Concerto Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Facial: Glistening Tears], [Deepspace Wish: Limited*20], His General Photo Pose [Consideration], Memory Ascension materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Facial will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Concerto Pack] and [Silvery Cascade Gift] series will be available for purchase, including [Deepspace Wish: Limited], all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Xavier

Love and Deepspace

392,707 Aufrufe • vor 1 Jahr

BOMBSHELL: The Defense for Tyler Robinson just revealed that there is a TWENTY TWO PAGE document that they want to REDACT. This is almost definitely the Twiggs recording. "We have provided the state, I want to say about a day and a half ago, with proposed redactions because we believe that portions of it are not admissible even in a preliminary hearing." "And the way we did that is we provided a transcript and then a redacted version of the transcript of the entire video-recorded interview." "It may be that the court wants to look at the proposed redactions not while court is in session, but during some other period of time, because the transcript of the video, uh, which we prepared and we provided to the state so the state could raise any concerns about the accuracy of the transcript." "That's not an issue. The admissibility of portions of the video, or the issue, is about 20 pages, maybe 22 pages." Why is the defense pushing to REDACT the truth?
1:50

BOMBSHELL: The Defense for Tyler Robinson just revealed that there is a TWENTY TWO PAGE document that they want to REDACT. This is almost definitely the Twiggs recording. "We have provided the state, I want to say about a day and a half ago, with proposed redactions because we believe that portions of it are not admissible even in a preliminary hearing." "And the way we did that is we provided a transcript and then a redacted version of the transcript of the entire video-recorded interview." "It may be that the court wants to look at the proposed redactions not while court is in session, but during some other period of time, because the transcript of the video, uh, which we prepared and we provided to the state so the state could raise any concerns about the accuracy of the transcript." "That's not an issue. The admissibility of portions of the video, or the issue, is about 20 pages, maybe 22 pages." Why is the defense pushing to REDACT the truth?

Gunther Eagleman™

364,954 Aufrufe • vor 4 Tagen

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!
2:57

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

200,752 Aufrufe • vor 1 Jahr

Love and Deepspace | The New 5-Star Memory Wish Pool [Zayne: Everlasting Wish] is Coming Soon! "You'll know when my wish comes true." 💫Event Duration From 5:00 AM on Mar. 9 to 4:59 AM on Mar. 18 (server time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Zayne: Everlasting Wish] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Eternal Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Headwear: Dynamic Hairband], [Deepspace Wish: Limited*20], General Photo Pose [Warm Embrace], Memory Ascension Materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Headwear will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Dazzling Pack] and [Twilight Pack] series, which include [Deepspace Wish: Limited], will be available for purchase, all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Zayne
0:44

Love and Deepspace | The New 5-Star Memory Wish Pool [Zayne: Everlasting Wish] is Coming Soon! "You'll know when my wish comes true." 💫Event Duration From 5:00 AM on Mar. 9 to 4:59 AM on Mar. 18 (server time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Zayne: Everlasting Wish] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Eternal Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Headwear: Dynamic Hairband], [Deepspace Wish: Limited*20], General Photo Pose [Warm Embrace], Memory Ascension Materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Headwear will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Dazzling Pack] and [Twilight Pack] series, which include [Deepspace Wish: Limited], will be available for purchase, all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Zayne

Love and Deepspace

507,837 Aufrufe • vor 1 Jahr

Love and Deepspace | The New 5-Star Memory Limited Wish Pool [Caleb: Indulgent Scheme] is Coming Soon! "You know, you used to say thank you before every meal. What about now?" 💫Event Duration: From 05:00 on Dec. 18 to 04:59 on Dec. 27 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the limited 5-Star Memory [Caleb: Indulgent Scheme] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific number of Wishes. If the 5-Star Memory you have obtained from the Limited Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and pity count from this Wish Pool will also be applied to the upcoming Limited Wish Pool. *You can read more about the event on the in-game rule page. 💫Freshmorn Blessing: Wish Rewards During the event, after making a certain number of Wishes, you can claim various rewards, including Universal Decal [Marked by Love], [Deepspace Wish: Limited*20], General Photo Pose [The Calmnest], Memory Ascension Materials, etc. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Hat will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Loveswoon Pack] and [Morningfeast Pack] series, which include [Deepspace Wish: Limited] and other materials, will be available in Shop. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. ——— 🪐Official Discord: #LoveandDeepspace #Caleb
0:40

Love and Deepspace | The New 5-Star Memory Limited Wish Pool [Caleb: Indulgent Scheme] is Coming Soon! "You know, you used to say thank you before every meal. What about now?" 💫Event Duration: From 05:00 on Dec. 18 to 04:59 on Dec. 27 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the limited 5-Star Memory [Caleb: Indulgent Scheme] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific number of Wishes. If the 5-Star Memory you have obtained from the Limited Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and pity count from this Wish Pool will also be applied to the upcoming Limited Wish Pool. *You can read more about the event on the in-game rule page. 💫Freshmorn Blessing: Wish Rewards During the event, after making a certain number of Wishes, you can claim various rewards, including Universal Decal [Marked by Love], [Deepspace Wish: Limited*20], General Photo Pose [The Calmnest], Memory Ascension Materials, etc. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Hat will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Loveswoon Pack] and [Morningfeast Pack] series, which include [Deepspace Wish: Limited] and other materials, will be available in Shop. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. ——— 🪐Official Discord: #LoveandDeepspace #Caleb

Love and Deepspace

494,283 Aufrufe • vor 6 Monaten

The President and the First Lady honored the memory of the victims of the Holodomors in Ukraine. Our state remembers the millions of people whose lives were taken by artificial famine, and we know who is responsible for this. Unity in memory is as important as unity in defense. Eternal memory to all victims of the Holodomor-genocides. Today, at 4.00 PM, wherever we are – in Ukraine or abroad – let us light a candle of remembrance near our window.
0:33

The President and the First Lady honored the memory of the victims of the Holodomors in Ukraine. Our state remembers the millions of people whose lives were taken by artificial famine, and we know who is responsible for this. Unity in memory is as important as unity in defense. Eternal memory to all victims of the Holodomor-genocides. Today, at 4.00 PM, wherever we are – in Ukraine or abroad – let us light a candle of remembrance near our window.

тату Аня зубко 🫡 Xena

124,130 Aufrufe • vor 7 Monaten

What is your earliest memory? Memory in young children is a fascinating - and complex - topic. On one hand, newborns exhibit immediate recognition of their mothers’ voices and music played regularly while they were in utero. And within days they can recall their mother’s faces and distinguish them from those of others. Clearly newborns have some capacity for memory. But this early sensory memory is different from the long term, autobiographical memory we depend on to recall our lives’ most significant events. Indeed, few adults can claim memories earlier than 2-3 years of age. Why don’t we remember our earliest years? Researchers point to a couple of potential factors, beginning with language. As adults we rely heavily on language to organize our autobiographical memories into narratives, but this capacity is still developing during infancy. In addition, there is the issue of the hippocampus, the region of the brain responsible for converting short-term memories into long-term memories - which, during infancy, is still very much in development itself. The result is a phenomenon sometimes called infantile amnesia. So take plenty of videos to share with your child as they get older, because the most vivid memories of their early years are likely to be yours. How about you? What is your earliest memory and how old would you have been at the time? This adorable father/son conversation was shared to IG by Owens.Shaun.
0:29

What is your earliest memory? Memory in young children is a fascinating - and complex - topic. On one hand, newborns exhibit immediate recognition of their mothers’ voices and music played regularly while they were in utero. And within days they can recall their mother’s faces and distinguish them from those of others. Clearly newborns have some capacity for memory. But this early sensory memory is different from the long term, autobiographical memory we depend on to recall our lives’ most significant events. Indeed, few adults can claim memories earlier than 2-3 years of age. Why don’t we remember our earliest years? Researchers point to a couple of potential factors, beginning with language. As adults we rely heavily on language to organize our autobiographical memories into narratives, but this capacity is still developing during infancy. In addition, there is the issue of the hippocampus, the region of the brain responsible for converting short-term memories into long-term memories - which, during infancy, is still very much in development itself. The result is a phenomenon sometimes called infantile amnesia. So take plenty of videos to share with your child as they get older, because the most vivid memories of their early years are likely to be yours. How about you? What is your earliest memory and how old would you have been at the time? This adorable father/son conversation was shared to IG by Owens.Shaun.

Dan Wuori

169,702 Aufrufe • vor 2 Jahren