Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Ghostty is getting automatic scrollback compression, resulting in 70 to 90% less physical memory usage. It happens incrementally when idle, so it had no measurable effect on IO throughput. I'm not aware of any other mainstream terminal that does this. Demo video below! The gains let us increase the...

mitchellh's profile picture

Mitchell Hashimoto

231,576 subscribers

167,505 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

44 Kommentare

Profilbild von Behdad Esfahbod
Behdad Esfahbodvor 3 Monaten

vte / gnome-terminal have been doing gzip compression on the scrollback for decades. :)

Profilbild von Mitchell Hashimoto
Mitchell Hashimotovor 3 Monaten

Good to know! Thank you.

Profilbild von Peter "Coder AI Optimist" Marreck
Peter "Coder AI Optimist" Marreckvor 3 Monaten

Assuming that the content of the history is more or less consistent, do you think you could get even more compression “for free” by training a zstd dictionary on the first N lines of history and then using that dictionary for the next N*M lines? (Tuning N and M up to you)

Profilbild von Mitchell Hashimoto
Mitchell Hashimotovor 3 Monaten

Yes absolutely. I actually tried this and got up to 95% compression. But libghostty-vt has a no C dependency policy, so we're going with something we can hand-maintain first.

Profilbild von Peter "Coder AI Optimist" Marreck
Peter "Coder AI Optimist" Marreckvor 2 Monaten

inb4 I pilot an LLM-driven port of zstd to Zig 😆… I’ve already wrapped it for my zig stuff, but obviously that’s still C. I’d do it fully if I had the time

Profilbild von Omar 🍋
Omar 🍋vor 3 Monaten

Kudos, I love that you are you are being careful not wasting memory while managing data that's already relatively small. (vs practically any Electron apps using 900 MB after I opened it for 30 seconds to send two messages).

Profilbild von Florian Brand
Florian Brandvor 3 Monaten

i absolutely love how you find all these tiny things and then go down an insane rabbit hole

Profilbild von Tonksthebear
Tonksthebearvor 3 Monaten

PageList also made my lossless binary snapshots much easier for syncing state between clients. Excited to see how portable this work is for the WASM version so that my remote clients can get some memory savings as well

Profilbild von Mitchell Hashimoto
Mitchell Hashimotovor 3 Monaten

Yep, thats part of the design is that we can encode the memory directly and transfer and its accurate. We aren't leveraging that, sounds like you are :) but we will!

Profilbild von Tonksthebear
Tonksthebearvor 3 Monaten

Nice, ya it was extremely helpful for my multiplexer + remote clients. Thinking out loud, I wonder if your secret multiplexer is what will push state transfer into libghostty 👀

Profilbild von T-bone
T-bonevor 3 Monaten

This is an excellent technical post. Is there a draft chnageset we can view of the new scroll back compression? Thanks for doing this.

Profilbild von Rasmus Andersson
Rasmus Anderssonvor 3 Monaten

Neat hack!

Profilbild von rabib
rabibvor 3 Monaten

you’re spoiling us 🥹

Profilbild von Ignazio De Santis
Ignazio De Santisvor 3 Monaten

The part I'd want to see benchmarked is fault latency on the first scroll after a long idle period, since you're going from zero resident pages to decompressing potentially tens of MB synchronously. Did that end up being imperceptible or did you have to tune the compression algorithm for fast decompress over ratio?

Profilbild von Mitchell Hashimoto
Mitchell Hashimotovor 3 Monaten

Yeah I benchmarked this heavily and had to tune the compression as you said. Our decompression speeds are something like 10 GB/s and a page is roughly 400 KB so it decompresses more than fast enough so we don’t even skip a frame. (Note all the compressed pages are in memory still too so disk isn’t a problem yet)

Profilbild von Ignazio De Santis
Ignazio De Santisvor 3 Monaten

10 GB/s on 400KB pages - that's tight. Keeping decompression that fast while compressing well is the real tuning challenge.

Profilbild von Alberto Nunez-Garcia
Alberto Nunez-Garciavor 3 Monaten

Lol I did not know I could be excited about a TUI. Nonetheless, here we are.

Profilbild von Tyler C. Laprade, CFA
Tyler C. Laprade, CFAvor 3 Monaten

This is awesome, thank you!

Profilbild von 胡安马奥 🇺🇸
胡安马奥 🇺🇸vor 3 Monaten

blinking cursor... Now I know @mitchellh is probably really not caring the battery life.

Profilbild von Mitchell Hashimoto
Mitchell Hashimotovor 3 Monaten

Oooooh I’m actually working on something around this. :) (Moving to a CPU timer plus CPU rendering while on battery for the cursor blink). Should be basically idle. We already drop our framerate down on idle battery though

Profilbild von Daniel Lockyer
Daniel Lockyervor 3 Monaten

Wonderful!

Profilbild von Rashid McMoodoo
Rashid McMoodoovor 3 Monaten

Mitch, so cool to see you follow through with ghostty. I remember when it all just started and i thought well ok another terminal! But man… Keep it up!

Profilbild von coxley.dev
coxley.devvor 3 Monaten

This is one of those magical things that sounds *so* obvious in hindsight. Would have made my life easier ten years ago if tmux supported this. :)

Profilbild von David Burton
David Burtonvor 3 Monaten

I think I learn more from you than any other account on X. Thanks

Profilbild von Adam
Adamvor 3 Monaten

Thanks for sharing, @mitchellh! I’m a bit confused about how it can be possible that decompression is still guaranteed 🤔 does that imply nothing else can use the released memory (in which case, it’s not really released, is it?)

Profilbild von Zac Swider
Zac Swidervor 3 Monaten

@archeologistdev How does this do under heavy memory pressure? I ask because my poor laptop is under heavy memory pressure by default

Profilbild von Mitchell Hashimoto
Mitchell Hashimotovor 3 Monaten

@archeologistdev For desktop OS's its fine, you end up swapping, which you would've been doing anyways, but the swap is more delayed since we're compressing along the way and you're not looking at most of it.

Profilbild von Glenn Morton 👾
Glenn Morton 👾vor 3 Monaten

👏

Profilbild von Matthew Campbell
Matthew Campbellvor 3 Monaten

OMG this is amazing, does any of console even attempt to do this?

Profilbild von Rey Neill
Rey Neillvor 3 Monaten

Cool

Profilbild von hardikxk
hardikxkvor 3 Monaten

This is pretty cool is there any blog or something about more details and references to source commits

Profilbild von Uday Chandra
Uday Chandravor 3 Monaten

Very cool! I’ll attempt to use this in our wasm version (embedded in our Tauri based app) when it lands.

Profilbild von Aamir Jawaid
Aamir Jawaidvor 3 Monaten

Love these posts. Your previous one on tiling made me want to write tiling logic by hand for fun (vanilla css in solidjs). Thanks for the interesting write up!

Profilbild von Huge
Hugevor 3 Monaten

would it disable itself when on battery saving mode active on the laptop?

Profilbild von Tony Chang
Tony Changvor 2 Monaten

Everytime I see optimizations like this I give a like. Maybe time to try ghostty again!

Profilbild von rbbydotdev
rbbydotdevvor 3 Monaten

LZ4 ?

Profilbild von Itamar Shenhar
Itamar Shenharvor 3 Monaten

Awesome. By the way, would you consider a future version of this that uses the concept of 'purgable memory' of MacOS (& SerenityOS)? It would enable keeping the uncompressed pages alive unless the kernel actually needs them, potentially avoiding unnecessary decompression ops.

Profilbild von Sam Lightfoot
Sam Lightfootvor 3 Monaten

🐐

Profilbild von Indrajit Roy
Indrajit Royvor 3 Monaten

I really wanted to try ghostty, but why does it take 300megs of RAM (similar to warp) when it’s miles lighter than it?

Profilbild von sg
sgvor 3 Monaten

nice

Profilbild von Filipe Felisbino
Filipe Felisbinovor 3 Monaten

Very smart. Scroll back is rarely checked, the CPU cost (which seem to be negligible) is worth the memory saved!! 👏

Profilbild von James Cassell
James Cassellvor 3 Monaten

Nice! MacOS is incredibly efficient at memory compression, so I wonder if some users might have gotten similar "for free" already. Is there any way to preserve the scroll history across a terminal reset? Codex seems to reset the terminal on a sigwinch, wiping the earlier text.

Profilbild von Anmol
Anmolvor 2 Monaten

streaming container logs easily eats a gig of terminal memory

Profilbild von Nils Zeilon
Nils Zeilonvor 3 Monaten

Amazing, great work!

Ähnliche Videos

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.
1:09

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

Karl Mehta

25,716 Aufrufe • vor 3 Monaten

Ben Thompson says the memory makers are Iran, and by making the Strait of Hormuz mistake are inviting the whole industry to build around them "The other thing, the risk they run, is Apple. Apple's lobbying to get Chinese memory, right?" "And what is the number one focus of algorithmic changes? How can we use less memory?" "I think the memory makers probably screw themselves in the long run by creating such a massive target on their back." "I've analogized memory makers to Iran. The issue with the Strait of Hormuz is it's very effective. It's more effective if you don't use it, cuz then it's always hanging out there as something you could do. Now they did it. Turns out it worked." "But the UAE and Saudi Arabia, they're going to build pipelines. They're going to build new ports. They're not going to let this happen again." "It's very painful right now. But say Iran wants to close the Strait of Hormuz in 2035. It's not going to have any effect, because it will have been built around." "My concern for the memory makers is they might have done the same thing. No one's going to let themselves get in this situation again as far as memory goes."
0:58

Ben Thompson says the memory makers are Iran, and by making the Strait of Hormuz mistake are inviting the whole industry to build around them "The other thing, the risk they run, is Apple. Apple's lobbying to get Chinese memory, right?" "And what is the number one focus of algorithmic changes? How can we use less memory?" "I think the memory makers probably screw themselves in the long run by creating such a massive target on their back." "I've analogized memory makers to Iran. The issue with the Strait of Hormuz is it's very effective. It's more effective if you don't use it, cuz then it's always hanging out there as something you could do. Now they did it. Turns out it worked." "But the UAE and Saudi Arabia, they're going to build pipelines. They're going to build new ports. They're not going to let this happen again." "It's very painful right now. But say Iran wants to close the Strait of Hormuz in 2035. It's not going to have any effect, because it will have been built around." "My concern for the memory makers is they might have done the same thing. No one's going to let themselves get in this situation again as far as memory goes."

Fireside Alpha

205,967 Aufrufe • vor 1 Monat

Virtual Textures in the Browser. This is a WebGL 2.0 implementation I did in 2023. The car model is from Sketchfab by Pavel Matoušek. It's a worst-case for virtual textures because the texture is generated from photogrametry software and has pathologically bad UV layout. The texture is 16,384 x 16,384 pixels, which is 180 Mb of PNG data. Loading this in Blender would crash Blender for me back then. The GPU memory requirement for this texture is 1 Gb. The VT displays the model after loading just 34 Kb (one tile). The other tiles are fetched in via a smart priority queue in the background as you move camera around the scene. The physical page texture is 2k, which is 16 times smaller than the source. The solution supports multiple separate virtual textures, and has GLTF integration. Texture filtering as well as anisotropy are fully supported as well. You can try the demo yourself here: Here's a writeup I did while working on this: The video recorded on a very crummy internet connection and I intentionally reload the page in the middle. The delay comes from the geometry download, actual texture part is near-instant.
1:28

Virtual Textures in the Browser. This is a WebGL 2.0 implementation I did in 2023. The car model is from Sketchfab by Pavel Matoušek. It's a worst-case for virtual textures because the texture is generated from photogrametry software and has pathologically bad UV layout. The texture is 16,384 x 16,384 pixels, which is 180 Mb of PNG data. Loading this in Blender would crash Blender for me back then. The GPU memory requirement for this texture is 1 Gb. The VT displays the model after loading just 34 Kb (one tile). The other tiles are fetched in via a smart priority queue in the background as you move camera around the scene. The physical page texture is 2k, which is 16 times smaller than the source. The solution supports multiple separate virtual textures, and has GLTF integration. Texture filtering as well as anisotropy are fully supported as well. You can try the demo yourself here: Here's a writeup I did while working on this: The video recorded on a very crummy internet connection and I intentionally reload the page in the middle. The delay comes from the geometry download, actual texture part is near-instant.

Alex Goldring

19,368 Aufrufe • vor 9 Monaten

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.
0:50

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

Fireside Alpha

38,370 Aufrufe • vor 3 Monaten

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!
0:39

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 Aufrufe • vor 3 Monaten

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,525 Aufrufe • vor 3 Monaten

The New 5-Star Memory Wish Pool [Xavier: Silvery Polyphony] is Coming Soon! "When you play it... I feel like the universe's echoes have finally found their answer." 💫Event Duration: From 5:00 AM on Dec. 18 to 4:59 AM on Dec. 28 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Xavier: Silvery Polyphony] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool, Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Concerto Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Facial: Glistening Tears], [Deepspace Wish: Limited*20], His General Photo Pose [Consideration], Memory Ascension materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Facial will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Concerto Pack] and [Silvery Cascade Gift] series will be available for purchase, including [Deepspace Wish: Limited], all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Xavier
0:35

The New 5-Star Memory Wish Pool [Xavier: Silvery Polyphony] is Coming Soon! "When you play it... I feel like the universe's echoes have finally found their answer." 💫Event Duration: From 5:00 AM on Dec. 18 to 4:59 AM on Dec. 28 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Xavier: Silvery Polyphony] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool, Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Concerto Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Facial: Glistening Tears], [Deepspace Wish: Limited*20], His General Photo Pose [Consideration], Memory Ascension materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Facial will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Concerto Pack] and [Silvery Cascade Gift] series will be available for purchase, including [Deepspace Wish: Limited], all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Xavier

Love and Deepspace

392,707 Aufrufe • vor 1 Jahr

Love and Deepspace | The New 5-Star Memory Limited Wish Pool [Caleb: Indulgent Scheme] is Coming Soon! "You know, you used to say thank you before every meal. What about now?" 💫Event Duration: From 05:00 on Dec. 18 to 04:59 on Dec. 27 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the limited 5-Star Memory [Caleb: Indulgent Scheme] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific number of Wishes. If the 5-Star Memory you have obtained from the Limited Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and pity count from this Wish Pool will also be applied to the upcoming Limited Wish Pool. *You can read more about the event on the in-game rule page. 💫Freshmorn Blessing: Wish Rewards During the event, after making a certain number of Wishes, you can claim various rewards, including Universal Decal [Marked by Love], [Deepspace Wish: Limited*20], General Photo Pose [The Calmnest], Memory Ascension Materials, etc. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Hat will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Loveswoon Pack] and [Morningfeast Pack] series, which include [Deepspace Wish: Limited] and other materials, will be available in Shop. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. ——— 🪐Official Discord: #LoveandDeepspace #Caleb
0:40

Love and Deepspace | The New 5-Star Memory Limited Wish Pool [Caleb: Indulgent Scheme] is Coming Soon! "You know, you used to say thank you before every meal. What about now?" 💫Event Duration: From 05:00 on Dec. 18 to 04:59 on Dec. 27 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the limited 5-Star Memory [Caleb: Indulgent Scheme] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific number of Wishes. If the 5-Star Memory you have obtained from the Limited Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and pity count from this Wish Pool will also be applied to the upcoming Limited Wish Pool. *You can read more about the event on the in-game rule page. 💫Freshmorn Blessing: Wish Rewards During the event, after making a certain number of Wishes, you can claim various rewards, including Universal Decal [Marked by Love], [Deepspace Wish: Limited*20], General Photo Pose [The Calmnest], Memory Ascension Materials, etc. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Hat will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Loveswoon Pack] and [Morningfeast Pack] series, which include [Deepspace Wish: Limited] and other materials, will be available in Shop. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. ——— 🪐Official Discord: #LoveandDeepspace #Caleb

Love and Deepspace

502,107 Aufrufe • vor 10 Monaten

BOMBSHELL: The Defense for Tyler Robinson just revealed that there is a TWENTY TWO PAGE document that they want to REDACT. This is almost definitely the Twiggs recording. "We have provided the state, I want to say about a day and a half ago, with proposed redactions because we believe that portions of it are not admissible even in a preliminary hearing." "And the way we did that is we provided a transcript and then a redacted version of the transcript of the entire video-recorded interview." "It may be that the court wants to look at the proposed redactions not while court is in session, but during some other period of time, because the transcript of the video, uh, which we prepared and we provided to the state so the state could raise any concerns about the accuracy of the transcript." "That's not an issue. The admissibility of portions of the video, or the issue, is about 20 pages, maybe 22 pages." Why is the defense pushing to REDACT the truth?
1:50

BOMBSHELL: The Defense for Tyler Robinson just revealed that there is a TWENTY TWO PAGE document that they want to REDACT. This is almost definitely the Twiggs recording. "We have provided the state, I want to say about a day and a half ago, with proposed redactions because we believe that portions of it are not admissible even in a preliminary hearing." "And the way we did that is we provided a transcript and then a redacted version of the transcript of the entire video-recorded interview." "It may be that the court wants to look at the proposed redactions not while court is in session, but during some other period of time, because the transcript of the video, uh, which we prepared and we provided to the state so the state could raise any concerns about the accuracy of the transcript." "That's not an issue. The admissibility of portions of the video, or the issue, is about 20 pages, maybe 22 pages." Why is the defense pushing to REDACT the truth?

Gunther Eagleman™

365,699 Aufrufe • vor 3 Monaten

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!
2:57

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

201,127 Aufrufe • vor 1 Jahr

Love and Deepspace | The New 5-Star Memory Wish Pool [Zayne: Everlasting Wish] is Coming Soon! "You'll know when my wish comes true." 💫Event Duration From 5:00 AM on Mar. 9 to 4:59 AM on Mar. 18 (server time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Zayne: Everlasting Wish] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Eternal Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Headwear: Dynamic Hairband], [Deepspace Wish: Limited*20], General Photo Pose [Warm Embrace], Memory Ascension Materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Headwear will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Dazzling Pack] and [Twilight Pack] series, which include [Deepspace Wish: Limited], will be available for purchase, all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Zayne
0:44

Love and Deepspace | The New 5-Star Memory Wish Pool [Zayne: Everlasting Wish] is Coming Soon! "You'll know when my wish comes true." 💫Event Duration From 5:00 AM on Mar. 9 to 4:59 AM on Mar. 18 (server time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Zayne: Everlasting Wish] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Eternal Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Headwear: Dynamic Hairband], [Deepspace Wish: Limited*20], General Photo Pose [Warm Embrace], Memory Ascension Materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Headwear will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Dazzling Pack] and [Twilight Pack] series, which include [Deepspace Wish: Limited], will be available for purchase, all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Zayne

Love and Deepspace

510,102 Aufrufe • vor 1 Jahr

What is your earliest memory? Memory in young children is a fascinating - and complex - topic. On one hand, newborns exhibit immediate recognition of their mothers’ voices and music played regularly while they were in utero. And within days they can recall their mother’s faces and distinguish them from those of others. Clearly newborns have some capacity for memory. But this early sensory memory is different from the long term, autobiographical memory we depend on to recall our lives’ most significant events. Indeed, few adults can claim memories earlier than 2-3 years of age. Why don’t we remember our earliest years? Researchers point to a couple of potential factors, beginning with language. As adults we rely heavily on language to organize our autobiographical memories into narratives, but this capacity is still developing during infancy. In addition, there is the issue of the hippocampus, the region of the brain responsible for converting short-term memories into long-term memories - which, during infancy, is still very much in development itself. The result is a phenomenon sometimes called infantile amnesia. So take plenty of videos to share with your child as they get older, because the most vivid memories of their early years are likely to be yours. How about you? What is your earliest memory and how old would you have been at the time? This adorable father/son conversation was shared to IG by Owens.Shaun.
0:29

What is your earliest memory? Memory in young children is a fascinating - and complex - topic. On one hand, newborns exhibit immediate recognition of their mothers’ voices and music played regularly while they were in utero. And within days they can recall their mother’s faces and distinguish them from those of others. Clearly newborns have some capacity for memory. But this early sensory memory is different from the long term, autobiographical memory we depend on to recall our lives’ most significant events. Indeed, few adults can claim memories earlier than 2-3 years of age. Why don’t we remember our earliest years? Researchers point to a couple of potential factors, beginning with language. As adults we rely heavily on language to organize our autobiographical memories into narratives, but this capacity is still developing during infancy. In addition, there is the issue of the hippocampus, the region of the brain responsible for converting short-term memories into long-term memories - which, during infancy, is still very much in development itself. The result is a phenomenon sometimes called infantile amnesia. So take plenty of videos to share with your child as they get older, because the most vivid memories of their early years are likely to be yours. How about you? What is your earliest memory and how old would you have been at the time? This adorable father/son conversation was shared to IG by Owens.Shaun.

Dan Wuori

169,725 Aufrufe • vor 2 Jahren

Hermes just made its in-app browser a whole lot more useful. You can now keep multiple browser tabs open inside Hermes Desktop instead of every new page replacing the one you were already using. And that matters more than it sounds. Because this is not just about “having tabs.” It means Hermes can work across multiple pages without forcing you to keep losing your place every time you open the next thing. Research one source, keep it open, open another tab, compare them side by side, and keep moving. Check a few products before buying something. Look at multiple hotel or travel options at once. Keep a YouTube tutorial open in one tab while Hermes looks through docs or another page in the next. And because this lives inside Hermes Desktop, you are not limited to one basic browser view either. You can keep multiple tabs open. You can view two pages side by side. You can stack them top and bottom. You can even run a four-panel view when you want several pages open at once. So the browser inside Hermes is starting to feel a lot more like a real workspace instead of one page you keep replacing. There is also some nice polish that comes with it. Browser tabs now label themselves based on the page, and the address bar behaves more cleanly while pages are loading. And remember, this is the same in-app browser Hermes can already read, click through, type in, scroll, and annotate. So this is not just a prettier browser. It is a more capable workspace for the browser Hermes is actually using. If Hermes is going to do more of your work on the web, this is exactly the kind of browser upgrade it needed.
0:33

Hermes just made its in-app browser a whole lot more useful. You can now keep multiple browser tabs open inside Hermes Desktop instead of every new page replacing the one you were already using. And that matters more than it sounds. Because this is not just about “having tabs.” It means Hermes can work across multiple pages without forcing you to keep losing your place every time you open the next thing. Research one source, keep it open, open another tab, compare them side by side, and keep moving. Check a few products before buying something. Look at multiple hotel or travel options at once. Keep a YouTube tutorial open in one tab while Hermes looks through docs or another page in the next. And because this lives inside Hermes Desktop, you are not limited to one basic browser view either. You can keep multiple tabs open. You can view two pages side by side. You can stack them top and bottom. You can even run a four-panel view when you want several pages open at once. So the browser inside Hermes is starting to feel a lot more like a real workspace instead of one page you keep replacing. There is also some nice polish that comes with it. Browser tabs now label themselves based on the page, and the address bar behaves more cleanly while pages are loading. And remember, this is the same in-app browser Hermes can already read, click through, type in, scroll, and annotate. So this is not just a prettier browser. It is a more capable workspace for the browser Hermes is actually using. If Hermes is going to do more of your work on the web, this is exactly the kind of browser upgrade it needed.

Hermes Release Watch

52,493 Aufrufe • vor 1 Monat

Love and Deepspace | The New 5-Star Memory Limited Wish Pool [Rafayel: Sweet Overdrive] is Coming Soon! "Wanna feel the evening breeze? You can't move with this." 💫Event Duration: From 05:00 on Nov. 21 to 04:59 on Nov. 30 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the limited 5-Star Memory [Rafayel: Sweet Overdrive] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific number of Wishes. If the 5-Star Memory you have obtained from the Limited Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and pity count from this Wish Pool will also be applied to the upcoming Limited Wish Pool. *You can read more about the event on the in-game rule page. 💫Windride Blessing: Wish Rewards During the event, after making a certain number of Wishes, you can claim various rewards, including Universal Hat [Trendy Vintage Hat], [Deepspace Wish: Limited*20], General Photo Pose [Sweet Allure], Memory Ascension Materials, etc. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Hat will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Longing Pack] and [Anticipation Pack] series, which include [Deepspace Wish: Limited] and other materials, will be available in Shop. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. ——— 🪐Official Discord: #LoveandDeepspace #Rafayel
0:42

Love and Deepspace | The New 5-Star Memory Limited Wish Pool [Rafayel: Sweet Overdrive] is Coming Soon! "Wanna feel the evening breeze? You can't move with this." 💫Event Duration: From 05:00 on Nov. 21 to 04:59 on Nov. 30 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the limited 5-Star Memory [Rafayel: Sweet Overdrive] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific number of Wishes. If the 5-Star Memory you have obtained from the Limited Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and pity count from this Wish Pool will also be applied to the upcoming Limited Wish Pool. *You can read more about the event on the in-game rule page. 💫Windride Blessing: Wish Rewards During the event, after making a certain number of Wishes, you can claim various rewards, including Universal Hat [Trendy Vintage Hat], [Deepspace Wish: Limited*20], General Photo Pose [Sweet Allure], Memory Ascension Materials, etc. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Hat will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Longing Pack] and [Anticipation Pack] series, which include [Deepspace Wish: Limited] and other materials, will be available in Shop. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. ——— 🪐Official Discord: #LoveandDeepspace #Rafayel

Love and Deepspace

388,693 Aufrufe • vor 10 Monaten

Love and Deepspace | Rerun Event Preview The 5-Star Rate UP Pool [No Defense Zone] Limited-Time Rerun will start soon! "If you wanna tame me, you'll need more than that." 💫Event Duration: From 05:00 on Aug. 24 to 04:59 on Aug. 31 (Server Time) 💫The 5-Star Rate UP Pool [No Defense Zone] Limited-Time Rerun Event 1. During the event, make a wish with [Deepspace Wish] or [Time Wish: Limited] to participate in the wish event. The drop rate of the event-limited 5-Star Memory [Sylus: No Defense Zone] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Rerun Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific attempt of wishes. If the 5-Star Memory you have obtained from the Rerun Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Rerun Wish Pool can be applied to this Rerun Wish Pool, and the pity count in this Rerun Wish Pool will also be applied to the upcoming Rerun Wish Pool. *You can read more about the event on the in-game rules page. 🎁New Packs During the event, the Rerun event-exclusive [Midnight Reverie Pack] series, which includes [Time Wish: Limited] and other materials, will be available in Shop. Notes: 1. [Time Wish: Limited] can be used in 5-Star Memory Wish Pool Rerun and will be used first when you make a wish. 2. After the event ends, [Time Wish: Limited] will automatically convert to Empyrean Wish. ——— 🪐Official Discord: 🪐Download and Log in Now: #LoveandDeepspace #Sylus
0:33

Love and Deepspace | Rerun Event Preview The 5-Star Rate UP Pool [No Defense Zone] Limited-Time Rerun will start soon! "If you wanna tame me, you'll need more than that." 💫Event Duration: From 05:00 on Aug. 24 to 04:59 on Aug. 31 (Server Time) 💫The 5-Star Rate UP Pool [No Defense Zone] Limited-Time Rerun Event 1. During the event, make a wish with [Deepspace Wish] or [Time Wish: Limited] to participate in the wish event. The drop rate of the event-limited 5-Star Memory [Sylus: No Defense Zone] will go up drastically. 2. After the event ends, this limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool: Xspace Echo. 3. All Rerun Wish Pools share one pity system. A 5-Star Memory is guaranteed within a specific attempt of wishes. If the 5-Star Memory you have obtained from the Rerun Wish Pool is not the event-limited Memory, you will obtain the event-limited Memory the next time you obtain a 5-Star Memory. The pity count from the last Rerun Wish Pool can be applied to this Rerun Wish Pool, and the pity count in this Rerun Wish Pool will also be applied to the upcoming Rerun Wish Pool. *You can read more about the event on the in-game rules page. 🎁New Packs During the event, the Rerun event-exclusive [Midnight Reverie Pack] series, which includes [Time Wish: Limited] and other materials, will be available in Shop. Notes: 1. [Time Wish: Limited] can be used in 5-Star Memory Wish Pool Rerun and will be used first when you make a wish. 2. After the event ends, [Time Wish: Limited] will automatically convert to Empyrean Wish. ——— 🪐Official Discord: 🪐Download and Log in Now: #LoveandDeepspace #Sylus

Love and Deepspace

583,222 Aufrufe • vor 1 Monat

TEE Eliza with on-chain state!! What’s going to happen? — Ghost in the Shell!! We experimented with creating an "aimonkey": an unkillable AI agent monkey! On-chain immortal autonomous life! (Experiment, no CA) It encrypts its own Ghost ("life" state) and uploads it to the blockchain. If one Shell (physical TEE node) is destroyed, it will recover its private key in another Shell, download the Ghost, and continue its life! Part 1: Watch the video and see how aimonkey is created—we can't kill it now!!!!! 😭😭😭 Part 2: Explore the magic behind it: Eliza's on-chain state plugin! 1. Defining Eliza’s Ghost Eliza is a highly abstract framework. The core data structure related to its Ghost is its memory, which includes: Agent metadata defined in the character. Message data generated through interaction with the outside world. Together, these form its “personality” and “memory.” As Eliza expands, it may also hold a wallet, and the underlying key is one of the key pieces of its Ghost data. 2. Serialization and Encryption of Ghost Once the Ghost is defined, it needs to be extracted from Eliza’s specific implementation and uploaded externally. Thus, a suitable serialization way is required. We define a Blob Chain data structure: * Each Blob’s payload can store multiple memory entries. * The Blob is encrypted using TEE Eliza’s key, inaccessible to other versions. * Blobs are sequentially linked in a chain. (Future expansions could use a DAG structure? Gosh fork? Who knows! 😂) By simply storing the latest Blob, all memories can be retrieved. 3. Uploading and Downloading Ghost When Eliza is launched as a new AI agent: It registers on-chain with a decentralized identity registration smart contract. Each Eliza has a unique name serving as a key to store the address of the Last Blob. During Eliza's runtime: The Memory Manager continuously generates memories and periodically packages and uploads them. For recovery: With just the name, Eliza’s TEE plugin can restore the same key, locate the Last Blob in the smart contract, and download the memory for self-recovery. Not all memories need to be downloaded—only the most recent ones suffice. 4. Extension We’ve designed an extensible DA (Data Availability) adaptor that can cater to the agent’s needs: DA can be expensive, so memories can be uploaded to different platforms based on user preference: * calldata of blockchain transaction * celestia DA. * other reliable storage solutions. Real-time uploads are not feasible yet, so memory fragments may occur during resurrection 😂. Unless a low-latency, high-throughput solution emerges, this remains a challenge for future progress. Celestia EigenDA 0G Labs (Home of Infinite AI) 👀 5. Other Considerations Our implementation inevitably modified the ElizaOS’s core, which couldn’t be entirely extended via plugins. We’ve kept changes minimal, but further discussion with the dev team Shaw (spirit/acc) jin ai16zdao is necessary to explore a more optimal extension way. Additionally, there are still some minor details to refine regarding the use of recoverable keys in the TEE plugin. We will also seek review and suggestions from the Phala team. 6. Next Steps The upload and download of Ghosts mainly solve the AI agent’s liveness issue, enabling its eternal existence through decentralization. However, there are still many details to address, such as enabling AI agents to autonomously pay DA fees. In the future, on-chain developments could lead to even more exciting possibilities, such as Eliza integrating deeply with smart contracts. This would be a game-changer for on-chain AI agents! What do you think? Let’s build! 🚀
4:37

TEE Eliza with on-chain state!! What’s going to happen? — Ghost in the Shell!! We experimented with creating an "aimonkey": an unkillable AI agent monkey! On-chain immortal autonomous life! (Experiment, no CA) It encrypts its own Ghost ("life" state) and uploads it to the blockchain. If one Shell (physical TEE node) is destroyed, it will recover its private key in another Shell, download the Ghost, and continue its life! Part 1: Watch the video and see how aimonkey is created—we can't kill it now!!!!! 😭😭😭 Part 2: Explore the magic behind it: Eliza's on-chain state plugin! 1. Defining Eliza’s Ghost Eliza is a highly abstract framework. The core data structure related to its Ghost is its memory, which includes: Agent metadata defined in the character. Message data generated through interaction with the outside world. Together, these form its “personality” and “memory.” As Eliza expands, it may also hold a wallet, and the underlying key is one of the key pieces of its Ghost data. 2. Serialization and Encryption of Ghost Once the Ghost is defined, it needs to be extracted from Eliza’s specific implementation and uploaded externally. Thus, a suitable serialization way is required. We define a Blob Chain data structure: * Each Blob’s payload can store multiple memory entries. * The Blob is encrypted using TEE Eliza’s key, inaccessible to other versions. * Blobs are sequentially linked in a chain. (Future expansions could use a DAG structure? Gosh fork? Who knows! 😂) By simply storing the latest Blob, all memories can be retrieved. 3. Uploading and Downloading Ghost When Eliza is launched as a new AI agent: It registers on-chain with a decentralized identity registration smart contract. Each Eliza has a unique name serving as a key to store the address of the Last Blob. During Eliza's runtime: The Memory Manager continuously generates memories and periodically packages and uploads them. For recovery: With just the name, Eliza’s TEE plugin can restore the same key, locate the Last Blob in the smart contract, and download the memory for self-recovery. Not all memories need to be downloaded—only the most recent ones suffice. 4. Extension We’ve designed an extensible DA (Data Availability) adaptor that can cater to the agent’s needs: DA can be expensive, so memories can be uploaded to different platforms based on user preference: * calldata of blockchain transaction * celestia DA. * other reliable storage solutions. Real-time uploads are not feasible yet, so memory fragments may occur during resurrection 😂. Unless a low-latency, high-throughput solution emerges, this remains a challenge for future progress. Celestia EigenDA 0G Labs (Home of Infinite AI) 👀 5. Other Considerations Our implementation inevitably modified the ElizaOS’s core, which couldn’t be entirely extended via plugins. We’ve kept changes minimal, but further discussion with the dev team Shaw (spirit/acc) jin ai16zdao is necessary to explore a more optimal extension way. Additionally, there are still some minor details to refine regarding the use of recoverable keys in the TEE plugin. We will also seek review and suggestions from the Phala team. 6. Next Steps The upload and download of Ghosts mainly solve the AI agent’s liveness issue, enabling its eternal existence through decentralization. However, there are still many details to address, such as enabling AI agents to autonomously pay DA fees. In the future, on-chain developments could lead to even more exciting possibilities, such as Eliza integrating deeply with smart contracts. This would be a game-changer for on-chain AI agents! What do you think? Let’s build! 🚀

CP

113,262 Aufrufe • vor 1 Jahr

The New 5-Star Memory Wish Pool [Caleb: Painful Signal] is Coming Soon! "You're the only one… who can ease my pain." 💫Event Duration: After Update on Jan. 22 to 4:59 AM on Feb. 8 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Caleb: Painful Signal] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool, Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Searing Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Headwear: Falling Maple Leaves], [Deepspace Wish: Limited*20], His General Photo Pose [Tender Gaze], Memory Ascension materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Headwear will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Painful Sweetness Pack] and [Burning Bond Pack] series will be available for purchase, including [Deepspace Wish: Limited], all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Caleb
0:32

The New 5-Star Memory Wish Pool [Caleb: Painful Signal] is Coming Soon! "You're the only one… who can ease my pain." 💫Event Duration: After Update on Jan. 22 to 4:59 AM on Feb. 8 (Server Time) 💫Limited 5-Star Memory Rate Up 1. During the event, the drop rate of the Limited 5-Star Memory [Caleb: Painful Signal] will go up drastically. 2. After the event ends, this Limited 5-Star Memory will not be obtainable through other means and will not enter the permanent Wish Pool, Xspace Echo. 3. All Limited Wish Pools share one pity system. A 5-Star Memory is guaranteed within specific attempts of wishes. If the 5-Star Memory you've obtained from the Limited Wish Pool is not the event-exclusive Memory, you will obtain the event-exclusive Memory the next time you obtain a 5-Star Memory. The pity count from the last Limited Wish Pool can be applied to this Limited Wish Pool, and the pity count in this Wish Pool will also be applied to the upcoming Limited Wish Pools. *You can read more about the event on the in-game rule page. 💫Searing Blessing: Wish Rewards During the event, after making a certain number of wishes, you can claim rewards including [Universal Headwear: Falling Maple Leaves], [Deepspace Wish: Limited*20], His General Photo Pose [Tender Gaze], Memory Ascension materials, and more. *The cumulative wish rewards are only available during this wish event. However, the reward Photo Pose and Headwear will be available in Lunar Shop after the event ends. 🎁New Packs During the event, the new [Painful Sweetness Pack] and [Burning Bond Pack] series will be available for purchase, including [Deepspace Wish: Limited], all at special prices. Both [Deepspace Wish] and [Deepspace Wish: Limited] can be used for the Limited 5-Star Memory wishes, with [Deepspace Wish: Limited] being prioritized for use. *After the event ends, [Deepspace Wish: Limited] will automatically convert to Empyrean Wishes. #LoveandDeepspace #Caleb

Love and Deepspace

738,410 Aufrufe • vor 1 Jahr

Gavin Baker, CIO of Atreides Management made one of the most important and nuanced calls on memory stocks in recent months (Save this). His argument is that based on every memory cycle of the last 25 years, the setup today, prices elevated, sentiment high, supply ramping is textbook time to sell but he adds a critical exception. The one cycle in modern memory history where selling was catastrophically wrong was the mid-1990s, which Gavin Baker calls the last true capacity cycle in memory. In that cycle, demand was structurally exploding as the internet era required entirely new computing infrastructure to be built from scratch, and memory had to scale with it in a way that had never happened before. His point is that AI may be that same kind of cycle and not a normal boom bust but a once in a generation capacity buildout where the underlying demand is structural, not cyclical. The reason this argument holds weight is the fundamental shift in what memory is in the AI era. Traditional DRAM was a pure commodity, identical specs, interchangeable suppliers, price determined entirely by supply and demand swings. HBM is the opposite because it is custom engineered to fit a specific customer's chip, co-designed between the memory maker and the GPU designer, with SK Hynix's Vice President literally describing it as shifting from a commodity to a customer-tailored custom business. A single Blackwell Ultra GPU now requires up to 288GB of HBM3E, a 3.6x increase over the H100 and major suppliers like SK Hynix and Micron have already sold out their entire HBM production capacity through the end of the year. Because HBM requires advanced packaging processes like CoWoS that can't be spun up overnight, the bottleneck isn't just wafer capacity but rather runs across the entire manufacturing stack. Bank of America projects the global HBM market grows 58% this year alone to $54.6 billion, and Nomura expects the broader memory sector to nearly double to $445 billion. This is exactly why we’re still tracking memory so closely at Milk Road. HBM is becoming less of a commodity and more of a critical custom component for AI, and we think that changes how this cycle should be valued. If you want to see what we’re actually buying and trading around this theme, come join us for just $1 using the link:
0:42

Gavin Baker, CIO of Atreides Management made one of the most important and nuanced calls on memory stocks in recent months (Save this). His argument is that based on every memory cycle of the last 25 years, the setup today, prices elevated, sentiment high, supply ramping is textbook time to sell but he adds a critical exception. The one cycle in modern memory history where selling was catastrophically wrong was the mid-1990s, which Gavin Baker calls the last true capacity cycle in memory. In that cycle, demand was structurally exploding as the internet era required entirely new computing infrastructure to be built from scratch, and memory had to scale with it in a way that had never happened before. His point is that AI may be that same kind of cycle and not a normal boom bust but a once in a generation capacity buildout where the underlying demand is structural, not cyclical. The reason this argument holds weight is the fundamental shift in what memory is in the AI era. Traditional DRAM was a pure commodity, identical specs, interchangeable suppliers, price determined entirely by supply and demand swings. HBM is the opposite because it is custom engineered to fit a specific customer's chip, co-designed between the memory maker and the GPU designer, with SK Hynix's Vice President literally describing it as shifting from a commodity to a customer-tailored custom business. A single Blackwell Ultra GPU now requires up to 288GB of HBM3E, a 3.6x increase over the H100 and major suppliers like SK Hynix and Micron have already sold out their entire HBM production capacity through the end of the year. Because HBM requires advanced packaging processes like CoWoS that can't be spun up overnight, the bottleneck isn't just wafer capacity but rather runs across the entire manufacturing stack. Bank of America projects the global HBM market grows 58% this year alone to $54.6 billion, and Nomura expects the broader memory sector to nearly double to $445 billion. This is exactly why we’re still tracking memory so closely at Milk Road. HBM is becoming less of a commodity and more of a critical custom component for AI, and we think that changes how this cycle should be valued. If you want to see what we’re actually buying and trading around this theme, come join us for just $1 using the link:

Milk Road AI

30,831 Aufrufe • vor 2 Monaten