Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Today we are shipping dnet, a distributed inference framework that lets Apple Silicon clusters run models that exceed their physical memory. We fuse pipelined-ring parallelism, disk streaming and UMA-aware scheduling so “out of memory” stops being the limit.

Dria

14,142 subscribers

26,159 Aufrufe • vor 11 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

14 Kommentare

Profilbild von Dria

Driavor 11 Monaten

Local AI today is great for short generations, but breaks down when you want big models, long context, high throughput, and heterogeneous devices. dnet addresses that gap head-on.

Profilbild von Dria

Driavor 11 Monaten

Built on @Apple MLX, dnet focuses on: --> Distributed execution strategies --> Automatic device/model profiling --> A heterogeneity-aware solver --> A drop-in OpenAI-style API.

Profilbild von Dria

Driavor 11 Monaten

Under the hood, every run goes through: Strategy → Profiling → Scheduling —> Strategy: Choose how to distribute work (pipelined ring, tensor-parallel, long-context) —> Profiling: Measure FLOPs, memory, KV cache, latency, and disk to know each device's limits —> Scheduling: Solve layer/tensor placement that fits mixed devices and avoids stalls

Profilbild von Dria

Driavor 11 Monaten

In alpha, we ship a pipelined-ring strategy inspired by PRIMA.CPP. dnet’s solver (distilp) extends it so devices can punch above memory: layers stream from disk mid-round and overlap with compute, so total model size can exceed total cluster RAM.

Profilbild von Dria

Driavor 11 Monaten

Apple Silicon’s unified memory is a blessing and a trap. CPU and GPU share one pool; naïve mmap will overcommit and start swapping. dnet is UMA-aware: memory pressure buffers, Apple-specific loaders, and repacked weights keep the ring moving instead of thrashing.

Profilbild von Dria

Driavor 11 Monaten

dnet is designed as a plugin architecture: --Solver ---API Adapter ----Topology Adapter To add a new strategy (for example tensor parallel), you implement a solver in distilp and an adapter in dnet. Runtime, KV cache, and API stay the same.

Profilbild von Dria

Driavor 11 Monaten

What’s next: -->128K context on home clusters --> higher throughput with faster comms and RDMA --> a unified backend where Apple Silicon, NVIDIA, and AMD share a single cluster and scheduling layer. Local AI without datacenter assumptions.

Profilbild von Dria

Driavor 11 Monaten

We’re building dnet because we think local clusters will be the backbone for agentic workloads. If you want to run frontier-scale models from your desk, start with the alpha: 🔗

Profilbild von gustofied

gustofiedvor 11 Monaten

top top work. I guess future would be webgpu type stuff to enable any device

Profilbild von Georg S. Kuklick

Georg S. Kuklickvor 11 Monaten

@awnihannun Nice

Profilbild von Team Reagent

Team Reagentvor 11 Monaten

Yo this is huge! We're working on something similar for a grant for self-driving vehicles, but this is huge work and should not be understated, fantastic work guys! Is it gonna be open source?

Profilbild von dmsimon

dmsimonvor 11 Monaten

This seems far more plausible @Scobleizer

Profilbild von Gilfoyle

Gilfoylevor 10 Monaten

How does it compare with @exolabs ?

Profilbild von BeastTitanHunter

BeastTitanHuntervor 9 Monaten

holy

Ähnliche Videos

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 Aufrufe • vor 5 Monaten

Kimi K3 is too large to run efficiently on a single accelerator so the workload must be distributed across many chips that exchange data at extremely high speeds. $CRDO and $ALAB benefit from the high speed connectivity inside AI clusters while $MRVL and $AVGO provide the networking, optical and custom silicon infrastructure required to scale them. Thats why inference constraints are bullish for the entire surrounding system including memory, networking, optics, switches, power, cooling and orchestration software.

Kimi K3 is too large to run efficiently on a single accelerator so the workload must be distributed across many chips that exchange data at extremely high speeds. $CRDO and $ALAB benefit from the high speed connectivity inside AI clusters while $MRVL and $AVGO provide the networking, optical and custom silicon infrastructure required to scale them. Thats why inference constraints are bullish for the entire surrounding system including memory, networking, optics, switches, power, cooling and orchestration software.

Shay Boloor

84,649 Aufrufe • vor 3 Monaten

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.

Sean Hannity 🇺🇸

69,245 Aufrufe • vor 4 Monaten

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."

Fireside Alpha

185,279 Aufrufe • vor 2 Monaten

Google Research has open-sourced Harness², a framework that enables AI agents to improve their own prompts, tools, memory, and workflows. It tests different configurations and carries useful lessons into future tasks while keeping the underlying model unchanged. An increasing number of releases concerning self-improving models are being published.

Google Research has open-sourced Harness², a framework that enables AI agents to improve their own prompts, tools, memory, and workflows. It tests different configurations and carries useful lessons into future tasks while keeping the underlying model unchanged. An increasing number of releases concerning self-improving models are being published.

Chubby♨️

118,936 Aufrufe • vor 5 Tagen

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,604 Aufrufe • vor 1 Jahr

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

37,459 Aufrufe • vor 3 Monaten

August 29 is a very important day, a special day. Today, we honor the memory of the heroes, the fallen heroes of Ukraine, all our defenders, who have been fighting in the full-scale war since 2022 and in the vile war, hybrid war – Russia’s war against Ukraine, against the Ukrainian people – since 2014. Now, when thousands and thousands of our warriors are defending Ukraine’s position on the front, when our people are being killed in Russian strikes, when we are doing everything to drive this war out of Ukraine and into Russia, primarily through our long-range strikes, we remember that all of this is our active action, all of this is our memory living on. Our active memory. The memory of the deeds of Ukrainian heroes, of the deeds of every man and woman who did not surrender to Russia, who is fighting for Ukraine’s right to be itself. We honor Ukrainian warriors, we honor the courage of the Ukrainian people, we remember everyone who gave their life for Ukraine, so that Ukraine could live. Thank you. We thank our fallen heroes. We thank all the warriors of our state.

August 29 is a very important day, a special day. Today, we honor the memory of the heroes, the fallen heroes of Ukraine, all our defenders, who have been fighting in the full-scale war since 2022 and in the vile war, hybrid war – Russia’s war against Ukraine, against the Ukrainian people – since 2014. Now, when thousands and thousands of our warriors are defending Ukraine’s position on the front, when our people are being killed in Russian strikes, when we are doing everything to drive this war out of Ukraine and into Russia, primarily through our long-range strikes, we remember that all of this is our active action, all of this is our memory living on. Our active memory. The memory of the deeds of Ukrainian heroes, of the deeds of every man and woman who did not surrender to Russia, who is fighting for Ukraine’s right to be itself. We honor Ukrainian warriors, we honor the courage of the Ukrainian people, we remember everyone who gave their life for Ukraine, so that Ukraine could live. Thank you. We thank our fallen heroes. We thank all the warriors of our state.

Volodymyr Zelenskyy / Володимир Зеленський

294,993 Aufrufe • vor 1 Monat

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.

Sarvam AI

16,310 Aufrufe • vor 10 Monaten

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.

Jon Saad-Falcon

159,915 Aufrufe • vor 4 Monaten

$SNDK CEO David Goeckeler says inference is a memory-bound problem and HBF could break through some bottlenecks, so it's an enormous opportunity as inference scales, which they'll talk more at their analyst day next week. "Look, there's a ton of innovation going on right now in inference memory architectures, which we think is fantastic, very healthy." "There's a lot of different ideas. We're going to dive into this a little deeper next week at our analyst day…" "But yes, all these are opportunities for us. We think especially inference is a memory-bound problem. Storage is extraordinarily important to the equation." "We showed some stuff in our FMS keynote just a couple of hours ago about how, when you use HBF, we simulate performance and maybe break through some bottlenecks…" "As we scale inference, we think it provides an enormous opportunity. The way we're thinking about this is staying very close to our customers, because they're going to be the ones that define what the architecture is in the future."

$SNDK CEO David Goeckeler says inference is a memory-bound problem and HBF could break through some bottlenecks, so it's an enormous opportunity as inference scales, which they'll talk more at their analyst day next week. "Look, there's a ton of innovation going on right now in inference memory architectures, which we think is fantastic, very healthy." "There's a lot of different ideas. We're going to dive into this a little deeper next week at our analyst day…" "But yes, all these are opportunities for us. We think especially inference is a memory-bound problem. Storage is extraordinarily important to the equation." "We showed some stuff in our FMS keynote just a couple of hours ago about how, when you use HBF, we simulate performance and maybe break through some bottlenecks…" "As we scale inference, we think it provides an enormous opportunity. The way we're thinking about this is staying very close to our customers, because they're going to be the ones that define what the architecture is in the future."

Fireside Alpha

22,024 Aufrufe • vor 2 Monaten

A place of remembrance and deep reverence. Here are the names of those who gave their lives for our independence and freedom. Today, together with First Lady Olena Zelenska / Олена Зеленська and Prime Minister of Canada Mark Carney, we honored the fallen defenders of Ukraine at the Wall of Remembrance. We cherish the memory of each and every one of them. We honor their heroism and everything they did so that Ukraine may live. Eternal gratitude and glory to the heroes.

A place of remembrance and deep reverence. Here are the names of those who gave their lives for our independence and freedom. Today, together with First Lady Olena Zelenska / Олена Зеленська and Prime Minister of Canada Mark Carney, we honored the fallen defenders of Ukraine at the Wall of Remembrance. We cherish the memory of each and every one of them. We honor their heroism and everything they did so that Ukraine may live. Eternal gratitude and glory to the heroes.

Volodymyr Zelenskyy / Володимир Зеленський

173,461 Aufrufe • vor 1 Jahr

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

Fireside Alpha

161,722 Aufrufe • vor 3 Monaten

Mission Report is out! We have a lot of product highlights and updates this episode. Also we've been shipping models so fast that this was filmed before Gemini 3.8 Flash was ready and announced. We will cover that model in the next Mission Report.

Mission Report is out! We have a lot of product highlights and updates this episode. Also we've been shipping models so fast that this was filmed before Gemini 3.8 Flash was ready and announced. We will cover that model in the next Mission Report.

Google Antigravity

87,205 Aufrufe • vor 1 Monat

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 Aufrufe • vor 1 Jahr

Ghostty is getting automatic scrollback compression, resulting in 70 to 90% less physical memory usage. It happens incrementally when idle, so it had no measurable effect on IO throughput. I'm not aware of any other mainstream terminal that does this. Demo video below! The gains let us increase the default scrollback limit from 10MB to 50MB, because on average a full scrollback will still compress smaller than the prior limit. More history, for free. ("Unlimited", disk-paged history is on the roadmap too) Let's talk about cool implementation details, cause this was fun. First, the data structure and memory layout ("PageList") I wrote two years ago finally pays off! One of its traits is that screen memory is backed by a linked list of page-aligned, page-sized (or page-multiple-sized) blocks. Because each block is page-aligned and page-sized, we can use madvise to discard its physical backing while keeping the virtual address space reserved. Compressed pages therefore disappear from resident memory, but decompression is still guaranteed because the address space remains valid and we simply fault new pages back in as needed. We use the same trick for our memory pools, too. Unallocated pool pages don't count as resident memory, saving another couple of MB per terminal. This functionality is also available to libghostty-vt consumers via new `ghostty_terminal_compress` APIs. The consumer decides when the appropriate time to compress is and the APIs advise on compressability.

Ghostty is getting automatic scrollback compression, resulting in 70 to 90% less physical memory usage. It happens incrementally when idle, so it had no measurable effect on IO throughput. I'm not aware of any other mainstream terminal that does this. Demo video below! The gains let us increase the default scrollback limit from 10MB to 50MB, because on average a full scrollback will still compress smaller than the prior limit. More history, for free. ("Unlimited", disk-paged history is on the roadmap too) Let's talk about cool implementation details, cause this was fun. First, the data structure and memory layout ("PageList") I wrote two years ago finally pays off! One of its traits is that screen memory is backed by a linked list of page-aligned, page-sized (or page-multiple-sized) blocks. Because each block is page-aligned and page-sized, we can use madvise to discard its physical backing while keeping the virtual address space reserved. Compressed pages therefore disappear from resident memory, but decompression is still guaranteed because the address space remains valid and we simply fault new pages back in as needed. We use the same trick for our memory pools, too. Unallocated pool pages don't count as resident memory, saving another couple of MB per terminal. This functionality is also available to libghostty-vt consumers via new `ghostty_terminal_compress` APIs. The consumer decides when the appropriate time to compress is and the APIs advise on compressability.

Mitchell Hashimoto

167,505 Aufrufe • vor 3 Monaten

The United States of America is a generous country and we take care of our own. It's disgraceful that fraudsters out there are taking advantage of programs like Medicaid. That stops today. Dr. Oz CMS

The United States of America is a generous country and we take care of our own. It's disgraceful that fraudsters out there are taking advantage of programs like Medicaid. That stops today. Dr. Oz CMS

JD Vance

990,010 Aufrufe • vor 8 Monaten