Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀 Decode speed: 17.9 tokens/sec 🔥 Memory used: ~ 760GB 👀 Again keep in mind it's a preliminary PR by super Pedro Cuenca still a WIP!

Ivan Fioravanti ᯅ

40,291 subscribers

34,175 görüntüleme • 2 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

MLX GLM 5.2 Distributed on two M3 Ultra 512GB 🔥 One M3 Ultra: 18.8 tokens/sec Two M3 Ultra: 23.4 tokens/sec Context: - PR by Pedro Cuenca is still open and probably there is room for improvement: - basic generation test to measure decoding performance here, I will do a full context benchmarking once PR is more mature - nvfp4 quantization used - Video alternates standard speed and x20, with one Mac first and distributed later. Enjoy! 🙌🏻

MLX GLM 5.2 Distributed on two M3 Ultra 512GB 🔥 One M3 Ultra: 18.8 tokens/sec Two M3 Ultra: 23.4 tokens/sec Context: - PR by Pedro Cuenca is still open and probably there is room for improvement: - basic generation test to measure decoding performance here, I will do a full context benchmarking once PR is more mature - nvfp4 quantization used - Video alternates standard speed and x20, with one Mac first and distributed later. Enjoy! 🙌🏻

Ivan Fioravanti ᯅ

87,805 görüntüleme • 2 ay önce

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.

Ivan Fioravanti ᯅ

327,687 görüntüleme • 8 ay önce

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 görüntüleme • 6 ay önce

GLM 4.6 runs quite fast on an M3 Ultra with mlx-lm even at higher precision. Pretty remarkable that it benchmarks competitive to the just-released Sonnet 4.5. Hope those benchmarks hold-up in day-to-day use. Here's a run using 5.5 bpw quantized model, generating 5.3k tokens at 17+ tok/sec using 244 GB. What prompts should I test?

GLM 4.6 runs quite fast on an M3 Ultra with mlx-lm even at higher precision. Pretty remarkable that it benchmarks competitive to the just-released Sonnet 4.5. Hope those benchmarks hold-up in day-to-day use. Here's a run using 5.5 bpw quantized model, generating 5.3k tokens at 17+ tok/sec using 244 GB. What prompts should I test?

Awni Hannun

68,539 görüntüleme • 11 ay önce

I did it! It works! Using GLM-4.7-4bit with mlx_lm.server and opencode to fix real code locally! 🔥 Here single M3 Ultra 512GB, nex step phase will be 2 using Tensor Parallelism and then apply same changes to exo. Prefill is slow on a single machine, but generation is good.

I did it! It works! Using GLM-4.7-4bit with mlx_lm.server and opencode to fix real code locally! 🔥 Here single M3 Ultra 512GB, nex step phase will be 2 using Tensor Parallelism and then apply same changes to exo. Prefill is slow on a single machine, but generation is good.

Ivan Fioravanti ᯅ

44,000 görüntüleme • 8 ay önce

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 görüntüleme • 1 yıl önce

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 görüntüleme • 29 gün önce

1,000 tok/s, SWE-1.7 is near-frontier intelligence at real-time speed. Building this way is a new category of DX. A task that used to justify walking away finishes before you've mentally moved on. There's a weird middle mode now: it's technically async, but fast enough that you just... watch. In this example I'm running it head to head with GLM-5.2 Max. • SWE-1.7 is now free in Devin CLI or Desktop • SWE-1.7 Fast (Lightning mode) is $2.5 / MTok In · $12.5 / MTok Out

1,000 tok/s, SWE-1.7 is near-frontier intelligence at real-time speed. Building this way is a new category of DX. A task that used to justify walking away finishes before you've mentally moved on. There's a weird middle mode now: it's technically async, but fast enough that you just... watch. In this example I'm running it head to head with GLM-5.2 Max. • SWE-1.7 is now free in Devin CLI or Desktop • SWE-1.7 Fast (Lightning mode) is $2.5 / MTok In · $12.5 / MTok Out

nader dabit

178,708 görüntüleme • 2 ay önce

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!

Ivan Fioravanti ᯅ

12,150 görüntüleme • 17 gün önce

Memory Skill for OpenClaw with 26k+ users in 1 week🚀 OpenClaw's memory system is broken by default. It requires curating massive MEMORY.md files or relying on duplicate-heavy generation. Hours are wasted tuning, and massive amounts of tokens are burned. It's time to stop. So we built the memory skill to solve that prob Here is our superpower ⚔️ 🎯 Top #1 market accuracy (92.19%) after 8+ months of intense architecture iteration 🧠 The ultimate solution to keep the timeline, facts, and meaning perfectly in place ☁️ Local & Cloud + Version control ⚡ Super easy setup

Memory Skill for OpenClaw with 26k+ users in 1 week🚀 OpenClaw's memory system is broken by default. It requires curating massive MEMORY.md files or relying on duplicate-heavy generation. Hours are wasted tuning, and massive amounts of tokens are burned. It's time to stop. So we built the memory skill to solve that prob Here is our superpower ⚔️ 🎯 Top #1 market accuracy (92.19%) after 8+ months of intense architecture iteration 🧠 The ultimate solution to keep the timeline, facts, and meaning perfectly in place ☁️ Local & Cloud + Version control ⚡ Super easy setup

andy nguyen

589,836 görüntüleme • 6 ay önce

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 görüntüleme • 4 ay önce

🚨 We finally cracked it. GLM-5.3, the FULL model (not the Flash), running on 4 DGX Sparks 🔥 ⚡️ 35 tok/s on real coding, peaking near 40 when it's hot 📊 Full C1 to C6 benchmark done, same speeds we were hitting on 5.2 🧠 200K context right now, tuning it higher from here 🎯 And this is only MTP4... we haven't even added DFlash yet I quantized it myself and I'm uploading the repo RIGHT NOW so everyone can go tune it with me 👀 Be tuned, it drops any minute. Let's get it 🚀

🚨 We finally cracked it. GLM-5.3, the FULL model (not the Flash), running on 4 DGX Sparks 🔥 ⚡️ 35 tok/s on real coding, peaking near 40 when it's hot 📊 Full C1 to C6 benchmark done, same speeds we were hitting on 5.2 🧠 200K context right now, tuning it higher from here 🎯 And this is only MTP4... we haven't even added DFlash yet I quantized it myself and I'm uploading the repo RIGHT NOW so everyone can go tune it with me 👀 Be tuned, it drops any minute. Let's get it 🚀

Tech2Wild

13,377 görüntüleme • 9 gün önce

timelapse #162 (17 hrs) - Idk what it is, grok makes it super easy to get into flow state. No model does this for me - Went deep into spec decoding architecture learning with grok - Continued Minecraft reimplementation in C. The biggest bottleneck is samples to verify against (capturing my own gameplay for a few mins rn) as well as fly wheel speed for an agent to iterate. was bottlenecked by replay and compile, now bottlenecked by tokens. this is the hardest task ive given an llm to date. - Contract work - Getting tinygrad’s glm 5.2 at c=1 to be stable - more meetings

timelapse #162 (17 hrs) - Idk what it is, grok makes it super easy to get into flow state. No model does this for me - Went deep into spec decoding architecture learning with grok - Continued Minecraft reimplementation in C. The biggest bottleneck is samples to verify against (capturing my own gameplay for a few mins rn) as well as fly wheel speed for an agent to iterate. was bottlenecked by replay and compile, now bottlenecked by tokens. this is the hardest task ive given an llm to date. - Contract work - Getting tinygrad’s glm 5.2 at c=1 to be stable - more meetings

Elliot Arledge

48,116 görüntüleme • 1 ay önce

One shot prompt with no loops: GPT-Sol Ultra: 35,192,545 tokens used Opus 5.0 Ultracode: 1,171,676,041 tokens used Fable 5.1 Ultracode: 127,366,207 tokensused Same prompt… different results: Fable 5.1 runs at 5fps so I had to speed it up to demonstrate the rendering. Would need further optimisation. Opus 5.0 runs very nicely on first prompt. Doesn't need much in terms of performance optimisation. Just a few odd visual things here and there. My verdict: Fable looks impressive but struggles to run smoothly. Opus is a solid model when prompted properly. Sol is garbage.

One shot prompt with no loops: GPT-Sol Ultra: 35,192,545 tokens used Opus 5.0 Ultracode: 1,171,676,041 tokens used Fable 5.1 Ultracode: 127,366,207 tokensused Same prompt… different results: Fable 5.1 runs at 5fps so I had to speed it up to demonstrate the rendering. Would need further optimisation. Opus 5.0 runs very nicely on first prompt. Doesn't need much in terms of performance optimisation. Just a few odd visual things here and there. My verdict: Fable looks impressive but struggles to run smoothly. Opus is a solid model when prompted properly. Sol is garbage.

Adam Lyttle

579,446 görüntüleme • 2 gün önce

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

271,839 görüntüleme • 2 ay önce

"In my mind, it's still second place" - Justin Gaethje says he isn’t satisfied with being interim champion 😲👀 “I’m not satisfied because it is, in my mind, still second place. However, I do know it’s a great achievement. In history, I’m gonna be a two-time interim champion, which on paper makes me a champion. In this one specific area, I have to be pessimistic and not be content because, obviously, same as last time.” (Via Justin Gaethje 🇺🇸 )

"In my mind, it's still second place" - Justin Gaethje says he isn’t satisfied with being interim champion 😲👀 “I’m not satisfied because it is, in my mind, still second place. However, I do know it’s a great achievement. In history, I’m gonna be a two-time interim champion, which on paper makes me a champion. In this one specific area, I have to be pessimistic and not be content because, obviously, same as last time.” (Via Justin Gaethje 🇺🇸 )

Red Corner MMA

47,703 görüntüleme • 7 ay önce

Engineer runs a Kimi K3 memory layer that costs $11 a month and remembers what a $500,000 vector database keeps losing. No embeddings. Four nodes and one rule about what's allowed to be forgotten. He published the whole schema. His version starts from the opposite idea. Memory is not a pile you search. It's a set of claims that expire unless something keeps paying to keep them. Four nodes. Every memory carries a clock someone has to reset: > WRITER - stores a fact with the reason it mattered, never raw text > DECAY - ages every memory down. Silence is deletion > RENEWER - only re-lifts a memory the model actually used again > GRAVE - holds what died, and why nobody reached for it Three nodes keep memory alive. One keeps the dead ones. Recall isn't storage here. It's rent a fact has to keep earning. That's the entire design. When everything is remembered forever, the useful and the stale retrieve identically. He replayed two months of agent context. 90,000 stored facts. 71,000 never retrieved once. The vector store returned all of them on similarity. Similarity graded closeness. Nobody graded whether the memory was ever right. Everyone else stuffs more into the context window and calls it memory. He built a layer that lets a fact die unless it keeps proving itself. The cost isn't storage. It's finding out how much of what your agent "knows" it has never once used. The article below is the full build - node prompts, the decay curve, the renewal rule. Save it. You'll want it open in the other tab.

Engineer runs a Kimi K3 memory layer that costs $11 a month and remembers what a $500,000 vector database keeps losing. No embeddings. Four nodes and one rule about what's allowed to be forgotten. He published the whole schema. His version starts from the opposite idea. Memory is not a pile you search. It's a set of claims that expire unless something keeps paying to keep them. Four nodes. Every memory carries a clock someone has to reset: > WRITER - stores a fact with the reason it mattered, never raw text > DECAY - ages every memory down. Silence is deletion > RENEWER - only re-lifts a memory the model actually used again > GRAVE - holds what died, and why nobody reached for it Three nodes keep memory alive. One keeps the dead ones. Recall isn't storage here. It's rent a fact has to keep earning. That's the entire design. When everything is remembered forever, the useful and the stale retrieve identically. He replayed two months of agent context. 90,000 stored facts. 71,000 never retrieved once. The vector store returned all of them on similarity. Similarity graded closeness. Nobody graded whether the memory was ever right. Everyone else stuffs more into the context window and calls it memory. He built a layer that lets a fact die unless it keeps proving itself. The cost isn't storage. It's finding out how much of what your agent "knows" it has never once used. The article below is the full build - node prompts, the decay curve, the renewal rule. Save it. You'll want it open in the other tab.

wast3

67,246 görüntüleme • 9 gün önce

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 görüntüleme • 2 yıl önce

tencent hunyuan just shipped their third major model in six months. that pace is hard to keep up with. Hy4 preview is now live in workbuddy, and it's fully open source. the specs: 770B total params, 49B active, 1M+ token context window. but the number that matters is where it's pointed — real work. coding, office tasks, research, documents. here's the part i find interesting. tencent didn't build this in isolation. they put the training data together with their own experts across software engineering, gaming, finance and security, then designed the model side by side with products like workbuddy. model and product shaping each other instead of one being bolted onto the other. and it shows up in testing. 163 internal experts ran a blind test across 203 engineering tasks. hy4 preview scored 2.99/4 — ahead of kimi k3 at 2.94 and glm 5.3 at 2.92. on public benchmarks it clearly beats glm 5.2 and sits about level with glm 5.3. then the pricing. $6 per million input tokens, $18 per million output, $0.3 per million on cache hits. flagship results at a price you don't have to think about. and for the next two weeks it's free inside workbuddy.

tencent hunyuan just shipped their third major model in six months. that pace is hard to keep up with. Hy4 preview is now live in workbuddy, and it's fully open source. the specs: 770B total params, 49B active, 1M+ token context window. but the number that matters is where it's pointed — real work. coding, office tasks, research, documents. here's the part i find interesting. tencent didn't build this in isolation. they put the training data together with their own experts across software engineering, gaming, finance and security, then designed the model side by side with products like workbuddy. model and product shaping each other instead of one being bolted onto the other. and it shows up in testing. 163 internal experts ran a blind test across 203 engineering tasks. hy4 preview scored 2.99/4 — ahead of kimi k3 at 2.94 and glm 5.3 at 2.92. on public benchmarks it clearly beats glm 5.2 and sits about level with glm 5.3. then the pricing. $6 per million input tokens, $18 per million output, $0.3 per million on cache hits. flagship results at a price you don't have to think about. and for the next two weeks it's free inside workbuddy.

Fakhr

55,316 görüntüleme • 9 gün önce

Starting off 2026 with a massive new drop. 🔊Turn up the sound. 👀Watch the video. Introducing MECHFX - ultra premium dice and holographic tokens compatible with the Gundam Card Game. This is my most ambitious project yet and DuelSpec’s first #Kickstarter. There are only 20 Super Early Pilot tiers and it's already going quickly. Have a store? We have bulk tiers available for vendors as well. 🎁 GIVEAWAY: Follow, Like & Repost your favorite Gundam for a chance to win a MECHFX set FREE! 👉 Back it on KICKSTARTER here: 🔗

Starting off 2026 with a massive new drop. 🔊Turn up the sound. 👀Watch the video. Introducing MECHFX - ultra premium dice and holographic tokens compatible with the Gundam Card Game. This is my most ambitious project yet and DuelSpec’s first #Kickstarter. There are only 20 Super Early Pilot tiers and it's already going quickly. Have a store? We have bulk tiers available for vendors as well. 🎁 GIVEAWAY: Follow, Like & Repost your favorite Gundam for a chance to win a MECHFX set FREE! 👉 Back it on KICKSTARTER here: 🔗

DuelSpec

23,583 görüntüleme • 8 ay önce