Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

Abdur Rahim

1,341 subscribers

107,726 просмотров • 11 дней назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 29

Фото профиля Abdur Rahim

Abdur Rahim11 дней назад

Kaggle Notebook:

Фото профиля Rahman Qureshi

Rahman Qureshi10 дней назад

@UnslothAI @Zai_org @_ARahim_ Rahim bhai, your GLM-5.3-Flash TPU notebook is fire 🔥 Can you please make one for uncensored models too? Especially Qwen3.8-27B and an uncensored version of GLM-5.3 Flash. Would be super useful! 🙏

Фото профиля JeffersonNunn.eth

JeffersonNunn.eth10 дней назад

@UnslothAI @Zai_org I, uh, wait what? Like … these are words that shouldn’t go together. WOW

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org Thanks ✌🏻

Фото профиля Karol Olszacki

Karol Olszacki10 дней назад

@UnslothAI @Zai_org Finally a good model running on Google's TPUs 😅

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org Thanks. In case you missed it, Qwen3.8-27B in bf16 also runs super fast.

Фото профиля Vasko

Vasko10 дней назад

@UnslothAI @Zai_org guys someone hire this guy cuz theres no fucking way this is so cool

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org Haha, thanks.

Фото профиля 0xSero

0xSero10 дней назад

@UnslothAI @Zai_org Wow

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org Thank you 😍

Фото профиля man wei

man wei10 дней назад

@UnslothAI @Zai_org This is wild. Just saw the notebook is public — 320B MoE on free Kaggle TPU. Forking this weekend to try it out. Thanks for sharing ~

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org Thanks. I'd appreciate any feedback you might have after trying it.

Фото профиля Dominik Gstöhl

Dominik Gstöhl10 дней назад

@UnslothAI @Zai_org bro thinking around 2 corners

Фото профиля internet user

internet user10 дней назад

@UnslothAI @Zai_org Can it run DSv4.1 Flash?

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org To my knowledge, not yet. But I am working on that. My TPU quota for this week is gone! So I will resume working on that next week. Let's hope for the best.

Фото профиля Squiggles

Squiggles10 дней назад

@UnslothAI @Zai_org Based based based

Фото профиля Jigs

Jigs10 дней назад

@UnslothAI @Zai_org Wow, that's cool

Фото профиля Emircan ERKUL

Emircan ERKUL10 дней назад

@UnslothAI @Zai_org How long does their free credit last for this? 10m?

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org It's not "free credits". It's a fixed 20 hours per week for a verified Kaggle account.

Фото профиля Emircan ERKUL

Emircan ERKUL10 дней назад

@UnslothAI @Zai_org Aha 20h/w is really good. I'll try

Фото профиля Yicheng Xia

Yicheng Xia10 дней назад

@UnslothAI @Zai_org No wonder Kaggle TPU queue is longer than ever these days

Фото профиля Abdul Rehman

Abdul Rehman10 дней назад

@UnslothAI @Zai_org W

Фото профиля Lemon🇹🇷🇵🇰🇵🇸🇸🇩

Lemon🇹🇷🇵🇰🇵🇸🇸🇩10 дней назад

@UnslothAI @Zai_org Crazy

Фото профиля thaypley

thaypley10 дней назад

@UnslothAI @Zai_org oh wow.. on my way to check this out!

Фото профиля Petr Glaser

Petr Glaser10 дней назад

@UnslothAI @Zai_org Would be great to have abliterated models or cyber-specialised models there! Do I need to do anything specific with HF model to let it run there?

Фото профиля No Body

No Body10 дней назад

@UnslothAI @Zai_org wtf this sounds cool why am i a rookie and never never heard of this

Фото профиля Rahman Qureshi

Rahman Qureshi10 дней назад

@UnslothAI @Zai_org You are crazy man

Фото профиля chotu

chotu10 дней назад

@UnslothAI @Zai_org Free tpu for everyone ?

Фото профиля Abdur Rahim

Abdur Rahim10 дней назад

@UnslothAI @Zai_org Yes, if you have a verified Kaggle account, you'll have 20 hours of TPU usage per week, free.

Похожие видео

You can now run Qwen3.8-27B in full BF16 on a free Kaggle TPU. - with about 130 tok/s decode, - 10k tok/s prefill, - Native 262k context. - No quantization. - No paid GPU. - Measured on TPU v5e-8 It also exposes an OpenAI-compatible endpoint, so it plugs into Claude Code, Codex, and OpenCode.

You can now run Qwen3.8-27B in full BF16 on a free Kaggle TPU. - with about 130 tok/s decode, - 10k tok/s prefill, - Native 262k context. - No quantization. - No paid GPU. - Measured on TPU v5e-8 It also exposes an OpenAI-compatible endpoint, so it plugs into Claude Code, Codex, and OpenCode.

Md Ismail Šojal 🕷️

50,394 просмотров • 19 дней назад

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

FHILY👑

35,794 просмотров • 20 дней назад

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

SGLang

99,353 просмотров • 29 дней назад

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

233,855 просмотров • 2 дней назад

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

39,811 просмотров • 23 дней назад

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

atomic.chat

32,515 просмотров • 23 дней назад

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

h100envy

496,829 просмотров • 2 месяцев назад

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

34,141 просмотров • 4 месяцев назад

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

311,363 просмотров • 2 месяцев назад

Apple’s “LLM in a Flash” is definitely worth checking out. Going to 2-bit for the shared-expert MLP means disk I/O is no longer dominant. 14–15 tok/s from SSD is still wild for a ~400B MoE model streamed from storage. Qwen3.5-397B-A17B Credit: Dan Woods

Apple’s “LLM in a Flash” is definitely worth checking out. Going to 2-bit for the shared-expert MLP means disk I/O is no longer dominant. 14–15 tok/s from SSD is still wild for a ~400B MoE model streamed from storage. Qwen3.5-397B-A17B Credit: Dan Woods

Anemll

35,664 просмотров • 6 месяцев назад

glm 5.3 flash is 7.5x cheaper, but 3.4x slower than gemini 3.7 flash Z.ai glm 5.3 flash – shipped aug 26, $0.07/$0.25 per 1m Google DeepMind gemini 3.7 flash – shipped aug 13, $0.38/$1.88 per 1m we put the two models on one job: write one html file that draws an animated 3d scene in the browser. no images, no downloads, and it has to look the same on every load. the setup: three scenes – a glass aquarium in a lit room, the solar system, a night city under a thunderstorm. identical brief word for word, reasoning effort high, 64k output cap. the numbers below are not the whole run. they cover the three scenes we kept – the best one per task from each model, the ones in the video. - total generation time for the three scenes #1 gemini 3.7 flash – 10m 36s #2 glm 5.3 flash – 36m 30s - tokens spent on those three scenes #1 glm 5.3 flash – 110k #2 gemini 3.7 flash – 111k - cost of those three scenes #1 glm 5.3 flash – $0.027 #2 gemini 3.7 flash – $0.202 observations: • glm's first 10 attempts: 7 blank pages. it kept inventing short random helpers and forgetting to define one of them. the fix was one line in the brief: use exactly one random helper, named rand(), and don't invent shorthands next to it. next 12 attempts: 11 alive, 0 crashes. • glm spends 66% of its output on reasoning, gemini 57%. that is the whole speed gap. • gemini's storm came back as a black rectangle in 4 of 6 runs. glm's best storm has a branching bolt, lit rain and wet asphalt – for $0.01. conclusion: same three scenes, same token spend – glm 5.3 flash billed $0.027 and took 36m 30s, gemini 3.7 flash billed $0.202 and took 10m 36s. glm wins gemini on price and made the best storm of the whole run follow thehype. for 24/7 ai news, analysis and breakdowns

glm 5.3 flash is 7.5x cheaper, but 3.4x slower than gemini 3.7 flash Z.ai glm 5.3 flash – shipped aug 26, $0.07/$0.25 per 1m Google DeepMind gemini 3.7 flash – shipped aug 13, $0.38/$1.88 per 1m we put the two models on one job: write one html file that draws an animated 3d scene in the browser. no images, no downloads, and it has to look the same on every load. the setup: three scenes – a glass aquarium in a lit room, the solar system, a night city under a thunderstorm. identical brief word for word, reasoning effort high, 64k output cap. the numbers below are not the whole run. they cover the three scenes we kept – the best one per task from each model, the ones in the video. - total generation time for the three scenes #1 gemini 3.7 flash – 10m 36s #2 glm 5.3 flash – 36m 30s - tokens spent on those three scenes #1 glm 5.3 flash – 110k #2 gemini 3.7 flash – 111k - cost of those three scenes #1 glm 5.3 flash – $0.027 #2 gemini 3.7 flash – $0.202 observations: • glm's first 10 attempts: 7 blank pages. it kept inventing short random helpers and forgetting to define one of them. the fix was one line in the brief: use exactly one random helper, named rand(), and don't invent shorthands next to it. next 12 attempts: 11 alive, 0 crashes. • glm spends 66% of its output on reasoning, gemini 57%. that is the whole speed gap. • gemini's storm came back as a black rectangle in 4 of 6 runs. glm's best storm has a branching bolt, lit rain and wet asphalt – for $0.01. conclusion: same three scenes, same token spend – glm 5.3 flash billed $0.027 and took 36m 30s, gemini 3.7 flash billed $0.202 and took 10m 36s. glm wins gemini on price and made the best storm of the whole run follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

15,997 просмотров • 1 месяц назад

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.

Alexey Fateev

19,218 просмотров • 1 месяц назад

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

Sudo su

31,174 просмотров • 7 дней назад

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

Anemll

29,421 просмотров • 5 месяцев назад

running Empero's Qwen3.8-35B-A3B on a single RTX 3060. 12GB VRAM, 16GB RAM. (config below) full 262K context. ~50 tok/s decode, ~550 tok/s prefill. it's Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. Ornith-1.5 is the engine behind my Hermes and Pi agents right now, so the real question is whether this can replace it. next: running it through my agentic coding benchmark against Ornith-1.5-35B, Qwen3.8-27B and base Qwen3.6-35B-A3B. then a decode speed vs context graph. config: llama-server -m Qwen3.8-35B-A3B-Q4_K_M.gguf -ngl 99 --n-cpu-moe 26 -c 262144 -fa on --jinja -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on

running Empero's Qwen3.8-35B-A3B on a single RTX 3060. 12GB VRAM, 16GB RAM. (config below) full 262K context. ~50 tok/s decode, ~550 tok/s prefill. it's Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. Ornith-1.5 is the engine behind my Hermes and Pi agents right now, so the real question is whether this can replace it. next: running it through my agentic coding benchmark against Ornith-1.5-35B, Qwen3.8-27B and base Qwen3.6-35B-A3B. then a decode speed vs context graph. config: llama-server -m Qwen3.8-35B-A3B-Q4_K_M.gguf -ngl 99 --n-cpu-moe 26 -c 262144 -fa on --jinja -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on

Doğukan

93,784 просмотров • 9 дней назад

Darkbloom is the first model provider to support Ternary Bonsai 2 27B -- concentrated intelligence that fits on your phone. Try it now: First 250 users get 100 million free tokens; PrismML's new flagship model: - a ternary compression of Qwen3.8 27B at 2bits per parameter. - 8.5 GB total, 5x smaller than original - keeps 98.2% of the Qwen's FP16 benchmark performance. - 75% cheaper than Qwen 27B. Qwen3.8 27B already operates comparably with Opus 4.6 and 5.6 Luna on certain tasks. This one does it in the memory of a phone. 262K context, image input, Apache 2.0. From our first run on the network, on a single M5 Max with no caching: - 35 tok/s decode at 1K context, - 31 tok/s at 10K, - 19 tok/s at 50K. But we expect more performance gain coming in a few weeks! That's a full 27B reasoning model running comfortably on any Mac. 1,000+ Macs are serving on Darkbloom right now. Go try it out!! Thank you to Babak Hassibi Sahin Lale Omead Pooladzandi Reza Sadri Tushar Bansal Evan Walters and the whole PrismML team. This is exactly the kind of model Darkbloom was built for. You can read the essay by Bonsai on Why Local AI Matters:

Darkbloom is the first model provider to support Ternary Bonsai 2 27B -- concentrated intelligence that fits on your phone. Try it now: First 250 users get 100 million free tokens; PrismML's new flagship model: - a ternary compression of Qwen3.8 27B at 2bits per parameter. - 8.5 GB total, 5x smaller than original - keeps 98.2% of the Qwen's FP16 benchmark performance. - 75% cheaper than Qwen 27B. Qwen3.8 27B already operates comparably with Opus 4.6 and 5.6 Luna on certain tasks. This one does it in the memory of a phone. 262K context, image input, Apache 2.0. From our first run on the network, on a single M5 Max with no caching: - 35 tok/s decode at 1K context, - 31 tok/s at 10K, - 19 tok/s at 50K. But we expect more performance gain coming in a few weeks! That's a full 27B reasoning model running comfortably on any Mac. 1,000+ Macs are serving on Darkbloom right now. Go try it out!! Thank you to Babak Hassibi Sahin Lale Omead Pooladzandi Reza Sadri Tushar Bansal Evan Walters and the whole PrismML team. This is exactly the kind of model Darkbloom was built for. You can read the essay by Bonsai on Why Local AI Matters:

Gajesh

46,273 просмотров • 8 дней назад

i burned through my entire GPT-6 Astra weekly quota in 3 days so i'm replacing my $200/mo Codex subscription with this... a free 32,960-star GitHub repo that runs GLM-5.3 Flash, Kimi K3, and DeepSeek V4 Flash locally [here's the exact setup i used:] 1. get colibri download the release archive for Linux, macOS, or Windows. no compiler needed. python3 coli info 2. or build it from source git clone cd colibri/c ./setup.sh cd .. 3. pick a model GLM-5.3 Flash: 321B, ~195 GB converted, 25 GB RAM DeepSeek V4 Flash: 284B, ~167 GB, or ~85 GB for the REAP 150B version Kimi K3: 2.8T, ~1.6 TB snapshot, 32 GB+ RAM none of them needs a GPU. a GPU only makes some runs faster. 4. get the weights GLM-5.3 Flash uses the dedicated converter: python3 tools/convert_glm53.py --outdir /nvme/glm53_i4 --min-free-gb 30 DeepSeek V4 Flash and Kimi K3 load from their original Hugging Face snapshots. pre-converted containers work too. 5. run it COLI_MODEL=/nvme/glm53_i4 ./coli chat ./coli web --model /nvme/glm53_i4 # dashboard ./coli serve --model /nvme/glm53_i4 # headless API 6. tune it ./coli doctor # readiness check ./coli plan # RAM/VRAM/disk placement ./coli tune # fastest safe profile `coli` reads `config.json` and picks the right engine for each model. the tradeoff is honest: no token bill, no hosted quota... but you need the storage. the frontier moved from your subscription to your SSD.

Avid

427,527 просмотров • 11 дней назад

introducing qwenfast the fastest inference for qwen3.8-27b, beating vanilla vllm, hosted on h200 i use the model's built-in mtp head for speculative decoding. large batches are bound by moving the 72 mb per sequence recurrent state, so i wrote my own triton deltanet decode kernels that run at over 80 percent of hbm bandwidth. everything runs inside cuda graphs. qwenfast: 133 t/s || vllm: 97 t/s, without compromising quality. gsm8k: 0.92 || ifeval: 0.98 vllm: 0.915 gsm8k, 0.96 ifeval try it yourself: also, no token limits, burn as many as you want..

introducing qwenfast the fastest inference for qwen3.8-27b, beating vanilla vllm, hosted on h200 i use the model's built-in mtp head for speculative decoding. large batches are bound by moving the 72 mb per sequence recurrent state, so i wrote my own triton deltanet decode kernels that run at over 80 percent of hbm bandwidth. everything runs inside cuda graphs. qwenfast: 133 t/s || vllm: 97 t/s, without compromising quality. gsm8k: 0.92 || ifeval: 0.98 vllm: 0.915 gsm8k, 0.96 ifeval try it yourself: also, no token limits, burn as many as you want..

harrrshall

21,023 просмотров • 28 дней назад

glm 5.3 vs qwen 3.8 vs gemini 3.7 vs deepseek v4 flash four models designed and built three structures each on a physics-backed site, with no dimensions anywhere in the brief the setup: our own agent loop on OpenRouter, a construction site as the tool set – footings, walls, arches, roofs, scaffold, a lamp. the site enforces physics and nothing else: unsupported brick falls, a roof needs walls under it, a worker reaches 3.2 m above whatever he stands on, an arch needs centring until the keystone is set, concrete cures before it carries. no budget ceiling – material cost is tallied and reported, never blocked. tasks: 1. house – a plot and a palette, no plan. shape, height and material are the model's call 2. lighthouse – a headland cut by a gully, with a rock stack standing 30 m offshore. the lamp must burn, it must be the highest thing built, and the keeper must be able to walk to it 3. bridge – a river with one islet and banks at different heights. cross it however you want models: Z.ai glm 5.3 flash, Qwen qwen 3.8 flash, Google DeepMind gemini 3.7 flash, DeepSeek v4 flash vision all twelve objects were finished and signed off by the models themselves. tallest lighthouse is qwen's at 38.4 m, planted on the offshore stack with a bridge run out to it – the only model that read the site that way. deepseek signed off its bridge on an empty riverbed: 0 bricks, 107 minutes, $1.16m of material tallied - total cost, three builds #1 glm 5.3 flash – $0.201 #2 gemini 3.7 flash – $0.871 #3 qwen 3.8 flash – $1.058 #4 deepseek v4 flash – $1.567 - wall clock, three builds #1 gemini 3.7 flash – 91m #2 glm 5.3 flash – 228m #3 deepseek v4 flash – 502m #4 qwen 3.8 flash – 912m - total tokens #1 gemini 3.7 flash – 3,567,052 #2 glm 5.3 flash – 4,732,748 #3 qwen 3.8 flash – 13,469,333 #4 deepseek v4 flash – 18,230,076 - defects logged by the site #1 deepseek v4 flash – 59 #2 gemini 3.7 flash – 132 #3 glm 5.3 flash – 221 #4 qwen 3.8 flash – 350 - material tallied across three builds #1 gemini 3.7 flash – $359,884 #2 glm 5.3 flash – $583,358 #3 deepseek v4 flash – $1,327,484 #4 qwen 3.8 flash – $2,188,625 observations: • glm is the cheap one and nothing here is close – $0.201 for three buildings, $0.042 per million tokens, 6x under gemini's rate • what glm spends it on is bulk, not care: 166,228 bricks in one house and 156 defect weight, the worst single object in the set • gemini is the efficiency line – 91 minutes and 3.57m tokens for all three and an eighth of qwen's clock • gemini also builds the smallest of everything. its lighthouse is 22.5 m against qwen's 38.4, its house 6.9 m against 19.3 • qwen is the maximalist: 1.18m bricks, $2.19m of material, tallest on all three tasks, and 912 minutes – 15 hours – to get there conclusion: twelve finished objects for $3.80 all in, and a 7.8x price spread between the cheapest model and the priciest! follow thehype. for 24/7 ai news, analysis and breakdowns

glm 5.3 vs qwen 3.8 vs gemini 3.7 vs deepseek v4 flash four models designed and built three structures each on a physics-backed site, with no dimensions anywhere in the brief the setup: our own agent loop on OpenRouter, a construction site as the tool set – footings, walls, arches, roofs, scaffold, a lamp. the site enforces physics and nothing else: unsupported brick falls, a roof needs walls under it, a worker reaches 3.2 m above whatever he stands on, an arch needs centring until the keystone is set, concrete cures before it carries. no budget ceiling – material cost is tallied and reported, never blocked. tasks: 1. house – a plot and a palette, no plan. shape, height and material are the model's call 2. lighthouse – a headland cut by a gully, with a rock stack standing 30 m offshore. the lamp must burn, it must be the highest thing built, and the keeper must be able to walk to it 3. bridge – a river with one islet and banks at different heights. cross it however you want models: Z.ai glm 5.3 flash, Qwen qwen 3.8 flash, Google DeepMind gemini 3.7 flash, DeepSeek v4 flash vision all twelve objects were finished and signed off by the models themselves. tallest lighthouse is qwen's at 38.4 m, planted on the offshore stack with a bridge run out to it – the only model that read the site that way. deepseek signed off its bridge on an empty riverbed: 0 bricks, 107 minutes, $1.16m of material tallied - total cost, three builds #1 glm 5.3 flash – $0.201 #2 gemini 3.7 flash – $0.871 #3 qwen 3.8 flash – $1.058 #4 deepseek v4 flash – $1.567 - wall clock, three builds #1 gemini 3.7 flash – 91m #2 glm 5.3 flash – 228m #3 deepseek v4 flash – 502m #4 qwen 3.8 flash – 912m - total tokens #1 gemini 3.7 flash – 3,567,052 #2 glm 5.3 flash – 4,732,748 #3 qwen 3.8 flash – 13,469,333 #4 deepseek v4 flash – 18,230,076 - defects logged by the site #1 deepseek v4 flash – 59 #2 gemini 3.7 flash – 132 #3 glm 5.3 flash – 221 #4 qwen 3.8 flash – 350 - material tallied across three builds #1 gemini 3.7 flash – $359,884 #2 glm 5.3 flash – $583,358 #3 deepseek v4 flash – $1,327,484 #4 qwen 3.8 flash – $2,188,625 observations: • glm is the cheap one and nothing here is close – $0.201 for three buildings, $0.042 per million tokens, 6x under gemini's rate • what glm spends it on is bulk, not care: 166,228 bricks in one house and 156 defect weight, the worst single object in the set • gemini is the efficiency line – 91 minutes and 3.57m tokens for all three and an eighth of qwen's clock • gemini also builds the smallest of everything. its lighthouse is 22.5 m against qwen's 38.4, its house 6.9 m against 19.3 • qwen is the maximalist: 1.18m bricks, $2.19m of material, tallest on all three tasks, and 912 minutes – 15 hours – to get there conclusion: twelve finished objects for $3.80 all in, and a 7.8x price spread between the cheapest model and the priciest! follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

26,360 просмотров • 28 дней назад

Qwen3.8-Flash-Next now fits on a 48gb macbook air Got q4 running on MLX using only 39gb of memory and decoding at 28 tok/s 🔥 Also q8 running on just 50gb of memory I stacked two tricks so far: 1. REAP pruning cut the expert pool from 68gb to 35gb 2. the 51b n-gram table now lives on the ssd instead of ram My patch memmaps the shards and reads ~100 bytes per lookup straight off disk, dequantized on cpu and verified bit-exact against the gpu path Qwen3.8-Flash-Next at 288 experts with its PLE table on nvme loads in 6 seconds, 39gb resident, 28 tok/s decode, 600 tok/s prefill. Logits identical to the ram-resident version in every way 97gb stock Q4 down to 68gb if pruned to 288 experts down to 39gb if pruned to 288 experts with the table on disk humaneval scoring 91.5% with 288 experts vs. 93.9% for stock I'm not done. next: 1. MTP drafter for speculative decode 2. n-gram lookup drafter both are built, benchmarking them now

Qwen3.8-Flash-Next now fits on a 48gb macbook air Got q4 running on MLX using only 39gb of memory and decoding at 28 tok/s 🔥 Also q8 running on just 50gb of memory I stacked two tricks so far: 1. REAP pruning cut the expert pool from 68gb to 35gb 2. the 51b n-gram table now lives on the ssd instead of ram My patch memmaps the shards and reads ~100 bytes per lookup straight off disk, dequantized on cpu and verified bit-exact against the gpu path Qwen3.8-Flash-Next at 288 experts with its PLE table on nvme loads in 6 seconds, 39gb resident, 28 tok/s decode, 600 tok/s prefill. Logits identical to the ram-resident version in every way 97gb stock Q4 down to 68gb if pruned to 288 experts down to 39gb if pruned to 288 experts with the table on disk humaneval scoring 91.5% with 288 experts vs. 93.9% for stock I'm not done. next: 1. MTP drafter for speculative decode 2. n-gram lookup drafter both are built, benchmarking them now

Eyal Toledano

10,468 просмотров • 1 месяц назад