Loading video...

Video Failed to Load

Go Home

Compared Qwen3.6 35B and 27B in the same conditions with Google TurboQuant Device: MacBook Pro M5Max 64GB RAM Outputs characteristics: Qwen3.6 35B: 6672 tokens, 2m 10s, 65 tok/s Qwen3.6 27B: 7344 tokens, 5m 22s, 24 tok/s Conclusion: Both models were asked to draw waves using HTML, 35B responded quickly...

55,540 views • 4 months ago •via X (Twitter)

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

If you have an RTX 3090 or 4090, Mia just shipped you a free massive upgrade in both speed and intelligence. I will explain to you why this will make your Qwen 3.8 27B on your card, even better, and my flags for running it. Qwen3.8-27B, EXL3 3.5bpw, DFlash2 speculative decode, RTX 4090. Single stream. The kit is from MiaAI-Lab, EXL3 is turboderp's format. I re-measured everything on my own card because the my first benchmarks seemed off. It turns out it really does run much faster. WHY EXL3 IS A DIFFERENT ANIMAL The old way (Q4_K_M) rounds each weight to the nearest 4-bit value independently. Every weight introduces its own rounding error. Those errors accumulate across millions of weights and causes drift (Slightly dumber). EXL3 is a fundamentally different compression algorithm. Instead of rounding each weight on its own, it encodes the entire weight vector as a path through a constrained codebook and spreads the rounding error across dimensions using a Hadamard transform. The result is that at the same bits per weight, more of the original model's intelligence is preserved. The important part is this CAN ACTUALLY BE MEASURED. The cleanest way to see that is KL divergence against a high-precision teacher. Lower means the quantized model thinks more like the original. On the malaiwah independent teacher-logit panel for GLM-5.3-Flash: EXL3 4bpw: 0.0246 nats Official FP8: 0.0206 nats NVFP4: 0.0605 nats EXL3 sits 0.004 nats behind native FP8 at half the size. NVFP4 at higher bit width is 2.5x further from the teacher. That panel is GLM-5.3-Flash, not Qwen 3.8. Cited as the mechanism, not as this run's data. But the point stands: EXL3 is not just smaller, it is smarter per bit than the formats most people are running. WHAT I MEASURED I first measure 108 tok/s from a single run. After that number looked too good to be true. I reran it. It looks like after a warm up, the numbers are even better. Basically, like people long thought, the RTX 3090 and RTX 4090 are actually superb AI computer cards. Hence why NVIDIA stopped shipping them with NVLINK since the 4090. Short context ceiling (~2k in, 1016-token output, TTFT-separated): 135, 138, 153, 174, 133, 129 tok/s across 6 runs. Sustained longform (2040-token essay): 105.4, 94.5, 98.4 tok/s Short answer (504 tokens): 93.2 tok/s The honest shape: ~130-150 tok/s at short context is the ceiling, ~94-105 sustained on longform. The ceiling matters because that is what people feel in chat. The old dense Q4_K_M on llama.cpp ran ~37 tok/s on this same card. (No MTP), with MTP about 60 tok/s Sustained is roughly 2.5-3x. Ceiling is closer to 4x. Same model, different quantization and engine. The multiplier comes from EXL3, the ExLlamaV2 engine, and DFlash2 together. CONCURRENCY IS A RTX 4090 LANE. Just like the old config on the 4090, the 24gb vram, means a long context can only hold one stream, and running concurrency requires to lower context length, because it runs fast it sort of makes up for it by being faster than slower GPU chips. CONTEXT LADDER The recipe doc measured prefill. I re-ran it with TTFT separated from decode, because decode is what you actually feel after the first token. ~5k in: decode 140 tok/s (TTFT 0.5s), needle HIT ~18k in: decode 85 tok/s (TTFT 0.3s*), needle HIT ~73k in: decode 28 tok/s (TTFT 1.8s), needle HIT ~146k in: decode 16 tok/s (TTFT 2.3s), needle HIT (*0.3s at 18k is a prefix-cache hit from the paired pass. Cold prefill for reference: ~2,020 tok/s at 17k falling to ~508 at 153k.) Needle hit at every depth, mine and the original 7/7. Retrieval is intact at max context. Speed is not: decode falls ~9x from short to max. Past ~50k tokens this stops being a chat tool and becomes a batch tool. At 146k it works, but nobody is typing interactively against 16 tok/s. WHERE IT BROKE The model's native context is 262k. The README says DFlash2 fits ~220k on a 24GB card. My 200,704-token attempt failed with insufficient VRAM. Dropped to 168,960 and it booted. The real ceiling is somewhere between 168,960 and 200,704 and I never tested that gap. I jumped to a value that worked and called it done. That is ~32k tokens of context I left on the table. One thing the numbers taught me: JSON tokenizes at ~1.5 chars/token, prose at ~3.9. "150k tokens of JSON" needs ~2.7x more filler than the same estimate in prose. Size by real tokens, not estimates. THE UPGRADE If you own a 4090 and you are running Q4_K_M on llama.cpp, you are leaving a good bit of speed and measurable intelligence on the table. The same model, on the same card, with a better quantization and engine, goes from 60 tok/s to 130-150 at short context and 94-105 sustained. The model also thinks closer to the original because EXL3 preserves more of the output distribution per bit than the old rounding method. The recipe is in the first reply. Everything above came from one 4090 and one afternoon of re-measuring. The decode ladder especially needs independent numbers. If your card gets different falloff, that is worth knowing. Recipe and flags/ findings in reply 👇

Yume_X

37,992 views • 5 days ago

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 views • 7 months ago

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?

Alok

39,189 views • 18 days ago

HTML Artifacts are a big part of how I work with agents now. Artifacts can be more than just static files. When combined with agents, they can take action or help you take action. This unlocks all kinds of interesting ways to work with agents. This is clearly the future. Check out this writing and scheduler artifact I built in a few minutes. It uses a bit of HTML and JS. All the data is in markdown (Obsidian vaults), so the agent can access and modify it at any time. No DB needed. No sophisticated functionalities. The agent decides all that for me based on the skills, context, and memory it has access to. The best part about this simple stack is that all the important information stays with me. This has allowed me to build a recursive self-improving system and automations that can better tap into coding agents like Codex or Claude Code. I could have paid or built an entire app for scheduling posts, and there are so many of them out there. But I don't need to. I've realized a simple artifact does the job. And the simplicity of it is actually an advantage. Very little maintenance for very high returns on personalization, time, and efficiency. The other benefit of this is that I can add features as I please. That level of personalization feels magical, and we should all be pursuing more of it. All of this just keeps compounding. Of course, this example is just about writing. But I have similar artifacts for research, design, experimentation, evaluation, and so much more. And no, I didn't actually publish the post example I shared in the clip. It was just for demonstration purposes. I actually spend more time than this when writing together with agents. Lastly, having built my own agent orchestrator tool has made me realize that simplifying the tool stack is a superpower. If you are curious about how all this works, I will do a live session next week:

elvis

18,374 views • 3 months ago

AI token usage is up 10x in 7 months, compounding 40%/MONTH! There is NO BUBBLE when demand is STILL accelerating And this is just OpenRouter, it doesn't count the labs direct token usage and APIs But here's what's interesting about these numbers, the demand is coming from everywhere at once US models (OpenAI, Anthropic, Google) keep growing, while Chinese open weight models (DeepSeek, Tencent, Xiaomi, Minimax) grew even faster and now drive over 60% of usage on OpenRouter Closed source and open source both compounding at the same time. This is literally the best case scenario for AI Infra investors It means both frontier model tokens and cheaper tokens have product market fit. This means the application layer is finding ways to use both and generate ROI with both types Demand for tokens IS demand for compute. This is why SpaceX is looking to build 10GW of compute by next year, because the demand is clearly here Now combine this demand set up, with NVIDIA yesterday announcing financing platforms with Apollo, BlackRock, Blackstone, Brookfield, Goldman Sachs and KKR to mobilize over $500 billion of third party capital for AI infrastructure And Jensen has said publicly he expects $3 to $4 TRILLION of AI infrastructure spend by 2030 The build out will have to continue for a lot longer than the market is expecting, that is very clear to me. Don't let this consolidation period in AI infra stocks shake you out, they will have their moment again and take their next leg higher p.s. if you want to see how im investing in this, you can track my real-time portfolio and the research of all 5 Milk Road PRO analysts with live trade notifications, and it's just $1 to try it out (insane price just to check it out). Learn more here: Good luck out there!

Kyle Reidhead | Milk Road

28,320 views • 28 days ago

🚀Just launched: Amazon Q, the most capable GenAI-powered assistant is generally available today: Customers are using Q to transform how their teams get work done. When employees chat with Amazon Q, it provides immediate, relevant information and advice to help streamline tasks, speedup decision-making, and help spark creativity and innovation at work. . Early indications signal Amazon Q could help our customers’ employees become more than 80% more productive at their jobs; and with the new features we’re planning on introducing in the future, we think this will only continue to grow. 🟠 Amazon Q Developer allows developers to spend more time coding and less time on maintenance and performing other tedious, repetitive tasks. Q assists developers and IT professionals (IT pros) with all of their tasks—from coding, testing, and upgrading applications, to troubleshooting, performing security scanning and fixes, and optimizing AWS resources. Q also comes with Q Developer Agents which can autonomously perform range of tasks and we expect it to be the state of the art accuracy in benchmarks like SWE-Bench. 🟠 Amazon Q Business empowers employees to be more data-driven, and helps customers make better, faster decisions using company knowledge and data. Q Business is a generative AI–powered assistant that can answer questions, provide summaries, generate content, and securely complete tasks based on data and information in enterprise systems 🟠 Amazon Q Apps, a new and powerful capability of Amazon Q Business, enables employees to use natural language to quickly and securely build their own generative AI applications to automate daily tasks without requiring any prior coding experience. Employees simply describe the type of app they want, in natural language, and Q Apps will quickly generate an app that accomplishes their desired task, helping them streamline and automate their daily work with ease and efficiency.

Swami Sivasubramanian

25,216 views • 2 years ago