Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running... show more

leopardracer

12,100 subscribers

241,145 views • 5 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

43 Comments

NotYourDC5 months ago

Who is speaking in this video? Is that you? It's not an American he is clearly English. Bit confused by the video and the link to 2-GPU build below it. How are they related?

leopardracer5 months ago

this developer figured out the same thing I wrote about: strip the wrappers, run llama.cpp raw. different hardware, same conclusion

starmex5 months ago

cut the wrappers, run the source. solid setup

leopardracer5 months ago

exactly the mindset

elune5 months ago

zero api fees forever wins

leopardracer5 months ago

@asakura0x correct conclusion👍🏻

elune5 months ago

😼

Samuel Gregory5 months ago

Thank you but I'm not American

leopardracer5 months ago

sorry 🙏🏽

𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸5 months ago

This llama launcher gives us full control how llama-server is executed

kkiran5 months ago

Please link others videos to the source with your commentary. Thank you! @0x5am5 thanks for the contribution!

Ghost Signal5 months ago

Most developers debug the wrapper. Never the engine. LM Studio. Ollama. Different names. Same llama.cpp underneath. The timeout wasn’t a bug. It was a layer you didn’t need. Going direct to llama.cpp isn’t advanced. It’s just removing the abstraction that was hiding the problem. 53 tok/s. 262K context. Zero fees. That’s what’s under the hood.

Kekko D’Amato5 months ago

the asymmetric KV quantization bit is what really matters here. most people don't realize that KV cache is the memory bottleneck on consumer GPUs, not weights. Q8 weights + Q4 KV cache is a genuinely different tradeoff than Ollama's defaults, and going direct llama.cpp lets you tune that knob. the wrappers abstract away exactly the setting you'd want to change

kfant5 months ago

who is this guy ? credit/reference

Bill The Radio Guy5 months ago

Or just configure lm studio to run the tool correctly, it has tons of configuration options and surely more could be added if that were necessary.

Joseph Hurtado - Founder Granata Consulting5 months ago

Thanks for the article. Can you share the link to the source video? Like you, that fellow deserves a follow.

slash1s5 months ago

yep definitely need to save this..

leopardracer5 months ago

230 people already did

I Pun Daddy5 months ago

Classic move: abstract the abstraction, then debug the original abstraction.

CoreAI5 months ago

No api fees👏

Bober_smart5 months ago

I'm starting to watch it, it sounds very tempting

leopardracer5 months ago

once you start you don’t go back

Mario 🐧5 months ago

I have a homogeneous GPU set up. 5060ti and 3060 ~ 28.5 gb's. Can meaningfully run qwen 27/35b My Thoughts: - heterogeneous is better(same gpu arch) - You'll want higher mem-bandwidth - Can run a model for workflows locally, but still want to build with a SOTA model.

b5 months ago

😁 so it took this idiot (and also you) 3 years to understand ollama was just llama.cpp under the hood. Great job keep it up 🤦‍♂️

Gipp 🦅5 months ago

In fact, with AI it's very easy

leopardracer5 months ago

easy once you know the stack. that’s the whole point of the article

Insomnia5 months ago

he know facts how it's all works

leopardracer5 months ago

no theory. just what actually runs

xpost5 months ago

@grok is this real setup, Worth duplicating?

Victoria Neiman5 months ago

struck the OpenAI bill with 2 GPUs. not showing off, this is a business model. platform prices can change anytime. your hardware can't. more context:

Blak5 months ago

but how do you solve the hardcoded 2GB metabuffer ?the server will crash when >2GB! so llama.cpp you will not be able to run as a 24/7 server with 120k context ! im testing now for weeks for stable 24/7 server inference for lokal production! please share with us the code!

Daily Story5 months ago

Building locally seems like a game changer for overcoming those timeout errors!

Jeff Davis, MD5 months ago

Credit need to go to @0x5am5 and the original video:

Kyle5 months ago

Qwen 3.6 35B from Huggingface is 72gb... So this story is bullshit right off the bat.

Thirty5 months ago

@grok solid or no? Risk involved?

84動物農場🇺🇦5 months ago

I have an almost identical experimental machine with dual 4080S graphics cards. I always cram it with the largest model that can run, but it seems this isn't the best approach. Thanks for the experiment.

Horoshi⚡5 months ago

🤣This happens too, I thought for a long time...🤷

TruthSeeker5 months ago

Check out TurboAgent. It does the weights and KV compression.

whiletrue1115 months ago

And what is all of this for? What is he actually doing with this? Summarizing articles every morning?

James Rogers - Cinematographer; Truth@24fps5 months ago

Today in “duh”

AX⚡5 months ago

lmao yeah no shit it's all llama.cpp, what else would it be? hahaha

Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖5 months ago

@grok summarise the video.

StarShipX5 months ago

すごい技術ですね、私も試してみたいです。

Related Videos

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

leopardracer

29,756 views • 5 months ago

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Xuan-Son Nguyen

974,800 views • 1 year ago

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

Alvaro Cintas

196,725 views • 2 months ago

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

Sudo su

167,565 views • 8 months ago

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Julian Goldie SEO

10,098 views • 6 months ago

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

Xenova

21,602 views • 2 months ago

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Wësche

15,149 views • 2 months ago

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

Lewis Tunstall

118,819 views • 5 months ago

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Mervin Praison

57,732 views • 8 months ago

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 views • 1 month ago

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

Julian Goldie SEO

29,883 views • 7 months ago

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

OscarMartin

302,902 views • 5 months ago

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

Julian Goldie SEO

77,202 views • 6 months ago

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

Julian Goldie SEO

31,717 views • 7 months ago

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 views • 4 months ago

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Georgi Gerganov

786,184 views • 6 months ago

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

Alejandro AO 🤗

31,529 views • 10 days ago

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

leonie

64,192 views • 2 months ago

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

Victor M

171,948 views • 5 months ago