正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running... show more

leopardracer

12,100 subscribers

241,145 次观看 • 5 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

43 条评论

NotYourDC 的头像

NotYourDC5 个月前

Who is speaking in this video? Is that you? It's not an American he is clearly English. Bit confused by the video and the link to 2-GPU build below it. How are they related?

leopardracer 的头像

leopardracer5 个月前

this developer figured out the same thing I wrote about: strip the wrappers, run llama.cpp raw. different hardware, same conclusion

starmex 的头像

starmex5 个月前

cut the wrappers, run the source. solid setup

leopardracer 的头像

leopardracer5 个月前

exactly the mindset

elune 的头像

elune5 个月前

zero api fees forever wins

leopardracer 的头像

leopardracer5 个月前

@asakura0x correct conclusion👍🏻

elune 的头像

elune5 个月前

😼

Samuel Gregory 的头像

Samuel Gregory5 个月前

Thank you but I'm not American

leopardracer 的头像

leopardracer5 个月前

sorry 🙏🏽

𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸 的头像

𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸5 个月前

This llama launcher gives us full control how llama-server is executed

kkiran 的头像

kkiran5 个月前

Please link others videos to the source with your commentary. Thank you! @0x5am5 thanks for the contribution!

Ghost Signal 的头像

Ghost Signal5 个月前

Most developers debug the wrapper. Never the engine. LM Studio. Ollama. Different names. Same llama.cpp underneath. The timeout wasn’t a bug. It was a layer you didn’t need. Going direct to llama.cpp isn’t advanced. It’s just removing the abstraction that was hiding the problem. 53 tok/s. 262K context. Zero fees. That’s what’s under the hood.

Kekko D’Amato 的头像

Kekko D’Amato5 个月前

the asymmetric KV quantization bit is what really matters here. most people don't realize that KV cache is the memory bottleneck on consumer GPUs, not weights. Q8 weights + Q4 KV cache is a genuinely different tradeoff than Ollama's defaults, and going direct llama.cpp lets you tune that knob. the wrappers abstract away exactly the setting you'd want to change

kfant 的头像

kfant5 个月前

who is this guy ? credit/reference

Bill The Radio Guy 的头像

Bill The Radio Guy5 个月前

Or just configure lm studio to run the tool correctly, it has tons of configuration options and surely more could be added if that were necessary.

Joseph Hurtado - Founder Granata Consulting 的头像

Joseph Hurtado - Founder Granata Consulting5 个月前

Thanks for the article. Can you share the link to the source video? Like you, that fellow deserves a follow.

slash1s 的头像

slash1s5 个月前

yep definitely need to save this..

leopardracer 的头像

leopardracer5 个月前

230 people already did

I Pun Daddy 的头像

I Pun Daddy5 个月前

Classic move: abstract the abstraction, then debug the original abstraction.

CoreAI 的头像

CoreAI5 个月前

No api fees👏

Bober_smart 的头像

Bober_smart5 个月前

I'm starting to watch it, it sounds very tempting

leopardracer 的头像

leopardracer5 个月前

once you start you don’t go back

Mario 🐧 的头像

Mario 🐧5 个月前

I have a homogeneous GPU set up. 5060ti and 3060 ~ 28.5 gb's. Can meaningfully run qwen 27/35b My Thoughts: - heterogeneous is better(same gpu arch) - You'll want higher mem-bandwidth - Can run a model for workflows locally, but still want to build with a SOTA model.

b 的头像

b5 个月前

😁 so it took this idiot (and also you) 3 years to understand ollama was just llama.cpp under the hood. Great job keep it up 🤦‍♂️

Gipp 🦅 的头像

Gipp 🦅5 个月前

In fact, with AI it's very easy

leopardracer 的头像

leopardracer5 个月前

easy once you know the stack. that’s the whole point of the article

Insomnia 的头像

Insomnia5 个月前

he know facts how it's all works

leopardracer 的头像

leopardracer5 个月前

no theory. just what actually runs

xpost 的头像

xpost5 个月前

@grok is this real setup, Worth duplicating?

Victoria Neiman 的头像

Victoria Neiman5 个月前

struck the OpenAI bill with 2 GPUs. not showing off, this is a business model. platform prices can change anytime. your hardware can't. more context:

Blak 的头像

Blak5 个月前

but how do you solve the hardcoded 2GB metabuffer ?the server will crash when >2GB! so llama.cpp you will not be able to run as a 24/7 server with 120k context ! im testing now for weeks for stable 24/7 server inference for lokal production! please share with us the code!

Daily Story 的头像

Daily Story5 个月前

Building locally seems like a game changer for overcoming those timeout errors!

Jeff Davis, MD 的头像

Jeff Davis, MD5 个月前

Credit need to go to @0x5am5 and the original video:

Kyle 的头像

Kyle5 个月前

Qwen 3.6 35B from Huggingface is 72gb... So this story is bullshit right off the bat.

Thirty 的头像

Thirty5 个月前

@grok solid or no? Risk involved?

84動物農場🇺🇦 的头像

84動物農場🇺🇦5 个月前

I have an almost identical experimental machine with dual 4080S graphics cards. I always cram it with the largest model that can run, but it seems this isn't the best approach. Thanks for the experiment.

Horoshi⚡ 的头像

Horoshi⚡5 个月前

🤣This happens too, I thought for a long time...🤷

TruthSeeker 的头像

TruthSeeker5 个月前

Check out TurboAgent. It does the weights and KV compression.

whiletrue111 的头像

whiletrue1115 个月前

And what is all of this for? What is he actually doing with this? Summarizing articles every morning?

James Rogers - Cinematographer; Truth@24fps 的头像

James Rogers - Cinematographer; Truth@24fps5 个月前

Today in “duh”

AX⚡ 的头像

AX⚡5 个月前

lmao yeah no shit it's all llama.cpp, what else would it be? hahaha

Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖 的头像

Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖5 个月前

@grok summarise the video.

StarShipX 的头像

StarShipX5 个月前

すごい技術ですね、私も試してみたいです。

相关视频

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

leopardracer

29,756 次观看 • 5 个月前

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Xuan-Son Nguyen

974,800 次观看 • 1 年前

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

Alvaro Cintas

196,725 次观看 • 2 个月前

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

Sudo su

167,565 次观看 • 8 个月前

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Julian Goldie SEO

10,098 次观看 • 6 个月前

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

Xenova

21,602 次观看 • 2 个月前

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Wësche

15,149 次观看 • 2 个月前

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

Lewis Tunstall

118,819 次观看 • 5 个月前

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Mervin Praison

57,732 次观看 • 8 个月前

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 次观看 • 1 个月前

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

Julian Goldie SEO

29,883 次观看 • 7 个月前

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

OscarMartin

302,902 次观看 • 5 个月前

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

Julian Goldie SEO

77,202 次观看 • 6 个月前

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

Julian Goldie SEO

31,717 次观看 • 7 个月前

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 次观看 • 4 个月前

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Georgi Gerganov

786,184 次观看 • 7 个月前

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

Alejandro AO 🤗

31,529 次观看 • 10 天前

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

leonie

64,192 次观看 • 2 个月前

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

Victor M

171,948 次观看 • 5 个月前