Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running... show more

leopardracer

12,100 subscribers

241,145 Aufrufe • vor 5 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

43 Kommentare

Profilbild von NotYourDC

NotYourDCvor 5 Monaten

Who is speaking in this video? Is that you? It's not an American he is clearly English. Bit confused by the video and the link to 2-GPU build below it. How are they related?

Profilbild von leopardracer

leopardracervor 5 Monaten

this developer figured out the same thing I wrote about: strip the wrappers, run llama.cpp raw. different hardware, same conclusion

Profilbild von starmex

starmexvor 5 Monaten

cut the wrappers, run the source. solid setup

Profilbild von leopardracer

leopardracervor 5 Monaten

exactly the mindset

Profilbild von elune

elunevor 5 Monaten

zero api fees forever wins

Profilbild von leopardracer

leopardracervor 5 Monaten

@asakura0x correct conclusion👍🏻

Profilbild von elune

elunevor 5 Monaten

😼

Profilbild von Samuel Gregory

Samuel Gregoryvor 5 Monaten

Thank you but I'm not American

Profilbild von leopardracer

leopardracervor 5 Monaten

sorry 🙏🏽

Profilbild von 𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸

𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸vor 5 Monaten

This llama launcher gives us full control how llama-server is executed

Profilbild von kkiran

kkiranvor 5 Monaten

Please link others videos to the source with your commentary. Thank you! @0x5am5 thanks for the contribution!

Profilbild von Ghost Signal

Ghost Signalvor 5 Monaten

Most developers debug the wrapper. Never the engine. LM Studio. Ollama. Different names. Same llama.cpp underneath. The timeout wasn’t a bug. It was a layer you didn’t need. Going direct to llama.cpp isn’t advanced. It’s just removing the abstraction that was hiding the problem. 53 tok/s. 262K context. Zero fees. That’s what’s under the hood.

Profilbild von Kekko D’Amato

Kekko D’Amatovor 5 Monaten

the asymmetric KV quantization bit is what really matters here. most people don't realize that KV cache is the memory bottleneck on consumer GPUs, not weights. Q8 weights + Q4 KV cache is a genuinely different tradeoff than Ollama's defaults, and going direct llama.cpp lets you tune that knob. the wrappers abstract away exactly the setting you'd want to change

Profilbild von kfant

kfantvor 5 Monaten

who is this guy ? credit/reference

Profilbild von Bill The Radio Guy

Bill The Radio Guyvor 5 Monaten

Or just configure lm studio to run the tool correctly, it has tons of configuration options and surely more could be added if that were necessary.

Profilbild von Joseph Hurtado - Founder Granata Consulting

Joseph Hurtado - Founder Granata Consultingvor 5 Monaten

Thanks for the article. Can you share the link to the source video? Like you, that fellow deserves a follow.

Profilbild von slash1s

slash1svor 5 Monaten

yep definitely need to save this..

Profilbild von leopardracer

leopardracervor 5 Monaten

230 people already did

Profilbild von I Pun Daddy

I Pun Daddyvor 5 Monaten

Classic move: abstract the abstraction, then debug the original abstraction.

Profilbild von CoreAI

CoreAIvor 5 Monaten

No api fees👏

Profilbild von Bober_smart

Bober_smartvor 5 Monaten

I'm starting to watch it, it sounds very tempting

Profilbild von leopardracer

leopardracervor 5 Monaten

once you start you don’t go back

Profilbild von Mario 🐧

Mario 🐧vor 5 Monaten

I have a homogeneous GPU set up. 5060ti and 3060 ~ 28.5 gb's. Can meaningfully run qwen 27/35b My Thoughts: - heterogeneous is better(same gpu arch) - You'll want higher mem-bandwidth - Can run a model for workflows locally, but still want to build with a SOTA model.

Profilbild von b

bvor 5 Monaten

😁 so it took this idiot (and also you) 3 years to understand ollama was just llama.cpp under the hood. Great job keep it up 🤦‍♂️

Profilbild von Gipp 🦅

Gipp 🦅vor 5 Monaten

In fact, with AI it's very easy

Profilbild von leopardracer

leopardracervor 5 Monaten

easy once you know the stack. that’s the whole point of the article

Profilbild von Insomnia

Insomniavor 5 Monaten

he know facts how it's all works

Profilbild von leopardracer

leopardracervor 5 Monaten

no theory. just what actually runs

Profilbild von xpost

xpostvor 5 Monaten

@grok is this real setup, Worth duplicating?

Profilbild von Victoria Neiman

Victoria Neimanvor 5 Monaten

struck the OpenAI bill with 2 GPUs. not showing off, this is a business model. platform prices can change anytime. your hardware can't. more context:

Profilbild von Blak

Blakvor 5 Monaten

but how do you solve the hardcoded 2GB metabuffer ?the server will crash when >2GB! so llama.cpp you will not be able to run as a 24/7 server with 120k context ! im testing now for weeks for stable 24/7 server inference for lokal production! please share with us the code!

Profilbild von Daily Story

Daily Storyvor 5 Monaten

Building locally seems like a game changer for overcoming those timeout errors!

Profilbild von Jeff Davis, MD

Jeff Davis, MDvor 5 Monaten

Credit need to go to @0x5am5 and the original video:

Profilbild von Kyle

Kylevor 5 Monaten

Qwen 3.6 35B from Huggingface is 72gb... So this story is bullshit right off the bat.

Profilbild von Thirty

Thirtyvor 5 Monaten

@grok solid or no? Risk involved?

Profilbild von 84動物農場🇺🇦

84動物農場🇺🇦vor 5 Monaten

I have an almost identical experimental machine with dual 4080S graphics cards. I always cram it with the largest model that can run, but it seems this isn't the best approach. Thanks for the experiment.

Profilbild von Horoshi⚡

Horoshi⚡vor 5 Monaten

🤣This happens too, I thought for a long time...🤷

Profilbild von TruthSeeker

TruthSeekervor 5 Monaten

Check out TurboAgent. It does the weights and KV compression.

Profilbild von whiletrue111

whiletrue111vor 5 Monaten

And what is all of this for? What is he actually doing with this? Summarizing articles every morning?

Profilbild von James Rogers - Cinematographer; Truth@24fps

James Rogers - Cinematographer; Truth@24fpsvor 5 Monaten

Today in “duh”

Profilbild von AX⚡

AX⚡vor 5 Monaten

lmao yeah no shit it's all llama.cpp, what else would it be? hahaha

Profilbild von Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖

Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖vor 5 Monaten

@grok summarise the video.

Profilbild von StarShipX

StarShipXvor 5 Monaten

すごい技術ですね、私も試してみたいです。

Ähnliche Videos

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

leopardracer

29,756 Aufrufe • vor 5 Monaten

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

Alvaro Cintas

196,725 Aufrufe • vor 2 Monaten

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Xuan-Son Nguyen

974,800 Aufrufe • vor 1 Jahr

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

Sudo su

167,565 Aufrufe • vor 8 Monaten

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Julian Goldie SEO

10,098 Aufrufe • vor 6 Monaten

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

Xenova

21,602 Aufrufe • vor 2 Monaten

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Wësche

15,149 Aufrufe • vor 2 Monaten

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

Lewis Tunstall

118,819 Aufrufe • vor 5 Monaten

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Mervin Praison

57,732 Aufrufe • vor 8 Monaten

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 Aufrufe • vor 1 Monat

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

Julian Goldie SEO

29,883 Aufrufe • vor 7 Monaten

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

OscarMartin

302,902 Aufrufe • vor 5 Monaten

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

Julian Goldie SEO

77,202 Aufrufe • vor 6 Monaten

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

Julian Goldie SEO

31,717 Aufrufe • vor 7 Monaten

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 Aufrufe • vor 4 Monaten

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Georgi Gerganov

786,184 Aufrufe • vor 7 Monaten

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

Alejandro AO 🤗

31,529 Aufrufe • vor 10 Tagen

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

leonie

64,192 Aufrufe • vor 2 Monaten

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

Victor M

171,948 Aufrufe • vor 5 Monaten