Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running... show more

leopardracer

12,100 subscribers

241,145 görüntüleme • 5 ay önce •via X (Twitter)

Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

43 Yorum

NotYourDC profil fotoğrafı

NotYourDC5 ay önce

Who is speaking in this video? Is that you? It's not an American he is clearly English. Bit confused by the video and the link to 2-GPU build below it. How are they related?

leopardracer profil fotoğrafı

leopardracer5 ay önce

this developer figured out the same thing I wrote about: strip the wrappers, run llama.cpp raw. different hardware, same conclusion

starmex profil fotoğrafı

starmex5 ay önce

cut the wrappers, run the source. solid setup

leopardracer profil fotoğrafı

leopardracer5 ay önce

exactly the mindset

elune profil fotoğrafı

elune5 ay önce

zero api fees forever wins

leopardracer profil fotoğrafı

leopardracer5 ay önce

@asakura0x correct conclusion👍🏻

elune profil fotoğrafı

elune5 ay önce

😼

Samuel Gregory profil fotoğrafı

Samuel Gregory5 ay önce

Thank you but I'm not American

leopardracer profil fotoğrafı

leopardracer5 ay önce

sorry 🙏🏽

𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸 profil fotoğrafı

𝔸𝕟𝕕𝕚 𝕁𝕒𝕖𝕘𝕖𝕣📯📸5 ay önce

This llama launcher gives us full control how llama-server is executed

kkiran profil fotoğrafı

kkiran5 ay önce

Please link others videos to the source with your commentary. Thank you! @0x5am5 thanks for the contribution!

Ghost Signal profil fotoğrafı

Ghost Signal5 ay önce

Most developers debug the wrapper. Never the engine. LM Studio. Ollama. Different names. Same llama.cpp underneath. The timeout wasn’t a bug. It was a layer you didn’t need. Going direct to llama.cpp isn’t advanced. It’s just removing the abstraction that was hiding the problem. 53 tok/s. 262K context. Zero fees. That’s what’s under the hood.

Kekko D’Amato profil fotoğrafı

Kekko D’Amato5 ay önce

the asymmetric KV quantization bit is what really matters here. most people don't realize that KV cache is the memory bottleneck on consumer GPUs, not weights. Q8 weights + Q4 KV cache is a genuinely different tradeoff than Ollama's defaults, and going direct llama.cpp lets you tune that knob. the wrappers abstract away exactly the setting you'd want to change

kfant profil fotoğrafı

kfant5 ay önce

who is this guy ? credit/reference

Bill The Radio Guy profil fotoğrafı

Bill The Radio Guy5 ay önce

Or just configure lm studio to run the tool correctly, it has tons of configuration options and surely more could be added if that were necessary.

Joseph Hurtado - Founder Granata Consulting profil fotoğrafı

Joseph Hurtado - Founder Granata Consulting5 ay önce

Thanks for the article. Can you share the link to the source video? Like you, that fellow deserves a follow.

slash1s profil fotoğrafı

slash1s5 ay önce

yep definitely need to save this..

leopardracer profil fotoğrafı

leopardracer5 ay önce

230 people already did

I Pun Daddy profil fotoğrafı

I Pun Daddy5 ay önce

Classic move: abstract the abstraction, then debug the original abstraction.

CoreAI profil fotoğrafı

CoreAI5 ay önce

No api fees👏

Bober_smart profil fotoğrafı

Bober_smart5 ay önce

I'm starting to watch it, it sounds very tempting

leopardracer profil fotoğrafı

leopardracer5 ay önce

once you start you don’t go back

Mario 🐧 profil fotoğrafı

Mario 🐧5 ay önce

I have a homogeneous GPU set up. 5060ti and 3060 ~ 28.5 gb's. Can meaningfully run qwen 27/35b My Thoughts: - heterogeneous is better(same gpu arch) - You'll want higher mem-bandwidth - Can run a model for workflows locally, but still want to build with a SOTA model.

b profil fotoğrafı

b5 ay önce

😁 so it took this idiot (and also you) 3 years to understand ollama was just llama.cpp under the hood. Great job keep it up 🤦‍♂️

Gipp 🦅 profil fotoğrafı

Gipp 🦅5 ay önce

In fact, with AI it's very easy

leopardracer profil fotoğrafı

leopardracer5 ay önce

easy once you know the stack. that’s the whole point of the article

Insomnia profil fotoğrafı

Insomnia5 ay önce

he know facts how it's all works

leopardracer profil fotoğrafı

leopardracer5 ay önce

no theory. just what actually runs

xpost profil fotoğrafı

xpost5 ay önce

@grok is this real setup, Worth duplicating?

Victoria Neiman profil fotoğrafı

Victoria Neiman5 ay önce

struck the OpenAI bill with 2 GPUs. not showing off, this is a business model. platform prices can change anytime. your hardware can't. more context:

Blak profil fotoğrafı

Blak5 ay önce

but how do you solve the hardcoded 2GB metabuffer ?the server will crash when >2GB! so llama.cpp you will not be able to run as a 24/7 server with 120k context ! im testing now for weeks for stable 24/7 server inference for lokal production! please share with us the code!

Daily Story profil fotoğrafı

Daily Story5 ay önce

Building locally seems like a game changer for overcoming those timeout errors!

Jeff Davis, MD profil fotoğrafı

Jeff Davis, MD5 ay önce

Credit need to go to @0x5am5 and the original video:

Kyle profil fotoğrafı

Kyle5 ay önce

Qwen 3.6 35B from Huggingface is 72gb... So this story is bullshit right off the bat.

Thirty profil fotoğrafı

Thirty5 ay önce

@grok solid or no? Risk involved?

84動物農場🇺🇦 profil fotoğrafı

84動物農場🇺🇦5 ay önce

I have an almost identical experimental machine with dual 4080S graphics cards. I always cram it with the largest model that can run, but it seems this isn't the best approach. Thanks for the experiment.

Horoshi⚡ profil fotoğrafı

Horoshi⚡5 ay önce

🤣This happens too, I thought for a long time...🤷

TruthSeeker profil fotoğrafı

TruthSeeker5 ay önce

Check out TurboAgent. It does the weights and KV compression.

whiletrue111 profil fotoğrafı

whiletrue1115 ay önce

And what is all of this for? What is he actually doing with this? Summarizing articles every morning?

James Rogers - Cinematographer; Truth@24fps profil fotoğrafı

James Rogers - Cinematographer; Truth@24fps5 ay önce

Today in “duh”

AX⚡ profil fotoğrafı

AX⚡5 ay önce

lmao yeah no shit it's all llama.cpp, what else would it be? hahaha

Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖 profil fotoğrafı

Tony Scott 🧄(🦆🐓🐵🧪🧬🪪)❌=↑🧄🧄🧄🥩🥚🧀↓👽👾🤖5 ay önce

@grok summarise the video.

StarShipX profil fotoğrafı

StarShipX5 ay önce

すごい技術ですね、私も試してみたいです。

Benzer Videolar

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

THIS ENGINEER RUNS CLAUDE CODE WITHOUT PAYING ANTHROPIC A SINGLE DOLLAR RTX 5090 under the desk, Qwen 3.5 35B, 140 tokens per second Claude Code pointed at localhost instead of Anthropic’s servers override two environment variables and your API bill drops to zero he built a full-stack Next.js app from scratch to prove it works then showed every bug and limitation other YouTubers hide same setup I run with two GPUs and Qwen 3.6 27B full breakdown ↓

leopardracer

29,756 görüntüleme • 5 ay önce

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

Alvaro Cintas

196,725 görüntüleme • 2 ay önce

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Real-time webcam demo with Hugging Face SmolVLM and ggml llama.cpp server. All running locally on a Macbook M3

Xuan-Son Nguyen

974,800 görüntüleme • 1 yıl önce

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in latest update) > speed: 112 tok/s generation, ~130 tok/s prefill > context: 262K tokens > flags: -ngl 99 -c 262144 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 > engine: llama.cpp > agent: Claude Code talk to localhost:8080 (llama.cpp now has native Anthropic API endpoint. no LiteLLM needed) q8_0 KV cache cuts VRAM usage in half vs f16 at 262K. -np 1 is default but worth noting. parallel slots multiply KV cache and at 262K that's an instant OOM. the prompt was more detailed than this but you get the idea: build a space shooter with parallax backgrounds, particle systems, procedural audio, 4 enemy types, boss fights, power-up system, and ship upgrades. 8 JavaScript modules. no libraries. game's called Octopus Invaders. gameplay footage dropping next.

Sudo su

167,565 görüntüleme • 8 ay önce

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Claude Code is now completely free forever 🤯 Just connect it to Ollama and run a local model like Google’s new Gemma 4. That means Claude Code agents can run directly on your computer with zero API bills and zero subscriptions. Google may have just ended AI subscription fees for good. 🚀

Julian Goldie SEO

10,098 görüntüleme • 6 ay önce

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

Xenova

21,602 görüntüleme • 2 ay önce

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Qwen3.8 Max UD-Q1_0 4× DGX Spark + llama.cpp/MTP3 53,448 tokens · 1h54m44s · 7.76 tok/s NVIDIA AI Qwen Qwen Developers

Wësche

15,149 görüntüleme • 2 ay önce

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

Lewis Tunstall

118,819 görüntüleme • 5 ay önce

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Stop paying for Claude Code! Use it for FREE with ollama I just set up Claude Code running locally on my computer. Zero API costs, complete privacy, and it actually works for real coding tasks. Full step-by-step guide in my latest video:

Mervin Praison

57,732 görüntüleme • 8 ay önce

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 görüntüleme • 1 ay önce

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

There's a free 120 billion parameter AI you can run on your own computer. Zero internet. Zero API costs. Zero monthly fees. It's called Nvidia Nemotron 3 Super. And it just changed what solopreneurs can do with AI agents. Here's the setup: → Download Ollama. Copy one command into your terminal. → Connect it to OpenClaw in under 10 minutes. → Your AI agent now runs 24/7 on WhatsApp, Discord, or Telegram. → It answers customers, runs tasks, and manages workflows while you sleep. → 256,000 token context window. It reads entire documents without forgetting. No subscriptions. No cloud. No one else touching your data. A full AI employee stack for $0. Save this. Set it up this weekend.

Julian Goldie SEO

29,883 görüntüleme • 7 ay önce

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

Ollama me daba 21 tok/s con Qwen3.6 35B (12 GB VRAM). Mismo modelo, misma GPU → llama.cpp + -ncmoe 15 = 70 tok/s. No es magia. Es un flag que Ollama no expone. Comando exacto: llama-cli -m ~/models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf -ngl 99 -ncmoe 15 -p "Hola" Demo real aquí 👇

OscarMartin

302,902 görüntüleme • 5 ay önce

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗹𝗹𝗮𝗺𝗮 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗶𝘀 𝗮 𝗳𝘂𝗹𝗹𝘆 𝗹𝗼𝗰𝗮𝗹 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗳𝗿𝗲𝗲 𝗳𝗼𝗿𝗲𝘃𝗲𝗿. #3 open model in the world. Runs on your laptop. Zero API costs. Zero data leaving your machine. Setup in 3 steps: → Install Ollama and run "ollama pull gemma4" — downloads the model → Install OpenClaw and select Ollama as your provider → Connect Telegram — message your agent from anywhere, completely local One prompt built a working SEO calculator — HTML, CSS, JavaScript, no debugging. Coding benchmark jumped from 110 to 2,150 vs the previous Gemma. Apache 2.0. No restrictions. No monthly bill. No surprises. Want the full setup guide? DM me.

Julian Goldie SEO

77,202 görüntüleme • 6 ay önce

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

You can now run OpenClaw completely free using Qwen 3.6 Plus 🤯 Just plug Alibaba’s free API into OpenClaw, Claude Code, Hermes or Kilo Code… and suddenly you’ve got a powerful AI agent with zero cost. This might be the best free AI setup right now. 🚀

Julian Goldie SEO

31,717 görüntüleme • 7 ay önce

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 görüntüleme • 4 ay önce

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)

Georgi Gerganov

786,184 görüntüleme • 7 ay önce

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

i just recorded a full tutorial on pi durable 1:48 Checkpoints + crash recovery 8:38 Harness vs. app 13:46 App architecture + storage 19:30 Six setup steps 23:11 Local Qwen + llama.cpp demo + restart recovery

Alejandro AO 🤗

31,529 görüntüleme • 10 gün önce

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

LFM2.5-2.6B is live! At only 2.6B, it shouldn't be this good at agentic tasks. Here you can see how a 100% local agent is cleaning up my Desktop. I wrote a step-by-step guide on how to set it up yourself: • Model: LFM2.5-2.6B • Server of choice: llama.cpp, LMStudio, and more • Harness of choice: OpenClaw, Hermes Agent, Pi

leonie

64,192 görüntüleme • 2 ay önce

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

Victor M

171,948 görüntüleme • 5 ay önce