THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running... on 127.0.0.1:8080 plugged it into VS Code, connected it to OpenClaw, 53 tok/s on an M1 Max with 262K context zero wrappers, zero timeout errors, zero API fees bookmark & like this before your next timeout error hits​​​​​​​​​​​​​​​​ full breakdown of my raw llama.cpp setup ↓show more

Who is speaking in this video? Is that you? It's not an American he is clearly English. Bit confused by the video and the link to 2-GPU build below it. How are they related?

this developer figured out the same thing I wrote about: strip the wrappers, run llama.cpp raw. different hardware, same conclusion

cut the wrappers, run the source. solid setup

exactly the mindset

zero api fees forever wins

@asakura0x correct conclusion👍🏻

😼

Thank you but I'm not American

sorry 🙏🏽

This llama launcher gives us full control how llama-server is executed

Please link others videos to the source with your commentary. Thank you! @0x5am5 thanks for the contribution!

Most developers debug the wrapper. Never the engine. LM Studio. Ollama. Different names. Same llama.cpp underneath. The timeout wasn’t a bug. It was a layer you didn’t need. Going direct to llama.cpp isn’t advanced. It’s just removing the abstraction that was hiding the problem. 53 tok/s. 262K context. Zero fees. That’s what’s under the hood.

the asymmetric KV quantization bit is what really matters here. most people don't realize that KV cache is the memory bottleneck on consumer GPUs, not weights. Q8 weights + Q4 KV cache is a genuinely different tradeoff than Ollama's defaults, and going direct llama.cpp lets you tune that knob. the wrappers abstract away exactly the setting you'd want to change

who is this guy ? credit/reference

Or just configure lm studio to run the tool correctly, it has tons of configuration options and surely more could be added if that were necessary.

Thanks for the article. Can you share the link to the source video? Like you, that fellow deserves a follow.

yep definitely need to save this..

230 people already did

Classic move: abstract the abstraction, then debug the original abstraction.

No api fees👏

I'm starting to watch it, it sounds very tempting

once you start you don’t go back

I have a homogeneous GPU set up. 5060ti and 3060 ~ 28.5 gb's. Can meaningfully run qwen 27/35b My Thoughts: - heterogeneous is better(same gpu arch) - You'll want higher mem-bandwidth - Can run a model for workflows locally, but still want to build with a SOTA model.

😁 so it took this idiot (and also you) 3 years to understand ollama was just llama.cpp under the hood. Great job keep it up 🤦‍♂️

In fact, with AI it's very easy

easy once you know the stack. that’s the whole point of the article

he know facts how it's all works

no theory. just what actually runs

@grok is this real setup, Worth duplicating?

struck the OpenAI bill with 2 GPUs. not showing off, this is a business model. platform prices can change anytime. your hardware can't. more context:

but how do you solve the hardcoded 2GB metabuffer ?the server will crash when >2GB! so llama.cpp you will not be able to run as a 24/7 server with 120k context ! im testing now for weeks for stable 24/7 server inference for lokal production! please share with us the code!

Building locally seems like a game changer for overcoming those timeout errors!

Credit need to go to @0x5am5 and the original video:

Qwen 3.6 35B from Huggingface is 72gb... So this story is bullshit right off the bat.

@grok solid or no? Risk involved?

I have an almost identical experimental machine with dual 4080S graphics cards. I always cram it with the largest model that can run, but it seems this isn't the best approach. Thanks for the experiment.

🤣This happens too, I thought for a long time...🤷

Check out TurboAgent. It does the weights and KV compression.

And what is all of this for? What is he actually doing with this? Summarizing articles every morning?

Today in “duh”

lmao yeah no shit it's all llama.cpp, what else would it be? hahaha

@grok summarise the video.

すごい技術ですね、私も試してみたいです。