all this, at home, fully offline :)

this is the speed you can expect from @FrameworkPuter Desktop, for models worth using it's basically as fast as using services like ChatGPT

beautiful!

@grok what do you need to run this ? How difficult or time consuming is this setup ?

it use llama-server for most of my on-device solutions C++ is 10x better than annoying py bindings

@FrameworkPuter @FrameworkPuter stahp my wallet can’t

not even a 4090 can run an 120b I think that's crazy

sweet thanks for posting this!

Okay I'm still stuck on N_K_M GGUF quants. Whats so magical about these FP4 formats?

This is crazy good

woaaa so many token a sec, I didn't knew framework desktop were this fast omg, aure they arm based with shared memory to be this fast ??

Drop tutorial

Looks like groks interface mixed with Ollama. If I’m running windows vms would llama.cpp still be usable or do I need to go Ubuntu or Linux?

gorgeous

how to?