left curve dev
@leftcurvedev_ • 6,251 subscribers
low iq, high vram — sharing local ai and coding stuff
Shorts
Videos
it's going to get harder to justify a $200 claude sub in the next few months
left curve dev229,555 просмотров • 2 месяцев назад
Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below
left curve dev166,063 просмотров • 2 месяцев назад
I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓
left curve dev29,946 просмотров • 2 месяцев назад
🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?
left curve dev21,224 просмотров • 2 месяцев назад
I know some bros want to see what happens when we push the value even further, so here we go: ⚪️ -ncmoe 25 → 41.8 tok/s RAM: 12GB & VRAM: 6.9GB 🔴 -ncmoe 23 → 43.8 tok/s RAM: 12.2GB & VRAM: 7.4GB 🟢 -ncmoe 21 → 38.6 tok/s RAM: 12.4GB & VRAM: 7.8GB 🔵 -ncmoe 19 → 19.8 tok/s RAM: 13.8GB & VRAM: 7.8GB As you can see, there's a sweet spot with the VRAM usage. Play around and monitor to find the right value for your setup, you can use llama.cpp web ui to monitor speeds easily Sweet spot seems to be 25-23 for 8GB VRAM ✅ +40tok/s for Qwen3.6 35B with 64k q8_0 context on a 8GB card is very impressive just by using base llama.cpp, and we didn't even try Turboquant, MTP or Dflash yet! I'll focus on these next 👀 (Server flags and setup in the quoted tweet)
left curve dev18,049 просмотров • 2 месяцев назад
here's what i vibecoded today: punchingface 🥊 an app to make Hugging Face models fight each other on coding and canvas challenges, built with qwen3.6 35b a3b in 24 hours! benchmarks numbers don't mean anything anymore, we need a way to visualize what the models are actually capable of, and canvas are one the best way to showcase it imo. why? because one single error in the code and everything breaks it shows the differences in a matter of seconds, way easier than manually reviewing the quality of the code produced for a complex project; much needed in the space with all the new finetunes dropping everyday! i did a little demo here with qwen3.6 vs qwopus glm 18b merged, the frankenstein model from Kyle Hessling the winner is clear here, qwen is crazy good and has nothing to prove. that said, qwopus 18b isn't terrible at all; the result isn’t the prettiest to the eye, but hey… it works! i've seen so many models just output a blank page (completely non-working code) so this is already a win frankenstein talks and thinks but he needs some extra brain surgery 🧠 results were expected (it's a very experimental model) but love the effort in the 18b direction from jackrong and kyle! the app was entirely vibecoded with qwen3.6, i didn't edit a single file manually. i can say with confidence that it really has the intelligence of claude sonnet 4.5 at a speed of 125tok/s on an rtx 5080 which models should i make fight next?
left curve dev19,114 просмотров • 3 месяцев назад
Больше нет контента для загрузки