yeah this isn’t happening 😭

i’m pretty sure m4 runs it well enough

@uwukko I’ll download it on my M4 Max just for you

@uwukko

i knew that it’d run very slowly but i still wanted to play around with it cuz it seems cool

is this just essentially running on swap space

most definitely

@istartedthebits Does macos even have swap?

@uwukko Macs have had swap since OS X 10.0. If you don't have some virtual memory system, programs would just get terminated/lock up/error out whenever you use up available RAM

You should try the 120 billion one

i don’t have enough storage for that

@0xeik What's the storwge requirement for that?

@God_Intelligens @uwukko 100 TB idk

@uwukko Wtf, how the heck one suppose to run that model locally?

@God_Intelligens @uwukko Bro I’m not serious Serious answer now : The 20b one was 12-13 gb so using an estimate you can try to find what the larger one could be

"thought for 0.01 seconds"

120b runs fast in @ollama for me on m3

air m4 16gb

@BounceNStretch runs great on M4 Max

Below are two tiny

,

self-

Is it lm studio?

yeah

try quantized version, will be made available by unsloth soon

Haven't tried on my Mac but my Ubuntu box with 32 GB RAM and 8 GB GPU runs oss:20b fine. Getting ~15 tps and 0.82s TTFT

super quick for me on my m4 mac:

0.1 tokens/s is enough

just wait for th quantized distilled 3b parameter 1bit version

below are two tiny... uh.... mmm... erm.... self -... contained...

The speed really reinforces the point it's a glorified autocomplete.

you need more RAM the 20b struggles with 16GB RAM , it is not enough.

My god! That model needs 24gigs of ram! I are using roughly 18g from your solid state drive for that to even run!

really pushing that 8gb

1 token per 2-3 business days

You did exactly what I wanted to do—and now I won’t have to. Kudos.

Doesn't matter what m-series chip you have, if you dont have min 24gb ram it won't run good

some people think if you have 24 gb ram you can run the 20b model

Below are two tiny

MLX version coming soon :)

Is flash attention available for Apple HW? llama.cpp cmd line args: -fa -np 3 --mlock --no-webui --metrics -ctk q4_0 -ctv q4_0 fa: flash attention ctk and ctv are important, as they save RAM (it instructs llama to use less precision, but for the 20b model it should be fine).

Oof, thanks for trying it out so I don’t have to lol. Been burned too many times running my own LLMs

same lol :(

holy shit just put it out of its misery 💀

pulling it, will report back shortly

Me on my exams:

on the bright side you probably only used a couple cups of water and maybe like 3 trees to make this :)

@WoodManAmen i haven’t used anything

you generated 4 words according to the ai environmentalists on tiktok you just burned down the amazon rainforest

@WoodManAmen and i’d do it again