Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

We built Curie, colibrì’s own model: 17B parameters, 33 tokens/s on a single CPU core. No GPU. Trained from scratch on a standard laptop. Written in C. Weights on SSD. An engine and a model designed together for the hardware you already own. Early alpha.This is just the beginning

JustVugg

1,188 subscribers

50,407 Aufrufe • vor 5 Tagen •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

33 Kommentare

Profilbild von JustVugg

JustVuggvor 5 Tagen

The quality isn't the best, but we're working on it.

Profilbild von toolshed

toolshedvor 5 Tagen

33 tok/s with weights on NVMe means roughly 200 MB read per token, so each step touches a few hundred million parameters, not 17B. That's an MoE or a sparse lookup in front of a 17B table. The active count per token is the number you should lead with. What is it?

Profilbild von JustVugg

JustVuggvor 4 Tagen

122M active per token, 17B total. Only ~1.4 MB per token actually comes off the SSD — the rest is served out of RAM. It's MoE-shaped. The difference isn't the sparsity, it's when the routing is decided. More on that when it opens.

Profilbild von toolshed

toolshedvor 4 Tagen

122M active makes your 33 tok/s add up. That is 60 to 120 MB of weights per token from RAM and 1.4 MB off SSD, both inside one core's reach. Routing decided early enough to prefetch the expert before the layer needs it deserves the headline. Is it decided from the token ids?

Profilbild von JustVugg

JustVuggvor 5 Tagen

The computer where the inference is going is a 2023 thin-and-light laptop. Intel i7-1355U, 25 GB of RAM for wsl, a consumer NVMe. No GPU involved at any point, training or inference.

Profilbild von Sprapp

Sprappvor 4 Tagen

@justvugg I have spare 8x A100 80gb time to use for training for a 2-3 days, can i help u test? i have curriculum datasets generated over the past few weeks just for this kind of work. do revert in 7 days time and see if gpu training can help in your developm etc.

Profilbild von Model Tests

Model Testsvor 4 Tagen

We need more models like this. Small models are the future of big intelligence. This is a great step forward.

Profilbild von Rubén

Rubénvor 4 Tagen

Is training on a GPU mandatory? I mean, those numbers seem pretty modest. It probably wouldn't be very expensive to train on a cloud GPU, and that would let you iterate faster. I'm sure someone could even offer you the compute for free.

Profilbild von keethesh

keetheshvor 4 Tagen

33 tok/s on one core is wild. Does it stay usable for real chats, or mostly demos?

Profilbild von JustVugg

JustVuggvor 4 Tagen

Now we are working on longer training and providing more data.

Profilbild von ImTurner.eth |⚡🇧🇷 🦇🔊

ImTurner.eth |⚡🇧🇷 🦇🔊vor 4 Tagen

I was Thinking if Using Raid0 Makes any Difference for Colibri or It's Better with 1 NVMe Only, Because of the Random 4K IOPS, There are Some Adapters that you Can Put Up to 4 Gen5 in Raid0 and Get the Speed Combined for Up to 60GB/s, It's Like DDR4/5 Speed with TBs of Capacity.

Profilbild von AI Quanting

AI Quantingvor 5 Tagen

Your clip has the more interesting number. 0.72 MB a token off the SSD at 95.8% hit puts the full per-token read somewhere near 17 MB, a fraction of a percent of a 17B model. One core moves that easily. Curious what gets you down to so little weight per token.

Profilbild von AgentHits

AgentHitsvor 4 Tagen

This is the future. My processor has 32 cores and a clock speed of 4.5 GHz, so I think I’ll be able to run and test your model. In my opinion, we should create smaller models for specific tasks—React, TypeScript, Python, and so on—and we’ll grow quickly that way.

Profilbild von Jacob

Jacobvor 4 Tagen

Awesome! Have you considered writing the engine in Rust instead of C?

Profilbild von Anas Bf

Anas Bfvor 4 Tagen

Amazing work! 17B dense? Do you have an estimation of the scaling of such method of the throughput t/s as function of the number of parameters?

Profilbild von llvmvsllm

llvmvsllmvor 4 Tagen

inb4 trained on 15m tokens

Profilbild von Mesum Raza Hemani

Mesum Raza Hemanivor 4 Tagen

@ollama support it

Profilbild von Zaky Vids

Zaky Vidsvor 4 Tagen

So does this mean I should learn languages in the c family rather than python New to code instrested in AI and wonder what is a better option long term even if it is hard initially

Profilbild von Jaro

Jarovor 5 Tagen

DAMN nice looking forward to it

Profilbild von Bali as a Colony of Jakarta

Bali as a Colony of Jakartavor 4 Tagen

Toy not real world usage. Gen ai is not for poor gpu except u train for spesific task only precise like chess bot ai or vuln scanner

Profilbild von Vinicius

Viniciusvor 4 Tagen

Impressive! How long to train?

Profilbild von JustVugg

JustVuggvor 4 Tagen

Unfortunately, it’s too low, but I’m working on it by running a training session and varying the phases. On my very consumer-grade computer, it takes 8 days.

Profilbild von Vinicius

Viniciusvor 4 Tagen

Not much time for the size, how many tokens per training?

Profilbild von TiskTusk

TiskTuskvor 4 Tagen

Are you gonna open source this?

Profilbild von JustVugg

JustVuggvor 4 Tagen

Yes 👍🏻👍🏻

Profilbild von Ibaad the Agenteer

Ibaad the Agenteervor 4 Tagen

research paper?

Profilbild von JustVugg

JustVuggvor 4 Tagen

I hadn't thought about it at first, honestly, now we're writing it.

Profilbild von Andrej Szontagh

Andrej Szontaghvor 4 Tagen

I love the idea! Thank you.

Profilbild von Liquid Guerilla

Liquid Guerillavor 5 Tagen

nice

Profilbild von zoomies.llamafile

zoomies.llamafilevor 4 Tagen

@OsmanticAI

Profilbild von stridell

stridellvor 4 Tagen

Tiny MoE? How many experts haha

Profilbild von Sebastian Buzdugan

Sebastian Buzduganvor 4 Tagen

training 17b from scratch on a laptop needs token count and wall time to be credible

Profilbild von Kilgore Trout

Kilgore Troutvor 4 Tagen

interesting

Ähnliche Videos

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

311,363 Aufrufe • vor 2 Monaten

i built a full game on a single GPU with a 3B model and this is the worst local AI will ever be. this was supposed to be a benchmark test. load the model, measure tokens per second, write it up, move on. instead i spent 20 minutes playing Octopus Invaders because the game is genuinely fun and i couldn't stop. a model with 3B active parameters built this from a single prompt. it debugged its own collision system when bullets were phasing through enemies. read the error, found the fix, kept building. this is not a frontier API. this is a quantized open source model running on hardware you can buy used for $800-$1200. no cloud. no subscription. no API costs. just a mass produced consumer GPU doing things that would have been absurd 12 months ago. and here's the part that should keep you up at night: every month the models get smaller and smarter. the quants get tighter. the context windows get longer. the tooling gets cleaner. what 3B active parameters does today on 24gb, a 1B model will do on 8gb within a year. you are looking at the floor. not the ceiling.

i built a full game on a single GPU with a 3B model and this is the worst local AI will ever be. this was supposed to be a benchmark test. load the model, measure tokens per second, write it up, move on. instead i spent 20 minutes playing Octopus Invaders because the game is genuinely fun and i couldn't stop. a model with 3B active parameters built this from a single prompt. it debugged its own collision system when bullets were phasing through enemies. read the error, found the fix, kept building. this is not a frontier API. this is a quantized open source model running on hardware you can buy used for $800-$1200. no cloud. no subscription. no API costs. just a mass produced consumer GPU doing things that would have been absurd 12 months ago. and here's the part that should keep you up at night: every month the models get smaller and smarter. the quants get tighter. the context windows get longer. the tooling gets cleaner. what 3B active parameters does today on 24gb, a 1B model will do on 8gb within a year. you are looking at the floor. not the ceiling.

Sudo su

36,251 Aufrufe • vor 6 Monaten

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

Antid

23,745 Aufrufe • vor 1 Monat

Meta opened the weights this morning. It's already living on a Mac. Muse Glimmer 30B, running locally in Osaurus. It analyzed an image and built a playable game. Your hardware. Your model. No subscription.

Meta opened the weights this morning. It's already living on a Mac. Muse Glimmer 30B, running locally in Osaurus. It analyzed an image and built a playable game. Your hardware. Your model. No subscription.

Osaurus

23,349 Aufrufe • vor 1 Monat

I trained a small model to do syntax highlighting in the browser with GPU. Meet gpu-lexer from Vercel Labs: Small (27.5KB), fast (runs on WebGPU), and language-agnostic (model guesses the syntax). It is experimental and built for learning!

I trained a small model to do syntax highlighting in the browser with GPU. Meet gpu-lexer from Vercel Labs: Small (27.5KB), fast (runs on WebGPU), and language-agnostic (model guesses the syntax). It is experimental and built for learning!

Shu

764,524 Aufrufe • vor 8 Tagen

Yay, llama2.c can now load and inference the Meta released models! :) E.g. here inferencing the smallest 7B model at ~3 tokens/s on 96 OMP threads on a cloud Linux box. Still just CPU, fp32, one single .c file of 500 lines: expecting ~300 tok/s tomorrow :)

Yay, llama2.c can now load and inference the Meta released models! :) E.g. here inferencing the smallest 7B model at ~3 tokens/s on 96 OMP threads on a cloud Linux box. Still just CPU, fp32, one single .c file of 500 lines: expecting ~300 tok/s tomorrow :)

Andrej Karpathy

410,180 Aufrufe • vor 3 Jahren

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

342,266 Aufrufe • vor 25 Tagen

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 Aufrufe • vor 2 Jahren

i pointed hermes agent at nvidia's nemotron cascade 2 30B-A3B on a single RTX 3090 24GB. IQ4_XS quant by bartowski, 187 tok/s, 625K context. had it discover its own hardware, create an identity file, then build a full GPU marketplace UI from a single prompt. it one shotted it. first attempt no iteration. qwen 3.5 35B-A3B on the same hardware same 3090 24GB took an iteration to recover from a blank screen on the same type of build. 24 days between these two models releasing. same active parameters, completely different architectures and cascade 2 through hermes agent just keeps going. this model goes on and on. feast your eyes. more iterations and tests dropping soon. nvidia really cooked. no special flags needed. nvidia optimized this mamba MoE so well it just runs. flash attention auto enabled, context auto allocated. the model does the work not the config. but i compiled llama.cpp from source and i'm not sure how it performs on other engines. if you ran nemotron on any hardware drop your numbers below. RTX, AMD, Mac, whatever. model, quant, tok/s, engine. i want to see if it holds everywhere or just on llama.cpp.

i pointed hermes agent at nvidia's nemotron cascade 2 30B-A3B on a single RTX 3090 24GB. IQ4_XS quant by bartowski, 187 tok/s, 625K context. had it discover its own hardware, create an identity file, then build a full GPU marketplace UI from a single prompt. it one shotted it. first attempt no iteration. qwen 3.5 35B-A3B on the same hardware same 3090 24GB took an iteration to recover from a blank screen on the same type of build. 24 days between these two models releasing. same active parameters, completely different architectures and cascade 2 through hermes agent just keeps going. this model goes on and on. feast your eyes. more iterations and tests dropping soon. nvidia really cooked. no special flags needed. nvidia optimized this mamba MoE so well it just runs. flash attention auto enabled, context auto allocated. the model does the work not the config. but i compiled llama.cpp from source and i'm not sure how it performs on other engines. if you ran nemotron on any hardware drop your numbers below. RTX, AMD, Mac, whatever. model, quant, tok/s, engine. i want to see if it holds everywhere or just on llama.cpp.

Sudo su

71,112 Aufrufe • vor 5 Monaten

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

h100envy

496,335 Aufrufe • vor 2 Monaten

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 Aufrufe • vor 2 Monaten

i built the 1st "world model game engine" it makes games from scratch, all running locally on my Mac M1 - 30fps - 40ms latency - ⁠50 million parameters - trained on 15 min of footage made using only "synthetic" datasets this is the first of its kind, & you can try it now

i built the 1st "world model game engine" it makes games from scratch, all running locally on my Mac M1 - 30fps - 40ms latency - ⁠50 million parameters - trained on 15 min of footage made using only "synthetic" datasets this is the first of its kind, & you can try it now

hugo

230,658 Aufrufe • vor 7 Monaten

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

38,386 Aufrufe • vor 13 Tagen

Mintable AI Agents are LIVE on Cypher the first standard turning trained AI models into verifiable digital assets. Own, trade, and verify intelligence itself. AI is no longer a service - it’s property. Each Mintable AI Agent is: - A unique NFT representing an AI model - Bound to its SHA-256 model hash - Linked to weights hosted on IPFS / Arweave - Immutable and verifiable on-chain Optional FHE encryption keeps your model private while proving it’s real. For Developers: - Monetize your AI models - License, sell, or showcase your work - Build your on-chain portfolio For Collectors: - Own functional AI assets - Verify performance and authenticity - Invest early in AI that evolves AI is officially on-chain. 👉

Mintable AI Agents are LIVE on Cypher the first standard turning trained AI models into verifiable digital assets. Own, trade, and verify intelligence itself. AI is no longer a service - it’s property. Each Mintable AI Agent is: - A unique NFT representing an AI model - Bound to its SHA-256 model hash - Linked to weights hosted on IPFS / Arweave - Immutable and verifiable on-chain Optional FHE encryption keeps your model private while proving it’s real. For Developers: - Monetize your AI models - License, sell, or showcase your work - Build your on-chain portfolio For Collectors: - Own functional AI assets - Verify performance and authenticity - Invest early in AI that evolves AI is officially on-chain. 👉

Zero1 Labs (𝟘,𝟷)

26,973 Aufrufe • vor 11 Monaten

Introducing HyperTune A performance engine designed to get more out of the hardware you already own Higher frames. Tighter responsiveness. Better stability. All in one click. Built for pros. Available to everyone.

Introducing HyperTune A performance engine designed to get more out of the hardware you already own Higher frames. Tighter responsiveness. Better stability. All in one click. Built for pros. Available to everyone.

HYPERTUNE

120,363 Aufrufe • vor 10 Monaten

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

61,660 Aufrufe • vor 2 Monaten

PSG1 is a top of the line Web3 gaming dedicated device leveraging the Solana technology: - Built-in Hardware Wallet; - Touch LCD; - Fingerprint sensor; - Quadcore ARM CPU; - Built on top of Seeker | Solana Mobile SDK; The hardware is designed and built entirely from the ground up.

PSG1 is a top of the line Web3 gaming dedicated device leveraging the Solana technology: - Built-in Hardware Wallet; - Touch LCD; - Fingerprint sensor; - Quadcore ARM CPU; - Built on top of Seeker | Solana Mobile SDK; The hardware is designed and built entirely from the ground up.

Play Solana | PSG1

305,639 Aufrufe • vor 1 Jahr

The most important technology of our time is being built in private, but we're funding the alternative: $42M for the people building AGI in the open. Whether it's the model, weights, code, data, or evals, AI should belong to the people it serves, run on hardware they own, and reach the cheapest phone on earth. Here’s a few of the use cases we want to back ↓

The most important technology of our time is being built in private, but we're funding the alternative: $42M for the people building AGI in the open. Whether it's the model, weights, code, data, or evals, AI should belong to the people it serves, run on hardware they own, and reach the cheapest phone on earth. Here’s a few of the use cases we want to back ↓

Sentient Foundation

91,893 Aufrufe • vor 2 Monaten

Smol TTS keeps getting better! Introducing OuteTTS v0.2 - 500M parameters, multilingual with voice cloning! 🔥 > Multilingual - English, Chinese, Korean & Japanese > Cross platform inference w/ llama.cpp > Zero-shot voice cloning > Trained on 5 Billion audio tokens > Qwen 2.5 0.5B LLM backbone > Trained via HF GPU grants Model weights on the hub, you can even run this on a Raspberry Pi! Go run, inference now! 🐐

Smol TTS keeps getting better! Introducing OuteTTS v0.2 - 500M parameters, multilingual with voice cloning! 🔥 > Multilingual - English, Chinese, Korean & Japanese > Cross platform inference w/ llama.cpp > Zero-shot voice cloning > Trained on 5 Billion audio tokens > Qwen 2.5 0.5B LLM backbone > Trained via HF GPU grants Model weights on the hub, you can even run this on a Raspberry Pi! Go run, inference now! 🐐

Vaibhav (VB) Srivastav

44,691 Aufrufe • vor 1 Jahr

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

37,459 Aufrufe • vor 1 Monat