Depth Anything 3 now runs as pure C++/ggml (ggml)... . No Python, no PyTorch, no CUDA toolkit at inference, just one self-contained GGUF. It's faster than PyTorch on CPU! and ties speed on GPU. The CPU win came from the last place..I'd have looked. Quantized GGUF on Hugging Face🤗 Shout out to Georgi Gerganov for ggml (we are building a ggml-world!❤️) and to ByteDance Open Source and Depth Anything 3 authors Bingyi Kang Jun Hao Liew Donny Y. Chen !show more

parakeet.cpp: native C++/ggml (ggml) inference for NVIDIA AI Developer's... Parakeet, one of the best speech-to-text models out there, from the LocalAI team. Every Parakeet model (TDT/CTC/RNNT/hybrid + cache-aware streaming), byte-for-byte identical output to NeMo, now running anywhere with no Python and even a bit faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIA AI Developer for releasing Parakeet! 🧵show more

You don't need a GPU for fast studio grade... voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?show more

Watching llama.cpp do 40 tok/s inference of the 7B... model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. Congratulations Georgi Gerganov ! This is a triumph.show more

UNCENSOR ANY AI MODEL ON YOUR OWN GAMING PC... heretic 2.0 just dropped and it strips the censorship out of open models with no retraining how to set it up: 1. install python 3.10 or newer 2. install pytorch 2.6+ for your gpu 3. open terminal and install heretic pip install -U heretic-llm 4. run it on any hugging face model heretic Qwen/Qwen3-4B-Instruct-2507 5. save it, upload to hugging face, chat with it, or run benchmarks if you have low vram then turn on 4-bit modeshow more

Just dropped on HF — NeuTTS Air Next-gen on-device... TTS that matches cloud-level quality while staying fully open source. > Real-time speech synthesis on CPU/GPU > 3-second voice cloning, no cloud or data upload > Compact: under 200 MB, runs on mobile and edge devices > Multilingual and expressive > Developed by Neuphonic , optimized for speed and fidelityshow more

llama.cpp isn't just for text LLMs anymore. Pure C++... zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!show more

8GB RTX 4060 Laptop. ~16GB Ornith IQ3_S GGUF. 44.5... tok/s streaming locally. FreeToken didn’t have the qwen35moe GGUF path needed for this model, so I implemented it and submitted PR #131 upstream. 👀 I also exposed the K/I GGUF quant types its CUDA kernels already supported through the Python loader. This is the result. Setup: Ornith-1.5-35B-A3B 35B MoE / ~3B active IQ3_S GGUF · ~16GB RTX 4060 Laptop · 8GB VRAM FreeToken · FlashInfer MoE backend: offload On the attached full-prompt run, I measured ~44.5 tok/s streaming on screen. In the same runs, FreeToken’s server decode counter reported 46.7 → 50.1 tok/s. VRAM stayed at 6,879 / 8,188 MiB, with 20GB host RAM and 98% GPU utilization during generation. Load time: 65s. My llama.cpp CPU reference on the same model was 11.1 tok/s. The interesting part isn’t just “35B on an 8GB GPU.” The code path itself was missing. FreeToken’s CUDA side already supported more GGUF quant types than its Python loader exposed, including the K/I quants needed for this IQ3_S model. And qwen35moe models like Ornith needed their own GGUF architecture/loading path. I implemented both pieces in my fork and opened them upstream in PR #131. Once loaded, FreeToken can use GPU + CPU + host RAM as an elastic MoE inference system instead of requiring the entire checkpoint to live in VRAM. That fits Ornith well because only a subset of its experts are active per token. Important caveat: PR #131 is still open. This is not merged upstream yet. I’m not calling this official Ornith support. I’m showing a working implementation, the code is public, and the attached terminal video is the actual run. PR #131: My fork: Next I’m going to test where this path holds up across more quants, longer context and real agent workloads.show more

so we already have an open source alternative to... jev... and it's 6-7x faster?! it's a typed-decision classification system: no chat, no generated text, just fast yes/no, scoring, or choice answers. > runs in under 1gb of memory > free on hugging face > runs on a laptop, or even a phone there's a real demo where it plays snake live, making a fresh decision every single move. two honest limits, straight from the project itself: > only 512-1,024 token context, some cases won't fit > weaker generalization than jev out of the boxshow more

We are building “Open Source Nano Banana for Video”... - here is open source demo v0.1 We are open sourcing Lucy Edit, the first foundation model for text-guided video editing! Get the model on Hugging Face 🤗, API on @FAL, and nodes on ComfyUI 🧵show more

Holy shit... Microsoft open sourced an inference framework that... runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.show more

pip install spectralquant ✂️ Up to 6.62x KV cache... compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inferenceshow more

They were building in stealth for 2 years, I... was building in stealth for 2 hours… Happy to open source Qwen-2.5-1B-RLCD, 5x faster on-device inference for JSON workloads that need to be type-safe. ⚡️Demo below on a M4 MacBook⚡️ every LLM has the ability to efficiently batch inference every key of a JSON at the same time and generate probabilities from a set of possible categories. No new training required, but it’s easy to optimize if you need! On hugging face now!show more

I don’t have the BRAM for a depth buffer... so it’s time to get the memory controller up and running for this fjord torus. 100% FPGA logic 3D pipeline, no CPU.show more

Microsoft killed the GPU mafia 🤯 They finally open-sourced... their 1-bit LLM inference framework called bitnet.cpp. It lets you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.show more

THIS ENGINEER SKIPPED 7,862 WORDS OF AI ROADMAP AND... SHIPPED ON A WEEKEND a Pi Zero 2W, an IMX500 camera and a solar panel, that's the whole stack inference runs on-chip so there's no cloud, no latency and no monthly bill Spotted Dove at 0.95, King Parrot at 0.88, runs forever on sun code is open source and the 3D files are free, you just print the case and buy the parts and it works better than most paid wildlife monitoring services out there full video: code (open source): 3D print files: solar power board: save this so you actually have everything in one place when you decide to build itshow more

Step 4 to achieve truly serverless GPUs for AI... inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.show more

Microsoft just changed the game 🤯 They've open-sourced bitnet.cpp,... a 1-bit LLM inference framework. It let's you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.show more

Someone just built a desktop app that that generates... 3D models from images and runs 100% locally. It's called Modly. It runs entirely on your GPU, no cloud, no API bills. Just drop an image and get a 3D mesh. 100% Open Source.show more

Llama 2: Now on Hugging Chat 🤗🦙 Try out... the 70B Chat model for free with super fast inference, web search, and powered by open-source tools! 👉show more

New vs Old! Epyc 64-Core CPU vs DEC VAX!... I like to compare weird things on an even playing field, and so I installed NetBSD 10.1 on both a VAX 4000-705A and on an Epyc 8534P, 64-core CPU with 128GB of RAM. The 1993 VAX runs at 112MHz (pretty fast for a VAX!) Then I set them both to building the NetBSD source tree. Epyc on the top, VAX on the bottom. And this VAX is at least 50x as fast as the original VAX 11/780!show more