parakeet.cpp: native C++/ggml (ggml) inference for NVIDIA AI Developer's... Parakeet, one of the best speech-to-text models out there, from the LocalAI team. Every Parakeet model (TDT/CTC/RNNT/hybrid + cache-aware streaming), byte-for-byte identical output to NeMo, now running anywhere with no Python and even a bit faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIA AI Developer for releasing Parakeet! 🧵show more

Depth Anything 3 now runs as pure C++/ggml (ggml)... . No Python, no PyTorch, no CUDA toolkit at inference, just one self-contained GGUF. It's faster than PyTorch on CPU! and ties speed on GPU. The CPU win came from the last place..I'd have looked. Quantized GGUF on Hugging Face🤗 Shout out to Georgi Gerganov for ggml (we are building a ggml-world!❤️) and to ByteDance Open Source and Depth Anything 3 authors Bingyi Kang Jun Hao Liew Donny Y. Chen !show more

As a supporter of the open weights ecosystem, we're... proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.show more

90% of "AI developers" just download pre packaged GGUF... files from Hugging Face, hit run, and call it a day. The top 10% know how to pull the raw safetensors, run the math, and quantize massive models into Q4_K_M themselves. If you think llama.cpp can only execute models, you’re missing the best part of the open source ecosystem. It’s a high performance optimization suite. Manually stripping 69% of the VRAM footprint off a brand new model architecture is where real infrastructure value is made. If you want to actually master local inference and deploy models like Google’s massive Gemma 4 12B it on consumer NVIDIA hardware using llama.cpp, you need to learn this pipeline. Let's build it. I just took the raw 22.7 GB Gemma 4 baseline and manually compressed it down to a 7.02 GB Q4_K_M GGUF artifact using llama.cpp. That is a 69% reduction in footprint. No quality loss. No VRAM bottlenecks. Just native, hardware accelerated C++ inference running a full 2,50,000 token context window on a dual NVIDIA Tesla T4 setup. Stop melting your VRAM on unoptimized weights and stop relying on other people's pipelines. Own your stack. I mapped this entire architecture from dynamic binary fetching to raw quantization and real time GPU streaming into a single, bulletproof notebook. Notebook link is in the comments below. Bookmark this blueprint for your next deployment and tell me which quantization works best for your workflow and model.show more

Frontier world models now run on edge GPUs. ⚡... Now openly available, NVIDIA Cosmos 3 Edge brings advanced world-model capabilities to local devices. The 4-billion-parameter omnimodel can understand and generate text, image, video, ambient sound and action for physical AI across robotics, autonomous vehicles and smart infrastructure. 🔗 #SIGGRAPH2026show more

The new open-weights Hailuo AI (MiniMax) H3 is live... on OpenRouter thanks to MiniMax (official)! H3 is a lightweight video model for text, image, audio, & video input, with native audio-visual output. Built for instruction-led edits, text/brand rendering, and video-2-video motion transfer.show more

We’re proud to share that $vMINT is now a... member of the NVIDIA Developer Program 🌟 This will provide us with the following things (among others): Access to over 150 SDKs for advanced software development Comprehensive training resources through the Deep Learning Institute and NVIDIA On-Demand for skill enhancement Early access programs Stay tuned for more exciting updates 👀 #Volumint $vMINT #nvidia $nvda #nvda #AI #Marketmakingshow more

SMRT x NVIDIA🤝 We are excited to announce we... have been enrolled into the NVIDIA Developer program! Included in this prestigious program is access to advanced tools and SDKs, training resources, early access programs and unlimited use of NVIDIA On-Demand Services. Additionally, the SmartMoney team will be attending the NVIDIA GTC AI Conference, networking with the top brains in AI development who will be in attendance, including Jensen Huang CEO of NVIDIA and Brad Lightcap COO of OpenAI . As a nascent AI Blockchain project, this partnership will be crucial as we scale and build out our platform and services, given NVIDIA's wealth of resources in the AI industry. Make sure to follow our twitter for more updates as we continue to partner with and learn from the strongest projects and firms in our field.show more

Today, Nesa is excited to announce cross-chain support with... Sei. As part of this new development, Nesa will be featuring a dedicated ramp for Sei’s rich ecosystem of applications to seamlessly integrate with AI. Now any Sei dapp can run AI inference on hundreds of models supported by Nesa, including the largest LLMs and most popular models across Vision, Language, and Generative AI. For developers, we’ll have information soon on how to access AI inference on Nesa via smart contracts from the Sei blockchain. Stay tuned for more details on this ground breaking development for decentralized AI.show more

Nvidia just put a $250,000 cloud workload on your... desk for $2,999 - and killed your $1,900/month AWS bill in the process You don't rent it, you don't manage it, you don't pay a single cloud bill - you just plug it in and let it eat the workloads you used to wire to AWS every month It looks like a small Mac mini, it's actually a full GB10 Grace Blackwell stack with 128GB of unified memory running models up to 200B parameters It's called DGX Spark, the consumer version of the rack Nvidia ships to OpenAI The reason Nvidia did this is simple Cloud GPU pricing is a tax on every developer building AI right now $1,900/month per seat, billions in margin flowing to AWS, Lambda, and CoreWeave Nvidia just cut themselves in by removing the cloud entirely Their solution is to skip the middleman, ship the rack to your desk, and let you keep every dollar of margin you used to wire to a hyperscaler This is much cheaper, faster, and you own the asset at the end But there is still a question nobody is answering yet, what happens to AWS, GCP, and Lambda when 500,000 developers move their inference back to a $2,999 box on their desk Also, technically you can stack four of these and run a 1.6 trillion parameter model locally for under $12,000 Even a single Spark out-performs the cloud subscription Anthropic engineers were running two years ago bookmark this, it pays back in 60 days 👇show more

🎉 Congrats to Thinking Machines on TML Inkling—a 1T-parameter... open-weight model supported in vLLM from Day 0. Highlights: • Natively multimodal across text, image, and audio • Up to 1M-token context • New architecture with relative attention, short convolutions, and MoE expert sinks • 8 MTP heads for speculative decoding vLLM supports both NVFP4 and BF16 checkpoints, optimized for NVIDIA Blackwell and Hopper, reaching up to 380 tok/s/user on 4× GB200 with MTP. Huge thanks to the Thinking Machines Lab team for the close collaboration 🙏 Read about implementation below 👇show more

llama.cpp isn't just for text LLMs anymore. Pure C++... zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!show more

Introducing 𝗦𝘂𝗽𝗲𝗿 𝗝𝗦𝗢𝗡 𝗠𝗼𝗱𝗲, a framework for low latency... structured output generation from LLMs. Generate JSON up to 𝟮𝟬𝘅 𝗳𝗮𝘀𝘁𝗲𝗿 from OpenAI and open source models. ❌ No need to threaten the model, tip the AI, etc ❌ Built with Alex Derhacobian 🔧 🧵👇show more

NVIDIA might have just declared war on the cloud... GPU business For years, AI builders had one option Rent compute Pay every month Watch the bill grow every time usage increased Now NVIDIA is putting serious AI hardware directly on people's desks Small enough to fit next to a monitor Powerful enough to run workloads that used to require expensive cloud infrastructure That's why this launch is getting so much attention The real story isn't the hardware specs It's the business model shift Every month, developers send money to cloud providers for inference, testing, fine-tuning and AI applications The question nobody can answer yet is what happens if enough developers decide they'd rather buy infrastructure once than rent it forever Because if local AI hardware keeps getting more powerful, the economics start changing very quickly Cloud providers built empires on renting access to compute NVIDIA is betting more people will eventually want to own it And that's a much bigger story than a new piece of hardware sitting on a deskshow more

They were building in stealth for 2 years, I... was building in stealth for 2 hours… Happy to open source Qwen-2.5-1B-RLCD, 5x faster on-device inference for JSON workloads that need to be type-safe. ⚡️Demo below on a M4 MacBook⚡️ every LLM has the ability to efficiently batch inference every key of a JSON at the same time and generate probabilities from a set of possible categories. No new training required, but it’s easy to optimize if you need! On hugging face now!show more

Introducing MLX-Swift-TS An SDK for running time series foundation... models fully on-device on Apple Silicon. When I joined Datadog, Inc. , I was introduced to Toto, our time series foundation model, and got excited about zero-shot forecasting across different domains. While building a health copilot app, I realized there wasn’t a simple way to run models like these locally on device. So I built one. MLX-Swift-TS exposes a common TimeSeriesForecaster interface for loading and running multiple time series architectures directly in Swift using MLX. No server required. The attached video shows on-device forecasting running inside a native Swift app. Huge thanks to Awni Hannun and the MLX team for building MLX and its Swift API, Prince Canuma for inspiration on MLX SDK patterns, and Ameet Talwalkar and the Datadog team for Toto.show more

MolmoAct2 is landing in LeRobot! Ai2's open Action Reasoning... Model combines a Molmo2-ER vision-language backbone with a flow-matching continuous action expert to predict robot action chunks from images, language instructions, and proprioceptive state. An open robot foundation model built for real-world control, with strong out-of-the-box performance and easy fine-tuning in LeRobot. Pick-and-place inference running on NVIDIA DGX Spark! Blog: Paper: Thanks to Ai2 Jiafei Duan Haoquan Fangshow more

Llama.cpp running on a Sun Ultra 45 from 2006.... Dual 1.6GHz UltraSPARC IIIs kick out ~0.7 tok/s on Gemma 3 270M Q8_0. Haven’t beat the compiler’s scalar code yet as the UltraSPARC’s VIS vector stuff isn’t ideal for int8 dotprod. May find some perf with Q4_0 nibble unpack. Thanks to Georgi Gerganov for a pretty portable codebase and for whomever saved me the trouble of handling big endian with the s390x port lolshow more