正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Hermes Agent Nous Research + Gemma 4 26B Google DeepMind, fully working on local Mac 🤯: one prompt → full CLI app + tests + pytest passing tool calls firing back to back — local Claude Code vibes Gemma 4 MoE is fast AND smart, and tool calling just... show more

raullen

64,103 subscribers

24,988 次观看 • 4 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,760 次观看 • 2 个月前

Google's new Gemma 4 is excellent, and the 26B MoE version is likely the best model to run on a 32GB Framework Desktop. It's fast, smart, and also great for tool calling if you use it with OpenClaw🦞 or other local agent platforms.

Google's new Gemma 4 is excellent, and the 26B MoE version is likely the best model to run on a 32GB Framework Desktop. It's fast, smart, and also great for tool calling if you use it with OpenClaw🦞 or other local agent platforms.

Framework

58,593 次观看 • 5 个月前

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 次观看 • 4 个月前

now doing gemma 4 moe. it'a crazy how usable this is. just a little web search tool (wiggles its head on tool calls), and it can serve as a mostly capable, local voice assistant. tomorrow i shall replace google auto crap with this in my car. just need to give it access to spotify.

now doing gemma 4 moe. it'a crazy how usable this is. just a little web search tool (wiggles its head on tool calls), and it can serve as a mostly capable, local voice assistant. tomorrow i shall replace google auto crap with this in my car. just need to give it access to spotify.

Mario Zechner

27,323 次观看 • 3 个月前

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

Julian Goldie SEO

11,363 次观看 • 1 个月前

The top question I got was does Halo support local LLM running on a Mac. Here's an example of the app using gemma-4-26b-a4b-qat via LM Studio running on a base M5/32GB. It's giving an average of 40 tok/s and in my brief testing is able to execute all tool calls correctly. Very usable with this app.

The top question I got was does Halo support local LLM running on a Mac. Here's an example of the app using gemma-4-26b-a4b-qat via LM Studio running on a base M5/32GB. It's giving an average of 40 tok/s and in my brief testing is able to execute all tool calls correctly. Very usable with this app.

Tanmay

11,682 次观看 • 2 个月前

small local model that falls apart in bloated agents like openclaw just runs like a wild horse in hermes agent. and that's not even my line, someone else called it that, i've just been quietly pointing people at this harness for months because it held up on everything i threw at it, 3b models all the way to one trillion params. watch this happen on my own machine. i pointed hermes agent at a local http endpoint, gemma 4 12b on my 3090 llama.cpp server, and it auto-detected the model and started working immediately. no config wrestling, no broken tool calls, no babysitting the output format, i typed in a url and it just went. the whole clip is exactly that, start to finish, no errors, no retries, butter smooth. and the tool calling, the one thing that quietly breaks most local setups, works here like it's nothing. it's not the model that's flaky, it's the harness around it. hermes agent is the first agent i've run that actually gets that right. one url, one local model on one card, and it runs like a wild horse.

small local model that falls apart in bloated agents like openclaw just runs like a wild horse in hermes agent. and that's not even my line, someone else called it that, i've just been quietly pointing people at this harness for months because it held up on everything i threw at it, 3b models all the way to one trillion params. watch this happen on my own machine. i pointed hermes agent at a local http endpoint, gemma 4 12b on my 3090 llama.cpp server, and it auto-detected the model and started working immediately. no config wrestling, no broken tool calls, no babysitting the output format, i typed in a url and it just went. the whole clip is exactly that, start to finish, no errors, no retries, butter smooth. and the tool calling, the one thing that quietly breaks most local setups, works here like it's nothing. it's not the model that's flaky, it's the harness around it. hermes agent is the first agent i've run that actually gets that right. one url, one local model on one card, and it runs like a wild horse.

Sudo su

27,339 次观看 • 2 个月前

I freaked out when my WiFi router suddenly died. then realized my autonomous Hermes agent is running fully local, nothing stopped. Hermes Agent + Gemma 4 26B A4B QAT MoE, 100% local on my laptop, building my side projects while I scroll my phone zero API calls. zero cost. 100% private. fully offline. This might be the most satisfying thing I’ve watched in a while. last post: showed Hermes + local Gemma 4 26B pull off backtest a trading strategy. this time I asked it to develop something i'd use myself everyday: # A full unpacked extension with: - React side panel UI - Local llama.cpp backend (offline AI) - Live tab sync + status tracking - Auto context extraction via Readability.js Vision on Demand → captures viewport screenshots as compressed JPEGs Deterministic action system -> model outputs tokens -> directly controls page scrolling It planned everything first. Then started executing step by step. all i did was say 'ok'. only once. # What’s wild: - It reports back after every phase - Auto compresses context when nearing limits - Actualy, stays on track llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 # Performance on a single NVIDIA RTX 4060 (8GB VRAM) + 16 GB DDR4 RAM Gaming Laptop: - 300 tokens/sec prefill - 25+ tokens/sec decode More than usable for real dev workflows. This isn’t AI demo territory anymore. This is autonomous local software actually building things.

I freaked out when my WiFi router suddenly died. then realized my autonomous Hermes agent is running fully local, nothing stopped. Hermes Agent + Gemma 4 26B A4B QAT MoE, 100% local on my laptop, building my side projects while I scroll my phone zero API calls. zero cost. 100% private. fully offline. This might be the most satisfying thing I’ve watched in a while. last post: showed Hermes + local Gemma 4 26B pull off backtest a trading strategy. this time I asked it to develop something i'd use myself everyday: # A full unpacked extension with: - React side panel UI - Local llama.cpp backend (offline AI) - Live tab sync + status tracking - Auto context extraction via Readability.js Vision on Demand → captures viewport screenshots as compressed JPEGs Deterministic action system -> model outputs tokens -> directly controls page scrolling It planned everything first. Then started executing step by step. all i did was say 'ok'. only once. # What’s wild: - It reports back after every phase - Auto compresses context when nearing limits - Actualy, stays on track llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 # Performance on a single NVIDIA RTX 4060 (8GB VRAM) + 16 GB DDR4 RAM Gaming Laptop: - 300 tokens/sec prefill - 25+ tokens/sec decode More than usable for real dev workflows. This isn’t AI demo territory anymore. This is autonomous local software actually building things.

Alok

56,428 次观看 • 2 个月前

Want a free private AI agent on your laptop? Here’s the simple setup. 👇 1. Install Hermes Agent from GitHub. 2. Install Ollama. 3. Download Gemma 4 26B. 4. Use Docker to keep everything contained. 5. Add Firecrawl so the agent can research the web. 6. Connect Telegram so you can chat with it from your phone. 7. Turn on context compression so it stays sharp. 8. Set the daily reset for 4 AM so it refreshes while you sleep. What you get at the end: → local AI agent → private workflow → web research built in → self improving skills → no monthly model rental That is a serious setup for anyone building with AI. Save this video, you’ll have the easiest beginner roadmap to follow later. Want the SOP? DM me. 💬

Want a free private AI agent on your laptop? Here’s the simple setup. 👇 1. Install Hermes Agent from GitHub. 2. Install Ollama. 3. Download Gemma 4 26B. 4. Use Docker to keep everything contained. 5. Add Firecrawl so the agent can research the web. 6. Connect Telegram so you can chat with it from your phone. 7. Turn on context compression so it stays sharp. 8. Set the daily reset for 4 AM so it refreshes while you sleep. What you get at the end: → local AI agent → private workflow → web research built in → self improving skills → no monthly model rental That is a serious setup for anyone building with AI. Save this video, you’ll have the easiest beginner roadmap to follow later. Want the SOP? DM me. 💬

Julian Goldie SEO

31,292 次观看 • 4 个月前

i tested a fully local ai coding setup: gemma4 from Google DeepMind running through LM Studio, paired with pi which is a minimal open source coding harness this is part 1 of a two part series. here we cover the why and the complete hands-on setup 0:00 - Running a coding agent fully local 0:47 - Why Pi as the coding agent harness 1:39 - Why Gemma 4 and which variant to pick 2:44 - Architecture: Pi, LM Studio and Gemma 4 3:45 - LM Studio setup and model download 5:01 - Starting the local server 5:19 - API sanity check with curl 6:08 - Installing the Pi coding agent 6:20 - Configuring models.json for local models 8:20 - First run: selecting the model in Pi 8:58 - Fixing the context length error 10:16 - Testing the working setup 10:50 - What is coming in Part 2

i tested a fully local ai coding setup: gemma4 from Google DeepMind running through LM Studio, paired with pi which is a minimal open source coding harness this is part 1 of a two part series. here we cover the why and the complete hands-on setup 0:00 - Running a coding agent fully local 0:47 - Why Pi as the coding agent harness 1:39 - Why Gemma 4 and which variant to pick 2:44 - Architecture: Pi, LM Studio and Gemma 4 3:45 - LM Studio setup and model download 5:01 - Starting the local server 5:19 - API sanity check with curl 6:08 - Installing the Pi coding agent 6:20 - Configuring models.json for local models 8:20 - First run: selecting the model in Pi 8:58 - Fixing the context length error 10:16 - Testing the working setup 10:50 - What is coming in Part 2

Nidhi Singh

16,289 次观看 • 2 个月前

Today we're shipping our biggest MLX-VLM release yet: v0.6.0 ...and we are raising 💸 This one's about turning your Apple devices into real local agent machines. From your desk to your pocket. What's new: ⚡ Speculative decoding everywhere — Gemma 4 EAGLE3 + DFlash, Qwen MTP, DeepSeek V4 MTP. Faster tokens, less waiting. 🤖 Agent-ready server — native Anthropic /v1/messages API, stateful /v1/responses, tool calls, Codex context budgets. Plug Claude Code & Codex straight into local models. 👁️ New models galore — DeepSeek V4, ZAYA1-VL, MiniCPM-V 4.6, LFM2 MoE, Step-3.7 Flash, Laguna + more. 🎨 Image gen & editing — FLUX.2 (base + klein), PrismML Bonsai. 🔊 Audio in — Qwen3 Omni, Gemma 4 audio, base64 chat audio. 🧮 TurboQuant KV cache — RHT-correct fast paths for leaner memory. 📦 Modular server, better metrics, cleaner streaming. Run real agents on the hardware already in your hands. Github:

Today we're shipping our biggest MLX-VLM release yet: v0.6.0 ...and we are raising 💸 This one's about turning your Apple devices into real local agent machines. From your desk to your pocket. What's new: ⚡ Speculative decoding everywhere — Gemma 4 EAGLE3 + DFlash, Qwen MTP, DeepSeek V4 MTP. Faster tokens, less waiting. 🤖 Agent-ready server — native Anthropic /v1/messages API, stateful /v1/responses, tool calls, Codex context budgets. Plug Claude Code & Codex straight into local models. 👁️ New models galore — DeepSeek V4, ZAYA1-VL, MiniCPM-V 4.6, LFM2 MoE, Step-3.7 Flash, Laguna + more. 🎨 Image gen & editing — FLUX.2 (base + klein), PrismML Bonsai. 🔊 Audio in — Qwen3 Omni, Gemma 4 audio, base64 chat audio. 🧮 TurboQuant KV cache — RHT-correct fast paths for leaner memory. 📦 Modular server, better metrics, cleaner streaming. Run real agents on the hardware already in your hands. Github:

Prince Canuma

65,868 次观看 • 3 个月前

HERMES AGENT WITHOUT TOOLS IS A CHATBOT. WITH THEM IT BUILDS 3D TOWERS IN BLENDER, CHECKS STOCK PRICES, AND DRIVES VS CODE. tonbi JUST DROPPED THE FULL GUIDE. module 6 of his 10-part Hermes masterclass. best breakdown of the tool layer anyone has published. what you need to know: TOOLS vs SKILLS vs MCP skills = instructions (markdown, loaded into context) tools = callable functions (Python, agent emits call, Hermes executes) MCP = adapters to external systems (Blender, Stripe, Linear, Notion) every tool has three parts: → the function (does the real work) → the schema (what the model sees to decide when to call it) → the registry (makes the tool exist in the agent) the model never runs the function directly. it emits a structured request (tool name + JSON args). Hermes executes and returns the result. if the tool fails, the error goes back as JSON. the agent recovers instead of crashing. TOOL SETS CONTROL THE SURFACE hermes chat --tool-sets web # only web tools loaded. no files, no terminal. hermes chat --tool-sets safe # read-only: web search, vision, image gen. # no file writes. no terminal. no code execution. mid-session: /tools enable video /tools disable terminal or toggle in the dashboard: hermes dashboard → Skills → Tool Sets MCP SERVERS two transport types: STDIO (local subprocess) or HTTP (remote endpoint). add a server: hermes mcp # or ask: "add the MCP server for Blender" filter tools with include/exclude per server. keep only what you trust. security: → OAuth 2.1 PKCE (no long-lived tokens in config) → package scanning via api.osv. dev before launch → all MCP calls go through approval gates HERMES AS MCP SERVER hermes mcp serve exposes 10 tools via FastMCP. connect VS Code Copilot or Cursor to your running Hermes instance. BUILD YOUR OWN TOOL He built a stock price tool live: → Python function calling Finnhub API → schema with name, description, parameters → registered in tool_sets.py → agent calls it automatically when relevant any repeating API call in your workflow can become a native tool. full Hermes architecture deep-dive in the article 👇

HERMES AGENT WITHOUT TOOLS IS A CHATBOT. WITH THEM IT BUILDS 3D TOWERS IN BLENDER, CHECKS STOCK PRICES, AND DRIVES VS CODE. tonbi JUST DROPPED THE FULL GUIDE. module 6 of his 10-part Hermes masterclass. best breakdown of the tool layer anyone has published. what you need to know: TOOLS vs SKILLS vs MCP skills = instructions (markdown, loaded into context) tools = callable functions (Python, agent emits call, Hermes executes) MCP = adapters to external systems (Blender, Stripe, Linear, Notion) every tool has three parts: → the function (does the real work) → the schema (what the model sees to decide when to call it) → the registry (makes the tool exist in the agent) the model never runs the function directly. it emits a structured request (tool name + JSON args). Hermes executes and returns the result. if the tool fails, the error goes back as JSON. the agent recovers instead of crashing. TOOL SETS CONTROL THE SURFACE hermes chat --tool-sets web # only web tools loaded. no files, no terminal. hermes chat --tool-sets safe # read-only: web search, vision, image gen. # no file writes. no terminal. no code execution. mid-session: /tools enable video /tools disable terminal or toggle in the dashboard: hermes dashboard → Skills → Tool Sets MCP SERVERS two transport types: STDIO (local subprocess) or HTTP (remote endpoint). add a server: hermes mcp # or ask: "add the MCP server for Blender" filter tools with include/exclude per server. keep only what you trust. security: → OAuth 2.1 PKCE (no long-lived tokens in config) → package scanning via api.osv. dev before launch → all MCP calls go through approval gates HERMES AS MCP SERVER hermes mcp serve exposes 10 tools via FastMCP. connect VS Code Copilot or Cursor to your running Hermes instance. BUILD YOUR OWN TOOL He built a stock price tool live: → Python function calling Finnhub API → schema with name, description, parameters → registered in tool_sets.py → agent calls it automatically when relevant any repeating API call in your workflow can become a native tool. full Hermes architecture deep-dive in the article 👇

YanXbt

32,680 次观看 • 2 个月前

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 次观看 • 2 个月前

HERMES AGENT DESKTOP APP IS INSANE. No more CLI. No more Discord hacks. Nous Research dropped the native Desktop App yesterday and the announcement already crossed 5 million views. What changed: → Full GUI installer. Click Install → fully working agent in under 5 minutes. → Clean ChatGPT-style interface with concurrent sessions. → Built-in skills and tool toggles (Claude Code, Kanban, web search, image gen, terminal, etc.). → Cron jobs creation directly in the UI. → Artifacts viewer, memory & context settings, voice mode, browser preview — all native. Running Hermes on your main machine is now actually pleasant and powerful. This removes the biggest friction point the project ever had. The desktop app just made Hermes 10x more approachable for real daily use. Now it’s time to unlock its full potential. The complete system how to turn Desktop Hermes into a real 24/7 operator is in the new article 👇

HERMES AGENT DESKTOP APP IS INSANE. No more CLI. No more Discord hacks. Nous Research dropped the native Desktop App yesterday and the announcement already crossed 5 million views. What changed: → Full GUI installer. Click Install → fully working agent in under 5 minutes. → Clean ChatGPT-style interface with concurrent sessions. → Built-in skills and tool toggles (Claude Code, Kanban, web search, image gen, terminal, etc.). → Cron jobs creation directly in the UI. → Artifacts viewer, memory & context settings, voice mode, browser preview — all native. Running Hermes on your main machine is now actually pleasant and powerful. This removes the biggest friction point the project ever had. The desktop app just made Hermes 10x more approachable for real daily use. Now it’s time to unlock its full potential. The complete system how to turn Desktop Hermes into a real 24/7 operator is in the new article 👇

YanXbt

32,134 次观看 • 2 个月前

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 次观看 • 2 个月前

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

259,993 次观看 • 2 个月前

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 次观看 • 2 个月前

$a regular laptop with just 8GB VRAM can now run a LOCAL AI AGENT directly on your device what you need: 1/ 8 GB VRAM 2/ 16 GB RAM 3/ $600+ laptop method: Kimi K2 + Hermes Agent Dekstop App what agent can do: 1/ read and write files 2/ patch code, open PRs on GitHub 3/ browse the web like a human 4/ work with Notion 5/ run scripts, debug, refactor this isn't 'ask a question, get an answer.' this is an agent that drives a task from start to finish on its own. before: local agent = either the cloud and $$$, or an RTX now: any gaming laptop from the last 3 years p.s. MoE is the key. Kimi K2 activates only a fraction of its params per token - that's why a 1T-param model runs on hardware where a dense 26B would already be choking did you try Hermes Dekstop?$

a regular laptop with just 8GB VRAM can now run a LOCAL AI AGENT directly on your device what you need: 1/ 8 GB VRAM 2/ 16 GB RAM 3/ $600+ laptop method: Kimi K2 + Hermes Agent Dekstop App what agent can do: 1/ read and write files 2/ patch code, open PRs on GitHub 3/ browse the web like a human 4/ work with Notion 5/ run scripts, debug, refactor this isn't 'ask a question, get an answer.' this is an agent that drives a task from start to finish on its own. before: local agent = either the cloud and $$$, or an RTX now: any gaming laptop from the last 3 years p.s. MoE is the key. Kimi K2 activates only a fraction of its params per token - that's why a 1T-param model runs on hardware where a dense 26B would already be choking did you try Hermes Dekstop?

kaize

58,735 次观看 • 2 个月前

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.

Sudo su

30,228 次观看 • 4 个月前

HERMES AGENT VS OPENCLAW. a local ai onboarding flow test. a 3.9gb bonsai served on localhost, both agents upstream and latest, i point each one at the endpoint and watch which one even finds it. > hermes opens a provider menu, thirty plus options, local servers sitting right there next to the cloud ones, i hand it 127.0.0.1:8899, it verifies the endpoint, one model visible, auto-detects the model by name, bonsai-27b-q1_0, reads the context length straight off the server, saves it, and starts reasoning and firing real tool calls on my local model. no key. no friction. > openclaw has no menu. it goes hunting for a codex login, an openai key, finds none because there are none, prints no models available three times, defaults to openai/gpt-5.5, a cloud model it cannot reach, and dead ends on run auth login --provider openai. read that back. it asked me for an openai key. to run a model already running on my own machine. it never once looked at localhost. to be fair, openclaw can run local if you hand wire endpoint yourself. what it will not do is find the model already sitting on your box. hermes agent found it in one line. now the part i owe you. the auto-detect that just won, the model name read, the .gguf strip, the context length probe off the server, that is my code, it is in hermes agent main right now, authorship preserved, #2051 and #4218. the wizard fix that stops an agent from silently routing you to someone else's creds, the exact trap openclaw still falls into, mine too, #4210. i contribute to hermes agent, i told you that going in. one agent is built to talk to whatever you are running, the other is built to talk to a cloud api, so one found my model and ran it and the other asked me to log into openai. onboarding flow of both, mapped, below.

HERMES AGENT VS OPENCLAW. a local ai onboarding flow test. a 3.9gb bonsai served on localhost, both agents upstream and latest, i point each one at the endpoint and watch which one even finds it. > hermes opens a provider menu, thirty plus options, local servers sitting right there next to the cloud ones, i hand it 127.0.0.1:8899, it verifies the endpoint, one model visible, auto-detects the model by name, bonsai-27b-q1_0, reads the context length straight off the server, saves it, and starts reasoning and firing real tool calls on my local model. no key. no friction. > openclaw has no menu. it goes hunting for a codex login, an openai key, finds none because there are none, prints no models available three times, defaults to openai/gpt-5.5, a cloud model it cannot reach, and dead ends on run auth login --provider openai. read that back. it asked me for an openai key. to run a model already running on my own machine. it never once looked at localhost. to be fair, openclaw can run local if you hand wire endpoint yourself. what it will not do is find the model already sitting on your box. hermes agent found it in one line. now the part i owe you. the auto-detect that just won, the model name read, the .gguf strip, the context length probe off the server, that is my code, it is in hermes agent main right now, authorship preserved, #2051 and #4218. the wizard fix that stops an agent from silently routing you to someone else's creds, the exact trap openclaw still falls into, mine too, #4210. i contribute to hermes agent, i told you that going in. one agent is built to talk to whatever you are running, the other is built to talk to a cloud api, so one found my model and ran it and the other asked me to log into openai. onboarding flow of both, mapped, below.

Sudo su

23,816 次观看 • 1 个月前