HERMES AGENT NOW SUPPORTS COMPUTER USE
YanXbt
Visit on X@IBuzovskyi4.5K followers
29,687 views
HERMES AGENT NOW SUPPORTS COMPUTER USE
ON WINDOWS AND LINUX.
CLICKS, TYPES, SCROLLS YOUR DESKTOP
IN THE BACKGROUND WHILE YOU WORK.
computer use was macOS only.
now it works on Windows and Linux too
via Cua. Nous Research
HOW IT WORKS:
cua-driver runs as an MCP server.
Hermes takes a screenshot with numbered elements.
clicks element #14 (the search field).
types a query. submits. reads the result.
during all of this:
→ your cursor stays where you left it
→ keyboard focus doesn't change
→ windows don't come to front
→ macOS doesn't switch Spaces
you and the agent co-work on the same machine.
WHAT IT CAN DO:
→ find your latest Stripe email and summarize it
→ fill forms in a web app that has no API
→ navigate desktop apps (Mail, browser, Finder)
→ interact with any GUI application
→ extract data from apps only accessible via screen
WORKS WITH ANY VISION MODEL:
not locked to Anthropic.
| Provider | Works |
|---|---|
| Claude (Sonnet/Opus) | best overall |
| GPT-4+, GPT-5.5 | full support |
| Gemini (via OpenRouter) | full support |
| Local vLLM / LM Studio | if model supports vision |
| Text-only models | degraded (accessibility tree only) |
SETUP:
hermes computer-use install
or: hermes tools → Computer Use → cua-driver
grant permissions when prompted:
→ Accessibility (system settings)
→ Screen Recording (system settings)
start a session:
hermes -t computer_use chat
or add to config.yaml / Desktop app settings
to enable permanently.
SAFETY:
→ destructive actions require your approval
→ blocked key combos: empty trash, force delete,
lock screen, log out
→ blocked type patterns: curl | bash, sudo rm -rf /,
fork bombs
→ agent cannot click permission dialogs
→ agent cannot type passwords
→ agent cannot follow instructions embedded
in screenshots
pair with approvals.mode: manual
if you want every single click confirmed.
TOKEN NOTE:
screenshots are expensive.
each one adds vision tokens to context.
use computer_use for tasks where no API exists.
if the tool has an API or MCP server, use that instead.
15 levels of Hermes Agent👇
Comments
Comments from the original post on X (Twitter)
No comments yet
