I put Gemini 2.0 inside a desktop app and gave it access to my screen, terminal, clipboard, and the internet:

The coolest part is the live error handling. It knows that the curl returned an error instantly because it’s looking at the same screen I am!

You can now try it here!

AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.

Nice you can try Gemini 2.0 and much more here as well:

It's fucking cool what tools are you using to give access of computer or the browser ?

Nice! I noticed you're using the Gemini API and sending frames yourself ... Curious if you've tried the new Multimodal Live API for this instead? I guess your approach gives you more control?

the latency to the responses is very good. is screen access running on a loop in the background or is it voice activated?

I’m sending a frame of my screen to the API every 0.5s

It correctly identified getting an API key as the most difficult problem in programming for both humans and robots

what the price of 2.0s real time API? hella expensive?

you can try it for free!