Introducing ambientGPT: an open-source and multimodal MacOS foundation model GUI Run GPT-4o and open-source models with full ambient knowledge of your screen. Foundation models have long been confined to the browser. With ambientGPT, your screen context is directly inferred as part of the query, ensuring you never need to... explicitly upload context again! Github:show more

Unlike OpenAI’s desktop app where you must provide a screenshot or upload a file, the context from your screen is automatically parsed. We also provide the ability to run secure local models like Gemma and Phi-3 multimodal from our interface. Due to the local model sizes, at least 16 GB RAM would be preferred. This was possible via the apple MLX library - shoutout to @awnihannun, @reach_vb

ambientGPT is open-source and we plan to integrate vllm and ollama to provide more extensive inference hosting abilities with our multimodal GUI. We also aim to release ambientGPT on the apple app store soon.

thanks to @mihiranan (mr. clutch) for his help with the demo once again!

Very cool but chatgpt mac app will have the ability to see everything on screen when they release the update ..

so what does the “full ambient knowledge of your screen” entail privacy wise?? why would someone trust this if unwanted data is being captured forever and sent to openai?

Great work! Does it work across multiple monitors? Could be a game changer if so

was waiting for something exactly like this!! what’s the token usage look like?

looks awesome, would love to showcase ambient on

Looks super useful . How does it determine context if you have multiple screens?

This is cool. I built a tool for taking to model via voice