Francesco

Imagine if language models could tap into the app ecosystem of your iPhone. Would the need for plugins and assistants become obsolete if we simply allowed a model to orchestrate our existing (and many years robust) user interfaces? This demonstrates the extent to which GPT-4V excels as a Generalist Mobile AI Agent – without any fine-tuning or grounding, and merely by integrating with a text model that has JSON mode enabled. I suggest watching this demo for a (maybe) wow factor and the results on iOS 17 using NavAIGuide, a mobile and web navigational agent framework for LLMs: