Can GPT-4-Vision Play Super Mario 64? I created 'Multimodal Gamer,' a framework enabling multi-modal models (combining text and visual inputs) to play games. Check out my video overview below and let me know your thoughts!

Repository:

YouTube if you prefer to watch it there:

You can pause the game for when the model is thinking so latency isn't a issue. I coded my own version a few weeks ago for Legends of Zelda but the issue was the Model had trouble "Seeing" the enemies relative to itself (and sight issues in general) and GPT-4 just walk into the enemies.

very cool, I compare it to my own (once token price come down). About what I expected. What frequency were you calling the api at? Wish we could "react" faster (faster api response), and

@chillzaza_ so cool!

Very cool!

🔥💎🤌

This is really cool. I’m gonna try this with pokemon haha

I know this would increase latency, but do you think passing a small screenshot every so often would significantly help the model's understanding of the play space?

awesome