"Can you form a snowman?" A simple canvas for human + AI co-manipulation. 🔴🟢🔵💬 Made using Gemini 2.0 w/ Kyle Phillips. Video has sound 🔊. Prompt + thoughts in 🧵

It's fun to test how Gemini can translate so many different concepts - color mixing, "scooching," "humungous," snowman - into actions. The feeling of co-manipulating objects live with AI is really neat.

Made using @hapticdata's Multimodal Live API starter code here:

One of the inspirations for this quick test was the really cool 1970 SHRDLU project by Terry Winograd:

Here, Gemini is calling functions within the code I have given. However, Gemini 2.0 can also write the full code itself, and even combine coding with other tool use pole Google Search. (Here's an example of that: It's fun to explore the right balance of constraint vs freedom to give the model.

Seamlessly switch between AI models like ChatGPT and Gemini. Why limit yourself? Explore the power of choice with #AI #Flexibility #AdvancedFeatures

@hapticdata very cool! when people use LLMs like this repeatedly and with very low latencies like it's some kind of free, persistent, almost disposable resource it gives me the "feel the AGI" feels.

@hapticdata Love this @alexanderchen!!! Here's a blend between your constrained 🔴🟢🔵 set and generating new shapes 🔺🟩 It's fun to try things like "give me another pair like this but different colors", and see if it can blend persistence with generation

@hapticdata I've been testing different "agents" with... "Draw a yellow duck with blue wings". 🤕🤕 Sometimes I get an oval filled in. Ha ha. Or they install a drawing app.

@hapticdata very cooool🔥🔥

@hapticdata Amazing! Is the AI writing and executing code in React.js? Or what is it? I didn’t know Gemini was so good at this! I’ve been exclusively using Claude Sonnet (new lol) for code execution. Like this prime harmonic motion analysis that reminded me of the swirling dervishes!

I love that harmonic series example! ❤️ In this example the model is not writing raw code, but it is doing function calling to manipulate properties within my existing @p5xjs sketch because I wanted to constrain it. But Gemini can write raw code, as I am doing in examples like this one: