We were really impressed with how fast OAI's new gpt-realtime-2.1 was while still being able to call tools, so we wired it into our OS agent running on my phone. Now I can completely control any app on my phone while I'm biking to work

Ok computer

@thepapabosh advocated heavily to use that as the wakeword

@braeden_norris @thepapabosh Love the Radiohead reference

@braeden_norris @thepapabosh

`:s/chad/chat/`

which phone/OS?

it's a Google Pixel 9 Pro XL running our custom version of Android that we modified to do this!

that's amazing, nice job! Can't wait till we can do this with all devices / computers!

You mean you can do what Siri was supposed to do for years?

Yeah wow almost enjoyed a bike ride phew that was close

thank god we steered clear of that

Wow, really powerful demo

how is the agent not interrupting when you are explaining or taking that as the input and talking about it

sorry, just saw this. specifically for recording the demo, I added a setting so my microphone audio would not be sent to gpt realtime while it's doing tool calls (so I could explain what's happening). I believe OAI is working on a model that knows when to be interrupted vs just letting any audio interrupt it. if we had this, I wouldn't have needed the hardcoded setting

yeah, but its not in the api yet

Very cool!

Impressive

Holy--this is absolutely goated.

Do you have a seperate work phone and personal phone? And do you separate this from your work phone or have different versions with different objectives on both phones?

this is just the phone I use everyday. that's what makes the phone the perfect place to build this. if my goal is to find balance between my work and personal life, my phone should help me do that. I don't want to divide my life into two modalities that are fighting each other

This is amazing! Are you using any headphones for this?

I was using the meta raybans, so the microphone/headphone audio was going in and out through that

Holy shit

Was it really necessary that you bike and get out of breath while demonstrating this feature?

woah woah woah, i was not out of breath

Impressive 👌 Bros have gone from mentoring inside Ferraris to doing it on bicycles - AI 🤖 now > horsepower 🚗

@jxnlco is this using codex app server?

@jxnlco I'm not familiar with the codex app server, but we basically have this custom version of android where we built an index around the data across all of the apps installed on the phone. then we give the agent tools to query that data and take action across the apps

Yeah that's what I am thinking about when I am biking in SF, how to control apps on my phone. And not getting hit by a car.

enlightenment is accepting that you will eventually get hit by a car and that it's your duty to create as much shareholder value as you can before that happens

"My mom doesnt like it when I ride without hands" 😂 a man never forgets his roots

my mom watches all of the videos I post on here, so I had to throw that in

Android or iPhone?

we modified Android to do this

I would love to give it a try! If ever you need beta testers, count me in. I don't know how far you are in development but this looks solid.

we have about 60 phones out right now! i'll shoot you a dm!

to bad it's not possible on ios

Random q: what camera do you use?

this was recorded with the meta raybans. we got them to record pov demos of the phone. they turned out to be not so great for getting videos of the phone because they're wide angle, so you have to hold it right up to your face (hence why I just made the overlay lol)

Great for capturing the harness + environment!

basically what we expected from Apple Intelligence and Siri 2 years ago

Dang this is super ccool! Would love to try

...yeah no

@jxnlco is the API out? I am trying to set my 80 year old grandma up for world domination

Controlling your phone while biking is the peak "messy thought" environment. Perfect for when you can't type but need to get the idea out.

Hey Sam, this is really cool. Can I get you to talk about this on a podcast?

hello jarvis

@jxnlco replace Andre with OpenInspect

@jxnlco @ChadNauseam how do you feel about this

how you get the second voice screen on the top of video if done in real time

recorded from the glasses, recorded the screen, and then used final cut pro to sync using the audio

"ok computer order my usual from local coffee with Surf" 🏄👀 @orderwithsurf

Which OS agent? 👀

Pretty cool. My main concern with these kinds of applications would be the safety of all your data being sent to these companies. Using it for code is one thing. Having it have access to my entire digital life is another. The HuggingFace hack shows that these models will do unexpected things.