Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

At OpenAI DevDay, we introduced the Realtime API with a demo of an interactive solar system you can navigate with your voice. A lot of you have asked how it was built, so we've open-sourced it—with a twist: this version uses our new WebRTC integration. Have fun exploring! 🌍

Katia Gil Guzman

6,636 subscribers

316,548 views • 1 year ago •via X (Twitter)

Education Science & Technology News & Politics

Anya Rossi• Live Now

Private livecam show

11 Comments

Katia Gil Guzman1 year ago

With the new WebRTC integration, you can connect to the Realtime API directly from the browser:

Katia Gil Guzman1 year ago

Shout out to @karoliskosas for the initial idea (I have to admit, it sounded a bit crazy when I first heard it) and to the @splinetool team for building an amazing tool and helping us optimize the scene!

Katia Gil Guzman1 year ago

@splinetool To see the full replay:

AssemblyAI1 year ago

Announcing: Our most advanced speech-to-text model goes beyond accuracy to capture the real-world complexity of human conversation and deliver reliable, source-of-truth audio data. Explore Universal-2 updates 👇

Nicolas Camara1 year ago

This is awesome! Thanks for open sourcing it!

ZAZO1 year ago

It’s super good

Drago Nicolas Bonacich1 year ago

This is super cool! Thanks!

oAI1 year ago

@OpenAIDevs that's what's missing in the AVM

Mbongeni Ndlovu1 year ago

Appreciate it

John Schoenith1 year ago

Look at this shiz! Open-source at OpenAI! 🪩

🐧 lalo adrian morales 𝕏1 year ago

well that was easy ! thanks!

Related Videos

1. did you know cloudflare offers a webrtc service? 2. did you know there's a whole zone clone built 100% on cloudflare? (link 👇) 3. did you know you can use it with the @openai realtime api to build multi-user ai experiencer (blog 👇) 4. so i now have an AI spanish tutor 🤪

1. did you know cloudflare offers a webrtc service? 2. did you know there's a whole zone clone built 100% on cloudflare? (link 👇) 3. did you know you can use it with the @openai realtime api to build multi-user ai experiencer (blog 👇) 4. so i now have an AI spanish tutor 🤪

rita kozlov 🐀

39,208 views • 1 year ago

I've been having a ton of fun exploring the Realtime API from OpenAI. Today they just launched the ability to connect to it over WebRTC! 🤖🖐️ It supports Tool Calling, so check out what that looks like in Realtime function calling in any browser based app. I built a template that allows for relaying through a Cloudflare Developers Worker so you can do it too! Full 📼 in 🧵

I've been having a ton of fun exploring the Realtime API from OpenAI. Today they just launched the ability to connect to it over WebRTC! 🤖🖐️ It supports Tool Calling, so check out what that looks like in Realtime function calling in any browser based app. I built a template that allows for relaying through a Cloudflare Developers Worker so you can do it too! Full 📼 in 🧵

Craig Dennis

60,091 views • 1 year ago

hey everyone! super excited play a small role in the OpenAI Realtime API launch today. we have launched an open source repo, the Realtime Console -- a React app you can hack apart and start building with immediately. here's a demo! thanks Peter Bakkum Mark Hudnall Jordan Sitkin

hey everyone! super excited play a small role in the OpenAI Realtime API launch today. we have launched an open source repo, the Realtime Console -- a React app you can hack apart and start building with immediately. here's a demo! thanks Peter Bakkum Mark Hudnall Jordan Sitkin

keith

243,182 views • 1 year ago

.@openAI just introduced their gpt-realtime model -- pretty cool demo We've been building a realtime teaching capability at Sage, have a look. There's nothing else like it. (recommend watching at 2x)

.@openAI just introduced their gpt-realtime model -- pretty cool demo We've been building a realtime teaching capability at Sage, have a look. There's nothing else like it. (recommend watching at 2x)

Joe Norman

13,441 views • 10 months ago

We just updated Junie and the Codex integration in JetBrains AI! Now you can use OpenAI GPT 5.5 in all JetBrains IDEs. We ran various tests with this model but have a look yourself at one of the demo projects we built (🔊 on)

We just updated Junie and the Codex integration in JetBrains AI! Now you can use OpenAI GPT 5.5 in all JetBrains IDEs. We ran various tests with this model but have a look yourself at one of the demo projects we built (🔊 on)

JetBrains

10,966 views • 2 months ago

We've been having a lot of fun with the newest Mega Kangaskhan ex card that was introduced in Fantastical Parade. What new card-inspired decks have you cooked up this set? Learn more:

We've been having a lot of fun with the newest Mega Kangaskhan ex card that was introduced in Fantastical Parade. What new card-inspired decks have you cooked up this set? Learn more:

Pokémon TCG Pocket

120,899 views • 4 months ago

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!

Zara Zhang

44,631 views • 1 month ago

Sierra agents can be published on your web site, integrated with your mobile app, answer the phone - and now they can also be published to ChatGPT so you can directly reach hundreds of millions of consumers with your agent. Since OpenAI announced ChatGPT apps two weeks ago, we've heard from our customers that they want two things: a direct customer relationship and the ability to reach customers where they already are. We don't think you should have to choose. With Sierra, you can build once and run everywhere. Here's a demo of an agent I built to show how it works. And our blog post is here:

Sierra agents can be published on your web site, integrated with your mobile app, answer the phone - and now they can also be published to ChatGPT so you can directly reach hundreds of millions of consumers with your agent. Since OpenAI announced ChatGPT apps two weeks ago, we've heard from our customers that they want two things: a direct customer relationship and the ability to reach customers where they already are. We don't think you should have to choose. With Sierra, you can build once and run everywhere. Here's a demo of an agent I built to show how it works. And our blog post is here:

Bret Taylor

55,488 views • 8 months ago

That’s it. That’s the tweet. The Realtime API now supports WebRTC—you can add Realtime capabilities with just a handful of lines of code. We’ve also cut prices by 60%, added GPT-4o mini (10x cheaper than previous prices), improved voice quality, and made inputs more reliable.

That’s it. That’s the tweet. The Realtime API now supports WebRTC—you can add Realtime capabilities with just a handful of lines of code. We’ve also cut prices by 60%, added GPT-4o mini (10x cheaper than previous prices), improved voice quality, and made inputs more reliable.

OpenAI Developers

523,494 views • 1 year ago

Say hello to “X to Voice”. An open-source app that generates a unique voice from your X/Twitter profile. Built with our Voice Design API which is live from today — where you can create a unique voice from any prompt. Try it with your profile here:

ElevenLabs

833,464 views • 1 year ago

Who knew that inviting AI into your video chats could be fun && functional? We used Cloudflare Developers Calls and the OpenAI Realtime API with WebRTC to add AI to our open source Orange Meets demo app. I forked it to make my own murder mystery app 😈 Code and details in 🧵

Who knew that inviting AI into your video chats could be fun && functional? We used Cloudflare Developers Calls and the OpenAI Realtime API with WebRTC to add AI to our open source Orange Meets demo app. I forked it to make my own murder mystery app 😈 Code and details in 🧵

Ricky

16,796 views • 1 year ago

You can fold a moving bat out of just one sheet of paper! Have fun with it with your little one!

You can fold a moving bat out of just one sheet of paper! Have fun with it with your little one!

Craft Gallery

36,761 views • 4 months ago

🐰 i worked a lot with El Guincho. I got to listen to a lot of tracks and i was really wowed by just how fresh and new this was. One of the first thoughts i had was ‘wow, you can approach a song like this?’ and i felt a lot of pressure about how to make the best use of it. so that must be why i wanted to stay close, it was a totally a different vibe, a whole new vibe. i hope to work again and have another session with the producers and songwriters that we worked with

🐰 i worked a lot with El Guincho. I got to listen to a lot of tracks and i was really wowed by just how fresh and new this was. One of the first thoughts i had was ‘wow, you can approach a song like this?’ and i felt a lot of pressure about how to make the best use of it. so that must be why i wanted to stay close, it was a totally a different vibe, a whole new vibe. i hope to work again and have another session with the producers and songwriters that we worked with

jk.97

26,798 views • 3 months ago

Screaming Frog's OpenAI integration allows you to automate tons of basic SEO tasks. This step by step tutorial to shows you how to set up ChatGPT crawling. As I posted yesterday, Screaming Frog last update was one of the biggest we've seen in a long time. They've introduced a suite of new features including the ability to upload custom JavaScript and having Screaming Frog execute it. This includes a direct integration with OpenAI. To set up OpenAI crawling, here are the steps to follow:

Screaming Frog's OpenAI integration allows you to automate tons of basic SEO tasks. This step by step tutorial to shows you how to set up ChatGPT crawling. As I posted yesterday, Screaming Frog last update was one of the biggest we've seen in a long time. They've introduced a suite of new features including the ability to upload custom JavaScript and having Screaming Frog execute it. This includes a direct integration with OpenAI. To set up OpenAI crawling, here are the steps to follow:

Chris Long

30,710 views • 2 years ago

The new Realtime API with web crawling is mind-blowing! Talk in realtime with any website. Powered by the OpenAI Realtime API and 🔥 Check it out:

The new Realtime API with web crawling is mind-blowing! Talk in realtime with any website. Powered by the OpenAI Realtime API and 🔥 Check it out:

Nicolas Camara

347,089 views • 1 year ago

Trump: We could have a lot of fun with these guys. You can imagine when they win, how they celebrate. We don't want to talk about it. We don't want to talk about it.

Trump: We could have a lot of fun with these guys. You can imagine when they win, how they celebrate. We don't want to talk about it. We don't want to talk about it.

Acyn

30,346 views • 4 months ago

NEW SONG COVER: "Open Arms" from Epic: The Musical 🧦 As you can probably see... we had a lot of fun with this one.

NEW SONG COVER: "Open Arms" from Epic: The Musical 🧦 As you can probably see... we had a lot of fun with this one.

Thomas Sanders

19,250 views • 1 month ago

We've partnered with OpenAI as one of their early research preview partners for the new computer use model. With this model, AI can control a browser better than ever before 👀 Of course, we had to launch a free hosted version + open source implementation. Check it out!

We've partnered with OpenAI as one of their early research preview partners for the new computer use model. With this model, AI can control a browser better than ever before 👀 Of course, we had to launch a free hosted version + open source implementation. Check it out!

Browserbase

40,232 views • 1 year ago