Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

I've been having a ton of fun exploring the Realtime API from OpenAI. Today they just launched the ability to connect to it over WebRTC! 🤖🖐️ It supports Tool Calling, so check out what that looks like in Realtime function calling in any browser based app. I built a... show more

Craig Dennis

7,414 subscribers

60,122 views • 1 year ago •via X (Twitter)

Education Gaming Science & Technology

Anya Rossi• Live Now

Private livecam show

11 Comments

Craig Dennis1 year ago

Full 📼

mobilewatcher21 year ago

Cloudless remote access to the app | With this app you are in control. Pay for the app and get Detect Events and Smoke Alarm, Record Video. Text and email will increase awareness. Access remotely using QR code on MobileWatcher2 ™ Main screen top.

Josh Larson1 year ago

@OpenAI I went from "wow this is neat" to "gosh I should build something with this!" to "I MUST buy a robot hand"

Craig Dennis1 year ago

@OpenAI Put it on your holiday list! Link in the repo!

Alex Volkov (Thursd/AI)1 year ago

@OpenAI This is SO dope! 🔥

Brian Ball1 year ago

@OpenAI "It's kinda hard to read.. can you fix that?"

Craig Dennis1 year ago

@OpenAI Bonkers right?!!!

William Allen1 year ago

@OpenAI 🤖🤯!!

Ray Fernando1 year ago

@OpenAI Wow!! This is so cool!!

jensen1 year ago

@OpenAI This is great. Made a relay using DO a few months ago, but WebRTC makes way more sense.

Ryan Morrison 🇺🇸 🇿🇦1 year ago

@OpenAI Very cool

Related Videos

Who knew that inviting AI into your video chats could be fun && functional? We used Cloudflare Developers Calls and the OpenAI Realtime API with WebRTC to add AI to our open source Orange Meets demo app. I forked it to make my own murder mystery app 😈 Code and details in 🧵

Who knew that inviting AI into your video chats could be fun && functional? We used Cloudflare Developers Calls and the OpenAI Realtime API with WebRTC to add AI to our open source Orange Meets demo app. I forked it to make my own murder mystery app 😈 Code and details in 🧵

Ricky

16,796 views • 1 year ago

At OpenAI DevDay, we introduced the Realtime API with a demo of an interactive solar system you can navigate with your voice. A lot of you have asked how it was built, so we've open-sourced it—with a twist: this version uses our new WebRTC integration. Have fun exploring! 🌍

At OpenAI DevDay, we introduced the Realtime API with a demo of an interactive solar system you can navigate with your voice. A lot of you have asked how it was built, so we've open-sourced it—with a twist: this version uses our new WebRTC integration. Have fun exploring! 🌍

Katia Gil Guzman

316,561 views • 1 year ago

This is crazy, I can talk to OpenAI's realtime API, and it'll respond back with voice, and then help call Codex to help me code! Got curious about Codex and its realtime integration, and found codex's feature flag `realtime_conversation` integrated for the codex app-server. How it works: - launch codex app-server with realtime - connect a client to realtime through WebRTC voice connection - Codex app-server is also attached on the side - talk to realtime - realtime decides whether coding is needed, and responds back with events - Codex app-server takes these events and runs codex agent turns - codex writes code like your codex cli exp Keep in mind OpenAI hasn't released this officially, they're still actively working on it, I just created a fork. This is a completely new UX than the usual chat interface for agentic coding. For example, when the agent responds back in voice, it's actually faster for me to read the response, and I don't actually find voice response that useful unless it does so in the background. But image realtime working in the background, you can talk to it, realtime keeps track of the different agents working on your tasks and your status. More to come on demo side, i'll make a more polished version and include voice, but thought this was cool to share. You can see in the responses the distinction between my voice input, realtime response, and realtime calling Codex to do work!

This is crazy, I can talk to OpenAI's realtime API, and it'll respond back with voice, and then help call Codex to help me code! Got curious about Codex and its realtime integration, and found codex's feature flag `realtime_conversation` integrated for the codex app-server. How it works: - launch codex app-server with realtime - connect a client to realtime through WebRTC voice connection - Codex app-server is also attached on the side - talk to realtime - realtime decides whether coding is needed, and responds back with events - Codex app-server takes these events and runs codex agent turns - codex writes code like your codex cli exp Keep in mind OpenAI hasn't released this officially, they're still actively working on it, I just created a fork. This is a completely new UX than the usual chat interface for agentic coding. For example, when the agent responds back in voice, it's actually faster for me to read the response, and I don't actually find voice response that useful unless it does so in the background. But image realtime working in the background, you can talk to it, realtime keeps track of the different agents working on your tasks and your status. More to come on demo side, i'll make a more polished version and include voice, but thought this was cool to share. You can see in the responses the distinction between my voice input, realtime response, and realtime calling Codex to do work!

Matthew Lam

17,674 views • 2 months ago

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!

Zara Zhang

45,125 views • 2 months ago

Wow, Anthropic Claude Code is AWESOME I built a Cloudflare Developers Workers AI-powered app in just a few minutes, letting Claude do 99% of the work Check it out ⬇️ 🧵with a few other resources too

Wow, Anthropic Claude Code is AWESOME I built a Cloudflare Developers Workers AI-powered app in just a few minutes, letting Claude do 99% of the work Check it out ⬇️ 🧵with a few other resources too

Kristian

34,661 views • 1 year ago

OpenAI shipped a new speech-to-speech model today: gpt-realtime-2 This is the first speech-to-speech model good enough to use in my voice agents that do "real work." Or real play, for that matter. Here's gpt-realtime-2 as the brain of the ship AI in Gradient Bang. The voice-to-voice response and tool calling times here are unedited, so you can see exactly what the interaction with the model is like in an agent with a very complex system instruction and frequent tool calls. (I did clip out the subagent task execution segments, after gpt-realtime-2 starts a subagent via a tool call. Subagents in this config used gpt-5.2 "medium" effort.)

OpenAI shipped a new speech-to-speech model today: gpt-realtime-2 This is the first speech-to-speech model good enough to use in my voice agents that do "real work." Or real play, for that matter. Here's gpt-realtime-2 as the brain of the ship AI in Gradient Bang. The voice-to-voice response and tool calling times here are unedited, so you can see exactly what the interaction with the model is like in an agent with a very complex system instruction and frequent tool calls. (I did clip out the subagent task execution segments, after gpt-realtime-2 starts a subagent via a tool call. Subagents in this config used gpt-5.2 "medium" effort.)

kwindla

55,222 views • 3 months ago

Voice is the OG modality. So excited for image inputs, function calling & MCP support in the Realtime API GA! `gpt-realtime` is a lot more natural and expressive, and every time a SOTA voice model is released, you know what I gotta do... Here is the new voice Marin, on KaraokeBench! scores 3 out of 7 again from me

Voice is the OG modality. So excited for image inputs, function calling & MCP support in the Realtime API GA! `gpt-realtime` is a lot more natural and expressive, and every time a SOTA voice model is released, you know what I gotta do... Here is the new voice Marin, on KaraokeBench! scores 3 out of 7 again from me

swyx

51,219 views • 11 months ago

another realtime api trick i stumbled upon 💡 to prevent the model from "jumping in" to speak whenever you pause, you can give it a stay_silent() function that does nothing it'll call it when it knows you haven't finished your thought, even if you pause - full prompt below👇 -

another realtime api trick i stumbled upon 💡 to prevent the model from "jumping in" to speak whenever you pause, you can give it a stay_silent() function that does nothing it'll call it when it knows you haven't finished your thought, even if you pause - full prompt below👇 -

ilan bigio

74,247 views • 1 year ago

I love calling out these accounts from overseas that have zero clue what it takes to purchase a firearm here in the United States. In fact, purchasing a firearm is one of the most strict things that you can do here. So let’s make it as easy to purchase a firearm as it is to vote.

I love calling out these accounts from overseas that have zero clue what it takes to purchase a firearm here in the United States. In fact, purchasing a firearm is one of the most strict things that you can do here. So let’s make it as easy to purchase a firearm as it is to vote.

Steve 🇺🇸

23,181 views • 1 month ago

Last week we released OpenAI Developers gpt-realtime our latest speech-to-speech model. Prompting S2S models can be extremely powerful. To get the most out of the model we compiled a series of prompting tips of what we saw work with early customers. To present them here's Cedar our new voice in gpt-realtime walk you through it. If you want to read it yourself check the links below 👇

Last week we released OpenAI Developers gpt-realtime our latest speech-to-speech model. Prompting S2S models can be extremely powerful. To get the most out of the model we compiled a series of prompting tips of what we saw work with early customers. To present them here's Cedar our new voice in gpt-realtime walk you through it. If you want to read it yourself check the links below 👇

dominik kundel

66,904 views • 11 months ago

Voice is an interface, and I feel we’re going to see voice explode. OpenAI gpt-realtime is out and allows for native speech-to-speech and precise tool calling in real time. It can also take image inputs and place SIP phone calls - good for a quick triage before human handoff.

Voice is an interface, and I feel we’re going to see voice explode. OpenAI gpt-realtime is out and allows for native speech-to-speech and precise tool calling in real time. It can also take image inputs and place SIP phone calls - good for a quick triage before human handoff.

Allie K. Miller

80,345 views • 11 months ago

OPENAI 🔥: The built-in web browser in the ChatGPT app is becoming more mature. Now it supports URL suggestions during typing. Besides that, the ChatGPT Chrome extension can now reference open tabs, highlight to ask, and more! I may need to try it as a default 👀

OPENAI 🔥: The built-in web browser in the ChatGPT app is becoming more mature. Now it supports URL suggestions during typing. Besides that, the ChatGPT Chrome extension can now reference open tabs, highlight to ask, and more! I may need to try it as a default 👀

🚨 AI News | TestingCatalog

62,691 views • 6 days ago

we've been disappointed at how the input into most apps is still just mouse and keyboard. So we built a tool that lets you write directly into any app - check it out

we've been disappointed at how the input into most apps is still just mouse and keyboard. So we built a tool that lets you write directly into any app - check it out

WillScuderi

3,741,838 views • 2 years ago

i hate the way tool calling works today, so i built a fix for it its called `executor` and it lets your agent run code to call any API / MCP / GraphQL server rather than installing CLIs this is a soft launch, you can try it with npm i -g executor but im still making it great

i hate the way tool calling works today, so i built a fix for it its called `executor` and it lets your agent run code to call any API / MCP / GraphQL server rather than installing CLIs this is a soft launch, you can try it with npm i -g executor but im still making it great

Rhys

129,065 views • 5 months ago

you don’t need a robot to see physical ai in action, you just need a browser. this week I finally tried something I've wanted to do for a while: design an SO-101 scene in three.js, connect it to a real leader arm, collect teleoperation data, train an ACT model, then port it to run in the browser. the result is the first ever demo of ACT that runs locally on the web. try it yourself now at

you don’t need a robot to see physical ai in action, you just need a browser. this week I finally tried something I've wanted to do for a while: design an SO-101 scene in three.js, connect it to a real leader arm, collect teleoperation data, train an ACT model, then port it to run in the browser. the result is the first ever demo of ACT that runs locally on the web. try it yourself now at

Binh Pham

120,916 views • 1 month ago

we've been disappointed at how the input into most apps is still just mouse and keyboard. So we built a tool that lets you write directly into any app - check it out

we've been disappointed at how the input into most apps is still just mouse and keyboard. So we built a tool that lets you write directly into any app - check it out

WillScuderi

5,826,968 views • 2 years ago

we've been disappointed at how the input into most apps is still just mouse and keyboard. So we built a tool that lets you write directly into any app - check it out

we've been disappointed at how the input into most apps is still just mouse and keyboard. So we built a tool that lets you write directly into any app - check it out

WillScuderi

3,069,494 views • 2 years ago