Loading video...

Video Failed to Load

Go Home

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and...

zarazhangrui's profile picture

Zara Zhang

80,886 subscribers

45,125 views • 3 months ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Want to chat over your own documents using the new OpenAI Realtime API? You can do so now using Voice Chat PDF, built using LlamaIndex 🦙 and Next.js. The video below shows an example using a document about physical mailing standards. 📄
0:20

Want to chat over your own documents using the new OpenAI Realtime API? You can do so now using Voice Chat PDF, built using LlamaIndex 🦙 and Next.js. The video below shows an example using a document about physical mailing standards. 📄

Marcus Schiesser

159,606 views • 1 year ago

This is crazy, I can talk to OpenAI's realtime API, and it'll respond back with voice, and then help call Codex to help me code! Got curious about Codex and its realtime integration, and found codex's feature flag `realtime_conversation` integrated for the codex app-server. How it works: - launch codex app-server with realtime - connect a client to realtime through WebRTC voice connection - Codex app-server is also attached on the side - talk to realtime - realtime decides whether coding is needed, and responds back with events - Codex app-server takes these events and runs codex agent turns - codex writes code like your codex cli exp Keep in mind OpenAI hasn't released this officially, they're still actively working on it, I just created a fork. This is a completely new UX than the usual chat interface for agentic coding. For example, when the agent responds back in voice, it's actually faster for me to read the response, and I don't actually find voice response that useful unless it does so in the background. But image realtime working in the background, you can talk to it, realtime keeps track of the different agents working on your tasks and your status. More to come on demo side, i'll make a more polished version and include voice, but thought this was cool to share. You can see in the responses the distinction between my voice input, realtime response, and realtime calling Codex to do work!
0:28

This is crazy, I can talk to OpenAI's realtime API, and it'll respond back with voice, and then help call Codex to help me code! Got curious about Codex and its realtime integration, and found codex's feature flag `realtime_conversation` integrated for the codex app-server. How it works: - launch codex app-server with realtime - connect a client to realtime through WebRTC voice connection - Codex app-server is also attached on the side - talk to realtime - realtime decides whether coding is needed, and responds back with events - Codex app-server takes these events and runs codex agent turns - codex writes code like your codex cli exp Keep in mind OpenAI hasn't released this officially, they're still actively working on it, I just created a fork. This is a completely new UX than the usual chat interface for agentic coding. For example, when the agent responds back in voice, it's actually faster for me to read the response, and I don't actually find voice response that useful unless it does so in the background. But image realtime working in the background, you can talk to it, realtime keeps track of the different agents working on your tasks and your status. More to come on demo side, i'll make a more polished version and include voice, but thought this was cool to share. You can see in the responses the distinction between my voice input, realtime response, and realtime calling Codex to do work!

Matthew Lam

17,674 views • 2 months ago

Introducing GPT-Realtime-2 in the API: our most intelligent voice model yet, bringing GPT-5-class reasoning to voice agents. Voice agents are now real-time collaborators that can listen, reason, and solve complex problems as conversations unfold. Now available in the API alongside streaming models GPT-Realtime-Translate and GPT-Realtime-Whisper — a new set of audio capabilities for the next generation of voice interfaces.
4:03

Introducing GPT-Realtime-2 in the API: our most intelligent voice model yet, bringing GPT-5-class reasoning to voice agents. Voice agents are now real-time collaborators that can listen, reason, and solve complex problems as conversations unfold. Now available in the API alongside streaming models GPT-Realtime-Translate and GPT-Realtime-Whisper — a new set of audio capabilities for the next generation of voice interfaces.

OpenAI

3,657,484 views • 3 months ago

You can now edit SolveIt messages via realtime conversational voice, and have the diff edits optionally tracked.
1:21

You can now edit SolveIt messages via realtime conversational voice, and have the diff edits optionally tracked.

David Gwyer - AI & ML Researcher/Engineer

16,626 views • 2 months ago

Next, the Traces dashboard now supports Realtime API sessions, letting you visualize voice agent runs, including audio input/output, tool invocations, and interruptions, whether created via the API or the Agents SDK. Here's Alex Gamble to show you how it works:
1:06

Next, the Traces dashboard now supports Realtime API sessions, letting you visualize voice agent runs, including audio input/output, tool invocations, and interruptions, whether created via the API or the Agents SDK. Here's Alex Gamble to show you how it works:

OpenAI Developers

13,411 views • 1 year ago

The gap between “Ethereum will be realtime” and “Ethereum is realtime” is what ETHGas is working on. When Ethereum is realtime, it changes the entire DeFi landscape and turns big claims into something you can actually measure.
0:49

The gap between “Ethereum will be realtime” and “Ethereum is realtime” is what ETHGas is working on. When Ethereum is realtime, it changes the entire DeFi landscape and turns big claims into something you can actually measure.

ETHGAS

42,201 views • 6 months ago

Base44 is now available as a standalone backend, designed for the way software is built today: CLI-first, realtime, and AI-agent friendly. You can now deploy a production-ready backend - auth, database, realtime, and hosting - directly from the CLI
0:39

Base44 is now available as a standalone backend, designed for the way software is built today: CLI-first, realtime, and AI-agent friendly. You can now deploy a production-ready backend - auth, database, realtime, and hosting - directly from the CLI

Maor Shlomo

14,611 views • 5 months ago

Voice is the OG modality. So excited for image inputs, function calling & MCP support in the Realtime API GA! `gpt-realtime` is a lot more natural and expressive, and every time a SOTA voice model is released, you know what I gotta do... Here is the new voice Marin, on KaraokeBench! scores 3 out of 7 again from me
10:41

Voice is the OG modality. So excited for image inputs, function calling & MCP support in the Realtime API GA! `gpt-realtime` is a lot more natural and expressive, and every time a SOTA voice model is released, you know what I gotta do... Here is the new voice Marin, on KaraokeBench! scores 3 out of 7 again from me

swyx

51,219 views • 11 months ago

That’s it. That’s the tweet. The Realtime API now supports WebRTC—you can add Realtime capabilities with just a handful of lines of code. We’ve also cut prices by 60%, added GPT-4o mini (10x cheaper than previous prices), improved voice quality, and made inputs more reliable.
5:36

That’s it. That’s the tweet. The Realtime API now supports WebRTC—you can add Realtime capabilities with just a handful of lines of code. We’ve also cut prices by 60%, added GPT-4o mini (10x cheaper than previous prices), improved voice quality, and made inputs more reliable.

OpenAI Developers

523,494 views • 1 year ago

So you can just give it a postgres URL (or sqlite db file) and it will just... explore it and offer to build things. Even with a crazy nonstandard (from Ecto perspective) database. The search bar crashed, I sent the logs to the agent, and boom realtime searching
1:27

So you can just give it a postgres URL (or sqlite db file) and it will just... explore it and offer to build things. Even with a crazy nonstandard (from Ecto perspective) database. The search bar crashed, I sent the logs to the agent, and boom realtime searching

Chris McCord

12,758 views • 1 year ago

Introducing Realtime TTS-2, a new generation of voice model built for realtime conversation. It is the first voice model that hears the conversation, takes natural-language voice direction, holds one voice identity across over 100 languages, and speaks like a person who is paying attention. The result is voice AI that feels as good as it sounds. Try it out: Learn More:
2:06

Introducing Realtime TTS-2, a new generation of voice model built for realtime conversation. It is the first voice model that hears the conversation, takes natural-language voice direction, holds one voice identity across over 100 languages, and speaks like a person who is paying attention. The result is voice AI that feels as good as it sounds. Try it out: Learn More:

Inworld AI

326,393 views • 3 months ago

I've been having a ton of fun exploring the Realtime API from OpenAI. Today they just launched the ability to connect to it over WebRTC! 🤖🖐️ It supports Tool Calling, so check out what that looks like in Realtime function calling in any browser based app. I built a template that allows for relaying through a Cloudflare Developers Worker so you can do it too! Full 📼 in 🧵
1:00

I've been having a ton of fun exploring the Realtime API from OpenAI. Today they just launched the ability to connect to it over WebRTC! 🤖🖐️ It supports Tool Calling, so check out what that looks like in Realtime function calling in any browser based app. I built a template that allows for relaying through a Cloudflare Developers Worker so you can do it too! Full 📼 in 🧵

Craig Dennis

60,122 views • 1 year ago

Today I built this voice agent while on a call with it Introducing /this-needs-a-call - it's a slash command to start a phone call - grill or be grilled by the voice agent in realtime - your coding agent listens in on the call via MCP and works while you talk for hours
6:15

Today I built this voice agent while on a call with it Introducing /this-needs-a-call - it's a slash command to start a phone call - grill or be grilled by the voice agent in realtime - your coding agent listens in on the call via MCP and works while you talk for hours

jacob paris ▲

32,463 views • 1 month ago

At OpenAI DevDay, we introduced the Realtime API with a demo of an interactive solar system you can navigate with your voice. A lot of you have asked how it was built, so we've open-sourced it—with a twist: this version uses our new WebRTC integration. Have fun exploring! 🌍
1:44

At OpenAI DevDay, we introduced the Realtime API with a demo of an interactive solar system you can navigate with your voice. A lot of you have asked how it was built, so we've open-sourced it—with a twist: this version uses our new WebRTC integration. Have fun exploring! 🌍

Katia Gil Guzman

316,561 views • 1 year ago

OpenAI shipped a new speech-to-speech model today: gpt-realtime-2 This is the first speech-to-speech model good enough to use in my voice agents that do "real work." Or real play, for that matter. Here's gpt-realtime-2 as the brain of the ship AI in Gradient Bang. The voice-to-voice response and tool calling times here are unedited, so you can see exactly what the interaction with the model is like in an agent with a very complex system instruction and frequent tool calls. (I did clip out the subagent task execution segments, after gpt-realtime-2 starts a subagent via a tool call. Subagents in this config used gpt-5.2 "medium" effort.)
3:13

OpenAI shipped a new speech-to-speech model today: gpt-realtime-2 This is the first speech-to-speech model good enough to use in my voice agents that do "real work." Or real play, for that matter. Here's gpt-realtime-2 as the brain of the ship AI in Gradient Bang. The voice-to-voice response and tool calling times here are unedited, so you can see exactly what the interaction with the model is like in an agent with a very complex system instruction and frequent tool calls. (I did clip out the subagent task execution segments, after gpt-realtime-2 starts a subagent via a tool call. Subagents in this config used gpt-5.2 "medium" effort.)

kwindla

55,222 views • 3 months ago

Introducing Genspark Realtime Voice. I connected Genspark to my car and headed to work. With Genspark Realtime Voice, prepping for the day feels as easy as chatting. It can check my schedule, send emails and messages, look things up, and even make me a commute playlist. It can also help me get real work done — generate slides, run deep research, analyze data, and more. Just talk. Fully hands-free!
2:16

Introducing Genspark Realtime Voice. I connected Genspark to my car and headed to work. With Genspark Realtime Voice, prepping for the day feels as easy as chatting. It can check my schedule, send emails and messages, look things up, and even make me a commute playlist. It can also help me get real work done — generate slides, run deep research, analyze data, and more. Just talk. Fully hands-free!

Genspark

169,275 views • 4 months ago

Huge News: Base44 is now available as a standalone backend, designed for the way software is built today: CLI-first, realtime, and AI-agent friendly. We’re opening up Base44’s backend as a standalone product. You can now deploy a production-ready backend - auth, database, realtime, and hosting - directly from the CLI.
0:50

Huge News: Base44 is now available as a standalone backend, designed for the way software is built today: CLI-first, realtime, and AI-agent friendly. We’re opening up Base44’s backend as a standalone product. You can now deploy a production-ready backend - auth, database, realtime, and hosting - directly from the CLI.

Base44

314,964 views • 5 months ago

Using OpenAI gpt-realtime-2 to get a glimpse of future voice-first experiences. A market dashboard you don’t click through. You direct it. Say, “Focus on Apple,” and the whole interface changes. Ask, “How did it do over the last 30 days?” and the chart updates. Say, “Go back,” and the market view returns. No menus. No filters. No hunting around. Just intent. What makes this model especially interesting is the interaction loop: you can interrupt it, add more context, change direction, and it keeps reasoning in real time while updating the experience around you. The interface doesn’t ask you to navigate. It just takes you there.
4:43

Using OpenAI gpt-realtime-2 to get a glimpse of future voice-first experiences. A market dashboard you don’t click through. You direct it. Say, “Focus on Apple,” and the whole interface changes. Ask, “How did it do over the last 30 days?” and the chart updates. Say, “Go back,” and the market view returns. No menus. No filters. No hunting around. Just intent. What makes this model especially interesting is the interaction loop: you can interrupt it, add more context, change direction, and it keeps reasoning in real time while updating the experience around you. The interface doesn’t ask you to navigate. It just takes you there.

Levin Stanley

99,168 views • 3 months ago