Loading video...

Video Failed to Load

Go Home

Voice is the OG modality. So excited for image inputs, function calling & MCP support in the Realtime API GA! `gpt-realtime` is a lot more natural and expressive, and every time a SOTA voice model is released, you know what I gotta do... Here is the new voice Marin,...

swyx's profile picture

swyx

127,338 subscribers

51,219 views • 9 months ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Introducing GPT-Realtime-2 in the API: our most intelligent voice model yet, bringing GPT-5-class reasoning to voice agents. Voice agents are now real-time collaborators that can listen, reason, and solve complex problems as conversations unfold. Now available in the API alongside streaming models GPT-Realtime-Translate and GPT-Realtime-Whisper — a new set of audio capabilities for the next generation of voice interfaces.
4:03

Introducing GPT-Realtime-2 in the API: our most intelligent voice model yet, bringing GPT-5-class reasoning to voice agents. Voice agents are now real-time collaborators that can listen, reason, and solve complex problems as conversations unfold. Now available in the API alongside streaming models GPT-Realtime-Translate and GPT-Realtime-Whisper — a new set of audio capabilities for the next generation of voice interfaces.

OpenAI

3,617,950 views • 1 month ago

The Realtime API is officially out of beta and ready for your production voice agents! We’re also introducing gpt-realtime—our most advanced speech-to-speech model yet—plus new voices and API capabilities: 🔌 Remote MCPs 🖼️ Image input 📞 SIP phone calling ♻️ Reusable prompts
1:40

The Realtime API is officially out of beta and ready for your production voice agents! We’re also introducing gpt-realtime—our most advanced speech-to-speech model yet—plus new voices and API capabilities: 🔌 Remote MCPs 🖼️ Image input 📞 SIP phone calling ♻️ Reusable prompts

OpenAI Developers

561,665 views • 9 months ago

Introducing Realtime TTS-2, a new generation of voice model built for realtime conversation. It is the first voice model that hears the conversation, takes natural-language voice direction, holds one voice identity across over 100 languages, and speaks like a person who is paying attention. The result is voice AI that feels as good as it sounds. Try it out: Learn More:
2:06

Introducing Realtime TTS-2, a new generation of voice model built for realtime conversation. It is the first voice model that hears the conversation, takes natural-language voice direction, holds one voice identity across over 100 languages, and speaks like a person who is paying attention. The result is voice AI that feels as good as it sounds. Try it out: Learn More:

Inworld AI

324,040 views • 1 month ago

Voice is an interface, and I feel we’re going to see voice explode. OpenAI gpt-realtime is out and allows for native speech-to-speech and precise tool calling in real time. It can also take image inputs and place SIP phone calls - good for a quick triage before human handoff.
1:20

Voice is an interface, and I feel we’re going to see voice explode. OpenAI gpt-realtime is out and allows for native speech-to-speech and precise tool calling in real time. It can also take image inputs and place SIP phone calls - good for a quick triage before human handoff.

Allie K. Miller

79,857 views • 9 months ago

OpenAI shipped a new speech-to-speech model today: gpt-realtime-2 This is the first speech-to-speech model good enough to use in my voice agents that do "real work." Or real play, for that matter. Here's gpt-realtime-2 as the brain of the ship AI in Gradient Bang. The voice-to-voice response and tool calling times here are unedited, so you can see exactly what the interaction with the model is like in an agent with a very complex system instruction and frequent tool calls. (I did clip out the subagent task execution segments, after gpt-realtime-2 starts a subagent via a tool call. Subagents in this config used gpt-5.2 "medium" effort.)
3:13

OpenAI shipped a new speech-to-speech model today: gpt-realtime-2 This is the first speech-to-speech model good enough to use in my voice agents that do "real work." Or real play, for that matter. Here's gpt-realtime-2 as the brain of the ship AI in Gradient Bang. The voice-to-voice response and tool calling times here are unedited, so you can see exactly what the interaction with the model is like in an agent with a very complex system instruction and frequent tool calls. (I did clip out the subagent task execution segments, after gpt-realtime-2 starts a subagent via a tool call. Subagents in this config used gpt-5.2 "medium" effort.)

kwindla

54,912 views • 1 month ago

Voice workflows just got stronger with gpt-realtime-1.5 in the Realtime API. The model offers more reliable instruction following, tool calling, and multilingual accuracy. Demo with Charlie Guo
1:46

Voice workflows just got stronger with gpt-realtime-1.5 in the Realtime API. The model offers more reliable instruction following, tool calling, and multilingual accuracy. Demo with Charlie Guo

OpenAI Developers

417,428 views • 4 months ago

the new OpenAI realtime voice model just released + gpt 5.5 fast mode brings us a new possibility - realtime speech to live presentation! i just talk, and the whiteboard would whiteboard itself prototype is open sourced. details in thread below -
1:46

the new OpenAI realtime voice model just released + gpt 5.5 fast mode brings us a new possibility - realtime speech to live presentation! i just talk, and the whiteboard would whiteboard itself prototype is open sourced. details in thread below -

Kun Chen

114,160 views • 1 month ago

OpenAI's o1 model is officially out of preview and available to developers via API! Plus, new capabilities for Realtime API builders (like the cool stuffed animal voice demo below). New o1 features include: > Function calling to connect with external data > Developer messages for tone/style control > Image analysis capabilities > "reasoning_effort" to control thinking time > Preference fine-tuning This is a huge unlock for developers (but the pricing is not cheap). $15 per ~750k words analyzed and $60 per ~750k words generated — roughly 3-4x more than GPT-4o. In addition to the o1 news, we also got updates to the Realtime API: > WebRTC integration (smoother voice apps) > 60% price drop for GPT-4o audio > New GPT-4o and 4o mini models > Background task support Big day for AI builders!
2:00

OpenAI's o1 model is officially out of preview and available to developers via API! Plus, new capabilities for Realtime API builders (like the cool stuffed animal voice demo below). New o1 features include: > Function calling to connect with external data > Developer messages for tone/style control > Image analysis capabilities > "reasoning_effort" to control thinking time > Preference fine-tuning This is a huge unlock for developers (but the pricing is not cheap). $15 per ~750k words analyzed and $60 per ~750k words generated — roughly 3-4x more than GPT-4o. In addition to the o1 news, we also got updates to the Realtime API: > WebRTC integration (smoother voice apps) > 60% price drop for GPT-4o audio > New GPT-4o and 4o mini models > Background task support Big day for AI builders!

Rowan Cheung

38,690 views • 1 year ago

just found out from Alex Volkov's show that @chatgptapp advanced voice upgraded singing capability this week so here is karaokebench would say it is like 3/7 so far are there other sota singing models?
5:34

just found out from Alex Volkov's show that @chatgptapp advanced voice upgraded singing capability this week so here is karaokebench would say it is like 3/7 so far are there other sota singing models?

swyx 🇸🇬 AIE Singapore!

71,555 views • 1 year ago

Alex Gamble Perplexity .Intercom is using the Realtime API to power Fin Voice, their AI Agent for phone support. The new model follows call scripts better, executes function calls more consistently, and hallucinates less, helping provide human-quality AI support, 24/7.
1:00

Alex Gamble Perplexity .Intercom is using the Realtime API to power Fin Voice, their AI Agent for phone support. The new model follows call scripts better, executes function calls more consistently, and hallucinates less, helping provide human-quality AI support, 24/7.

OpenAI Developers

23,409 views • 1 year ago

Voice-only programming with the new OpenAI Realtime API ... I spend a lot of time these days pair programming with LLMs. Often I'm talking rather than typing. This "voice dictation" use case has become an important vibe benchmark for me. Being able to create text input just by talking, flexibly, in a context dependent way, with tool calling, is a *hard* problem for today's models. Natural language dictation requires a very high degree of contextual intelligence, instruction following accuracy, and tool calling reliability. Today's new gpt-realtime model is quite good at this hard problem. The original realtime model release last year was impressive. Seeing what a speech-to-speech model could do got a lot of people excited about the possibilities of voice AI. The improvements since that first release are equally impressive. I can use this new model, now, for real world tasks that were past the edge of the "jagged frontier" before. Here's a video showing a couple of fun (and tricky) modes of voice input.
2:40

Voice-only programming with the new OpenAI Realtime API ... I spend a lot of time these days pair programming with LLMs. Often I'm talking rather than typing. This "voice dictation" use case has become an important vibe benchmark for me. Being able to create text input just by talking, flexibly, in a context dependent way, with tool calling, is a *hard* problem for today's models. Natural language dictation requires a very high degree of contextual intelligence, instruction following accuracy, and tool calling reliability. Today's new gpt-realtime model is quite good at this hard problem. The original realtime model release last year was impressive. Seeing what a speech-to-speech model could do got a lot of people excited about the possibilities of voice AI. The improvements since that first release are equally impressive. I can use this new model, now, for real world tasks that were past the edge of the "jagged frontier" before. Here's a video showing a couple of fun (and tricky) modes of voice input.

kwindla

50,806 views • 9 months ago

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!
1:18

Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!

Zara Zhang

44,631 views • 1 month ago

My favorite demo of the new gpt-realtime model from Matthieu LC -- Shoggoth Mini using Realtime API with image input
2:45

My favorite demo of the new gpt-realtime model from Matthieu LC -- Shoggoth Mini using Realtime API with image input

Peter Bakkum

67,777 views • 9 months ago

The new Realtime API from OpenAI is incredible... Watch it order 400 strawberries by actually CALLING the store with twillio. All with voice. 🍓🎤
6:28

The new Realtime API from OpenAI is incredible... Watch it order 400 strawberries by actually CALLING the store with twillio. All with voice. 🍓🎤

Ty

495,453 views • 1 year ago

That’s it. That’s the tweet. The Realtime API now supports WebRTC—you can add Realtime capabilities with just a handful of lines of code. We’ve also cut prices by 60%, added GPT-4o mini (10x cheaper than previous prices), improved voice quality, and made inputs more reliable.
5:36

That’s it. That’s the tweet. The Realtime API now supports WebRTC—you can add Realtime capabilities with just a handful of lines of code. We’ve also cut prices by 60%, added GPT-4o mini (10x cheaper than previous prices), improved voice quality, and made inputs more reliable.

OpenAI Developers

523,470 views • 1 year ago

Our latest speech-to-speech model is faster, more accurate, and excels at function calling. Watch @promptshant and Brian Fioca build a realtime voice agent that can search the web and hand off tasks to reasoning models with full context.
1:58

Our latest speech-to-speech model is faster, more accurate, and excels at function calling. Watch @promptshant and Brian Fioca build a realtime voice agent that can search the web and hand off tasks to reasoning models with full context.

OpenAI Developers

81,822 views • 1 year ago

Developers are building great voice experiences with the Realtime API. See how Lemonade uses it to power AI Maya, their friendly and engaging voice agent. The Realtime API's automatic voice detection and low latency enable them to offer 24/7 multilingual phone support 📞
4:50

Developers are building great voice experiences with the Realtime API. See how Lemonade uses it to power AI Maya, their friendly and engaging voice agent. The Realtime API's automatic voice detection and low latency enable them to offer 24/7 multilingual phone support 📞

OpenAI Developers

126,054 views • 1 year ago

We're adding MCP capability to Realtime API, I'm very excited about how well MCP tools work over voice. Here's a demo using a Notion MCP --
1:52

We're adding MCP capability to Realtime API, I'm very excited about how well MCP tools work over voice. Here's a demo using a Notion MCP --

Peter Bakkum

27,261 views • 9 months ago

The new Realtime API with web crawling is mind-blowing! Talk in realtime with any website. Powered by the OpenAI Realtime API and 🔥 Check it out:
1:13

The new Realtime API with web crawling is mind-blowing! Talk in realtime with any website. Powered by the OpenAI Realtime API and 🔥 Check it out:

Nicolas Camara

347,089 views • 1 year ago

Voice Design v3 is here. Create any voice you can imagine with a prompt. We’ve rebuilt the underlying Voice Design model to deliver higher quality and broader expressive range. Generate production-ready voices in 70+ languages with support for hundreds of localized accents.
0:52

Voice Design v3 is here. Create any voice you can imagine with a prompt. We’ve rebuilt the underlying Voice Design model to deliver higher quality and broader expressive range. Generate production-ready voices in 70+ languages with support for hundreds of localized accents.

ElevenLabs

153,848 views • 1 year ago