Built a "YouTube realtime copilot" browser extension using OpenAI's realtime 2 API: The agent watches the video alongside you, and can answer any question you have about what was just said via realtime voice chat. The crazy part to me is: It can differentiate the YouTube's audio stream and... your voice, so it doesn't confuse the video as commands, and stays silent unless you ask something!show more

This feels like the right direction for AI video. Not just “summarize this video”, but watch with me, answer in context, and help me understand the part I just missed. The UX shift is from search-after-watching to assistance-while-watching.

auto-pause when a user speaks so Copilot can engage seamlessly. Requiring manual intervention significantly degrades the experience; automation is key to overall quality.

wait, does it actually parse the audio stream or just grab captions? curious if that separation trick holds up on chaotic tech talks

on Github yet? need it!

Kudos for a great addition to my list of best tools to learn languages 💪🏼🔥🚀

This is not better than the YouTube so feature

How much time do you spend vibe coding a product?

太get這個邊看邊問的插件了！太強了，我幾天前還在盧頻扣字幕，現在已經能實時陪看了😂 我特別好奇這個插件從idea到上線要多久呢？對於普通vibecoder來說最難復製的是音頻流區分嗎？求指點

真的是非常需要了哈哈！

I would love to see this way of human-AI interaction in more personal, casual settings -- like watching a reality TV show together and chat about it like friends, which will definitely scratch an itch when my bf doesn't want to watch Love is Blind with me :)

watching together is the killer pattern for realtime. way more useful than the agent watching alone and summarizing. what's the latency feeling like in practice?

You can paste the link into Gemini amd ask questions about the video.

Multimodal context window doing the heavy lifting there.

Really nice!

such good use of the realtime api! never thought of piping multiple sources

This is great!

The cost is gonna be crazy bro

In the olden days your Mom or Dad and/or GrandMom did that 💋‼️😉

passive watching is officially dead, every video is now a conversation

Wow!!

have the same idea!

Audio stream differentiation without false triggers separates toys from tools. What latency are you hitting?

what is the cost of running this lol on Lex pod this could cost you a lot of money?!!!

cool

Ohh this is amazing And can come in handy

really cool idea zara 🔥

How strong is that differentiation I wonder, would it hold up if a video maker intentionally tried to inject prompts? We could really use not confusing context with commands in literally every ai product.

Great I am also thinking about buying a gun to kill a mosquito. And this line is about neither gun not mosquito.

So cool! It just listen to the audio right? What about diagrams / visuals on the video?

Why such an over-engineering solution? Even if you use gemini in chrome or perplexity comet, they both have this built in by default.

@gabrielchua You’ve gotta gothub this!

The voice/audio diarization piece is the part that quietly took years to get right. Tested OpenAI realtime 2 yesterday on background podcasts and it stayed silent through 90 minutes of video soundtrack until I directly addressed it. That separation is the real moat. Most realtime systems still spike on transient music.

intresting how can i use it too>

I've got a project on my to-do list that can extract key information and highlights from videos, though I was leaning into things like teams meetings over YouTube, using vision models and LLM, having something that can find and give me terminal commands used in a demo and so on