Head-tracked “Window Mode.” Your front camera finds your head. The view reprojects in real time so the screen feels like a window into the 3D scene. True3D, no glasses.

How it works: Estimate head pose relative to the display, then render with head-coupled perspective so near objects shift more than far ones. That motion parallax sells depth even in 2D

What you feel: A sense of presence on a flat screen. The view responds like real objects do when you move, so motion parallax sells depth without glasses. Here’s a favorite: Steamboat Willie in 3D with Window Mode on.

Why this matters: most 3D on phones collapses to a flat video. Asking viewers to drag the camera fights lean-back behavior. Window Mode adds presence without new habits.

Try it in your browser: (Works on phones and laptop webcams too.)

I wrote more on the design goals and tradeoffs. Latency budget, pose smoothing, and when to prefer single-view depth over stereo.

Paying respect: Johnny Lee’s 2007 Wii Remote head-tracking demo sparked this whole idea for many of us. Ours runs on-device with a front camera and lightweight models.

You can try the Steamboat Willie example here:

I’ll post in our Discord when we open source the code in a few weeks.

For those asking: Frontend is spatialjs and the file format for 3D videos ".splv" is in spatialstudio. Going open source in the next few weeks; we’ll post in Discord. Our backend and models are on @modal and @googlecloud. Using Modal & GCP for over a year, both have great docs and simple APIs. Highly recommend Spatial Studio - PyPI:

@googlecloud We also set up a subreddit where people share 3D videos they’ve created and follow along as we build in public:

@googlecloud We just open-sourced Window Mode Video explainer coming soon on our YouTube: Can’t wait to see what people build with it

now i want to see an infinity castle using this haha

just wait....

Nintendo 3DS did this a while back I miss it every day :(

same :/

Can’t laugh at people who move their head playing video games anymore

im more into 3d video viewing, but i think there is potential for gaming. things are weird until they aren't

3d scan the room and the table and chairs, use that instead of this demo grid stuff, make it so perfect and seamless as if your laptop's display is a see-through glass

check back in in a couple of days

🔥

It’s not true 3D because it’s still rending to a 2D screen.

Psuedo3D doesn't have the same ring to it

@flutterwhat Cool idea for sure, but yeah, it does lack a true 3D effect if your eyes are seeing the same image, you won't see depth, you just interpret it by moving your head around. Now if you could apply this to the tech that the 3DS uses, you got yourself a true 3D window.

@flutterwhat I prob need to just make my own hardware at this point

Microsoft patented this in the 90s. I know because I came up with the idea. It looks so good though! Keep it up and what is it built in?

Dam you were ahead of the curve, would be glad to chat. cool if i DM? Built this on top of Spatial Studio, a library we are open sourcing this month Will share more soon here

Absolutely! DM for sure.

well done. tried to do this like 10 years ago when i was at intel. it "worked" but the latency was pretty bad and the tracking wasn't reliable enough yet

consumer devices have come a long way, you should try again! Happy to help if you get stuck

thanks. i've been retired a few years now and not actively coding much now. have you seen @gfodor just released? would be an awesome combination!

@gfodor Just saw it earlier When I worked at Meta i did some work on MR tools for VR creators like this. Would be happy to help where i can

would it work with art if all the layers are separated vectors?

yeah, it would be very easy to integrate that. Going to open source the code over the next 2 days in the discord. Lmk if you need help integrating it

dope! i'll give it a shot

Yeah I've seen this like 15 years ago on youtube. The illusion works a lot better when you don't have 2 eyes. It's misleading to showcase this with a single camera view.

not sure if you read the full thread, but everything is pretty clearly explained in there

We used to need a Kinect or Wii hardware to get fast tracking for this effect. Now any modern laptop with a camera can achieve it. I hope we’ll see this as an option in more games and simulations and modeling software going forward!

We’ve got some fun follow-ups coming. Stay tuned.

Hahaha just wanted to reply „that looks like that one wii remote demo“ 🙈😅

It basically IS the wii remote demo just on face cams

@i_am_fabs I was looking for this. Good to see that someone finally put that old wiimote demo to use again.

@i_am_fabs Johnny Lee paved the way

Vr tubers will pay you billions to have adaptive eye tracking per viewer.

@TheValue_Guy Going to have some really fun things to share on this front in a few weeks

Remember doing this with Nintendo Wii Remote back in the days repeating Johnny Lee. Head tracking idea is awesome!

We tried to get ours as close to his as possible, he was a huge inspo for this experiment

You successfully provoked my entirely juvenile "I want one at any cost" response. That's a good thing. ;) If you can do it at reasonable *pu load it's like printing money. ;). Plug it into Unreal.

I think you’ll be happy with what’s coming

I hope so. But be prepared for being randomly hugged, high-fived, and treated as heroes for it. ;) lol

So this is how we have to give 24x7 spying permission to windows. Got it. They could have just straight asked for it. The game is so open, why bother making these junk?

Nothing is stored. Open-sourcing the code in a few days so you can verify it yourself.

Does it track cat heads too?

I would be interested in this test, would be very interested in video proof of the results

How well does it work with people with darker skin.

very well from what we've tested

Awesome!!!

How far have we gone...

not far enough

I guess using webcam to track two light sources would be still way easier, right?

How will this work for multiple heads ?

it doesn't

So, in the case of multiple heads, which head does it track? Can you fix which face to track? More compute and latency? Or does it track anything resembling a head shape?

@Sal_Chowdhury A deeper dive here It doesn’t work well with multiple people. The main limitations of this approach are: 1. No stereoscopic depth 2. Only supports a single viewer 3. Requires the front camera to be on Super important latency is tight

Thank you for sharing. This is really fascinating and I can imagine that uniquely interactive perspective based puzzle/mystery games could be developed with this. Might even be useful in medical imaging. Will defo take a look.

Nice! I experimented with this back in the day on PS2 using Eyetoy to find a leds attached to the players glasses. The effect was extremely effective. Didn't have the compute power to find head orientation from the video feed :-)

compute on consumer devices is climbing rapidly, almost exponentially excited to see what else gets unlocked

More cool stuff!

why do i get the distinct feeling i’ve seen this done before, a decade or two ago, using Nintendo Wii hardware? right down to the red/white targets-on-sticks aesthetic

Checkout the full thread, it links to the OG posting with the Wii mote from 17 years ago

ah i knew i wasn’t tripping. glad it wasn’t a Mandela Effect thing i was going through 😅 anyway, cool work dude, really neat. always love seeing innovative uses for existing tech like that, shame the methods aren’t applied more frequently

I don’t know how but this has a huge pontential to make money out there somewhere somehow

@KyleLearnsTech just wait till you see whats coming over the next few weeks

Lmao - just followed - make a waitlist and I’ll be there

break the targets

You can use window mode on video games too

Insanely trippy I want this on my wall

would be super easy to do that

That would be sick, I’m curious how it would handle multiple people

Coolest thing ive seen in a while

just wait til you see whats cooking

Amazing, looking forward to the open source, can think of a few real-world applications that this would be amazing for

excited to see what you build, will be sharing the code tomorrow in the discord

Alan @measure_plan, I'd love to see what you do with ideas like this!

@measure_plan open sourcing the code on fri in our discord, happy to help you setup whatever project you are working on

@measure_plan I just signed up to the Discord!

Unfortunately the effect looks better on camera than it does in real life :| It's disingenuous to call this True3D, it's nothing like glasses. Interestingly if you close one eye, it improves. Thus, Johnny Lee and I once joked that we should bundle this with a pirate eye patch :P

someone make a puzzle game that utilizes looking at the scene through new perspectives

Would love to see this

Getting break the targets from Super Smash Bro n64 game ptsd here

@BVega92 you can actually watch that in 3d too..... plz forgive my horrible time

but someone watching alongside you, sharing an experience, sees some weird lines and projections. I'm sure that'll be useful for one person in a place at a time, but it's otherwise not very valuable.

yeah only useful for solo viewing

I haven’t seen this demo in probably a decade. Flashback

17 years to be exact

Really cool! Looks like the baseball pitch view

not sure what the baseball pitch view is, any chance you have a screenshot / link ?