One more thing I am working on! Hand Tracking on Steam Frame!!! runs locally, using the headsets front facing tracking cameras for hand tracking! It will be open source, it will take a while to get it better, likely to train custom models, but the proof of concept worked!

Going to be working with some friends who are really good at custom model training, so this can be optimized, as well as I plan on working on the UX to allow for easy navigation of the Steam UI (you can see the laser in the video)

again this is VERY early, will need some time before it will be out publicly! In the meantime, I hate to ask, I am jobless right now, and while this gives me time to make projects, I still would like to eat :) if you enjoy this work, buy me a coffee!

also I will post a guide on how to get camera access once this is out!

(its fairly easy and im likely not the first, but I like to write big guides)

Cade you madman

my brain hur

@HardwareSteam Bro I say this with love but how did you go from zero to GOATed in no time flat

@HardwareSteam I have spent a LOT of time with this thing learning about it, trying to figure it out. as well as upping my skills as a dev since reality hack this year!

Can confirm, this man has hands

Does this works with the base tracking cameras already in the steam frame? Or do you need the Arcturus Vision camera?

its using the base cameras already on the Frame!

Oh sick!

I worked on something like this a bit but it was way too costly to run and kept crashing my headset (probably because I messed something up) glad to see someone else get it running too!

yeah its without a doubt rough around the edges, I dont think its viable without training custom models, and it has occasionally crashed the compositor, but I was able to clean it up and had it running fairly stable

Yeah I never got it stable, this was like 7 months ago though, llms have gotten a fair bit better in that short time haha. I quit before I even got the hands lined up correctly, just had laggy debug hands that followed my fingers :p Can prob record a video of it tomorrow...

Incredible! Can’t wait to help and see it take off

this guy is gonna cook with me

I want hand tracking on Steam Frame so bad good luck 🫡

you are incredible homie

this is the kind of work I hoped to see shortly after the frame dropped. great stuff

Did you’re a hero

Yes please 🙏🥺

The frame modding is already starting. I dont even have mine yet. Love how open valve is with this stuff.

:V

yo @valvesoftware hire this guy

Hype!

This is one of the cool things about the steam frame most people skip. If you want extra functionality like hand tracking, you can just implement it. No closed off garden, no privative limited access api. You can just do things

this is reminding me of the monado hand tracking that worked on the index

Oh dude I'd definitely be down to help if you wanted some extra testers and tinkerers. Hopefully when my frame comes in next week 😁

what do you do for the color pass through? i really really want a steam frame but i mainly want it for mix reality so the lack of color cameras is a huge turn off

amazing work! vr cs2 when?

Didn’t Valve say it’s already working on steam frame in an interview, it’s just underneath right now. Also what can we do with hand tracking once we get it and will it have occlusion

I mean yeah valve is doing it, but doesn't hurt to have fun and build stuff! and for questions like that, not a clue, this is a proof of concept, I haven't nearly gotten to the point where I can promise anything like that

Are you able to use the Frame's NPU?

This is very sick and awesome mr. framedrop but also please add linux support to framedrop mr framedrop please and thank you

Looks cool!

@SadlyItsBradley Is it possible to force leap motion tracking to work somehow?

Whaaaaaaa hand tracking nice

It seems really bad...

This cant be real wow already.

You're such a legend man

This looks cool asf!!!

Can this also be done for @pimaxofficial dream air? I understand that’s a different headset but camera feed is a camera feed, right? And pimax don’t seem to be doing anything to make theirs working anywhere else but pimax home, at least not in vrchat

@Cade_XR Can the arcturus run at higher fps but lower res?