Augmented Reality is the natural interface for robotics. Not only for giving commands, but also to visualise hardware data (like LiDAR) natively in 3D. SPECS AR glasses, Unitree Go2 Lite quadruped, Dimensional OS for the robot

Repo if you want to follow along the development:

Testing outdoors in real world conditions is such a different world than in a controlled indoors environment. Glasses thermal limits in the sun, WiFi range, uneven terrain. And tall grass (Final boss for the doggo). I get what all the robotics people are saying now…

@specs @UnitreeRobotics @dimensionalos semi related but I think the way we do "directions" for navigation is terrible. The number of hours trying to find an Airbnb.. drives me mad AR within the systems makes a lot of sense, and almost none of the rails to do this exist yet

@specs @UnitreeRobotics @dimensionalos Yes, 3D data for navigation essentially. @NianticSpatial is actually working on infrastructure for this

@specs @UnitreeRobotics @dimensionalos @NianticSpatial ah perfect. Someone tell them the intern went waaaay too hard on the fake followers / views lol

This is amazing, feel free to msg me on any pain points when developing since we want to make dimos as convinient as possible. For outdoors nav you can turn off onboard obstacle avoidance, just our planner will replan around obstacles and you'll be able to go over obstacles etc When working on realtime stuff try to write map/reduce type functions and use memory2 transforms, it makes it really easy to evaluate an algo on stored data (check `dimos map global` and docs on memory) We managed to make L1 lidar output actually usable and we are working on 3D path planning now, so all above will be much cooler soon

@specs @UnitreeRobotics @dimensionalos At some point once the Websocket protocol is stable enough, I’d like to make a PR into DimOS with only the (headset agnostic) AR Bridge. It’s still a bit out timewise though. A WebXR client would be a good next step…

@specs @UnitreeRobotics @dimensionalos Yeah we just have a big push on web, nice API will come out in a few weeks

@specs @UnitreeRobotics @dimensionalos Cool, any way to preview that somewhere or some info?

@specs @UnitreeRobotics @dimensionalos just sketch phase now can ping when we have an actual API proposal to go through

@specs @UnitreeRobotics @dimensionalos Will take a look, and a ping would be great!

@specs @UnitreeRobotics @dimensionalos This is fucking insane

@specs @UnitreeRobotics @dimensionalos VR/AR is making sense now. For years, it didn't work for any use cases

@specs @UnitreeRobotics @dimensionalos Hell ya I can’t wait for the specs to ship

@specs @UnitreeRobotics @dimensionalos me instructing my medical supplies to rapidly approach the missing hikers I just found

@specs @UnitreeRobotics @dimensionalos the honest answer is that most robotics work today still relies on outdated interfaces. ar is a game-changer because it lets humans visualize and interact with hardware data in real time, not just issue commands

@specs @UnitreeRobotics @dimensionalos Hell nah why didn't I think of this before

@specs @UnitreeRobotics @dimensionalos Do any protocols like ADS-B exist for systems like this yet? Whether robots or self-driving cars it seems like there needs to be a shared intent broadcast protocol

@specs @UnitreeRobotics @dimensionalos Today I learned about automatic dependent surveillance broadcast Thank you for that - not sure it exists, would assume no

@specs @UnitreeRobotics @dimensionalos Wow just wow 😳

@specs @UnitreeRobotics @dimensionalos it really looks natural to control them that way

@specs @UnitreeRobotics @dimensionalos so cool!

@specs @UnitreeRobotics @dimensionalos A tool to hold an put away is the correct interface.

@specs @UnitreeRobotics @dimensionalos watched operators trust bad lidar overlays faster than raw logs, calibration is the trap

@specs @UnitreeRobotics @dimensionalos Is it tho? (You can talk to humans and tell them to do things without needing AR glasses)

@specs @UnitreeRobotics @dimensionalos 🔥

@specs @UnitreeRobotics @dimensionalos follows instructions better than my dog

@specs @UnitreeRobotics @dimensionalos Refuses to walk through tall grass though 😅

@specs @UnitreeRobotics @dimensionalos Nice, do you use the tag on top of the Go2 for positioning against the specs?

@specs @UnitreeRobotics @dimensionalos Yes! It’s used for initial alignment of the two frames + for drift correction on long distance movements (the rest uses the odometry data from the robot as it moves)

@specs @UnitreeRobotics @dimensionalos What if you have to scratch your nose, will it jump on you to interface with your face?

@specs @UnitreeRobotics @dimensionalos I hope not lol (the marker is dragged by aiming + pinch gesture so…)

@specs @UnitreeRobotics @dimensionalos field data beats lab sims every time. ar overlay turns raw lidar into instant intuition

@specs @UnitreeRobotics @dimensionalos that's fantastic man, love the concept!

@specs @UnitreeRobotics @dimensionalos Thanks mate!

@specs @UnitreeRobotics @dimensionalos Unreal that it took this long for someone to actually buils this instead of just talking about it

@specs @UnitreeRobotics @dimensionalos So I can finally see why my robot dog is ignoring me? LiDAR overlays would definitely explain the attitude.

@specs @UnitreeRobotics @dimensionalos damn look's like cgi Is it real ?

@specs @UnitreeRobotics @dimensionalos Awesome! This is what I was discussing with my daughter that is using smart glasses, csi data and rf to provide the guidance. AR is the natural extension what we agreed to and should flow. Nice to see it in action.

@specs @UnitreeRobotics @dimensionalos Approved. You may continue.

@specs @UnitreeRobotics @dimensionalos Thank you, would have stopped otherwise of course

@specs @UnitreeRobotics @dimensionalos Fantastic work!!!

@specs @UnitreeRobotics @dimensionalos Thank you!

@specs @UnitreeRobotics @dimensionalos sweet demo

@specs @UnitreeRobotics @dimensionalos @Em_Nomadic

@specs @UnitreeRobotics @dimensionalos super cool

@specs @UnitreeRobotics @dimensionalos Very cool. Using the april tag to get the robot <--> glasses transform was very clever. I want to give this a try with my meta quest headset and one of the robot's i'm working on.

@specs @UnitreeRobotics @dimensionalos Hey btw If there's some hardware you'd like to experiment with, I can probably get it funded from our side. No strings, msg me if so

@specs @UnitreeRobotics @dimensionalos 🦾🦾🦾

@specs @UnitreeRobotics @dimensionalos Cold is also a problem on battery (unexpected drained energy/low battery), plastics (break), alternative material generates extra weight problem, humidity damages flexible printed cable (short term + causes undefined behaviours) & displays (acidic damage) & joints (long term)...