This would have been impossible a year ago. Last week we built a new approach to object detection for Sidequest.xyz: generative spatial understanding. Demo & explanation👇

The old way is very limiting - you might detect 100 basic items like laptops, books, etc with a YOLO model. Or you'd have to painstakingly train your own models for your individual use case.

In this demo, the label and the coordinates are generated by an LLM and anchored to the real world in real time. This gives you so much more context vs older methods.

From here you can easily build generative spatial interfaces for: - Teaching real world skills - AR games - Field work guides - Smart home device interactions

Traditional computer vision models are still useful for really nailing the coordinates and specific scenarios but vision language models give a ton of power for much less effort and they're only getting better.

⏰ Countdown alert! 🚀 Get free access to our new AI Writing feature before Dec 1st! Don't miss your chance to explore its power. Try it now—no cost, no catch! 🔗

@sidequest Opening up the camera api was so important

@sidequest Awesome!

@sidequest thanks for the support Robert!

@sidequest woah was surprised it got the llama from that angle and the latency is v impressive

@sidequest it's pretty magical - this is also with little optimizations for latency and we're sending more of the camera view than we need in another run it even got that the software on the laptop was for coding AR

@Scobleizer @sidequest seeing growth through innovation is inspiring. what journeys bring those ideas to life?