I'm sharing a tutorial on creating interactive 3D experiences, combining Three.js and MediaPipe hand tracking 👋💠 In this 🧵: - a step-by-step guide including the 4 prompts I used to build this demo - the open source code - Other thoughts about vibe coding

Prompt 1: "create a demo using mediapipe hand tracking and vanilla js. Show a full width/height webcam feed on the page (mirrored). draw hand landmarks on top of the user's hands. Keep it simple and ensure that it runs locally. We will add more features later on" This gives us a basic webcam feed with hand landmarks drawn on top. (btw, I'm using Claude web for this, but this should a;so work with ChatGPT, Gemini, Grok, etc.)

Prompt 2: "ok now let's integrate threejs into this demo. Draw a sphere in the scene. It should have a vibrant neon color fill and white wireframe outline. The scene should overlaid on top of the webcam feed and should have no background and no visual elements other than the floating shape. The shape should auto-rotate slowly and stay in the center of the screen. Keep it simple we will add hand tracking interaction after" This now gives us a threejs shape in the scene, but it's not yet linked with the hand movements

Prompt 3: "now let's add hand tracking interaction to this demo. Right hand: distance between thumb/index finger controls the size of the sphere. Left hand: if the index finger touches with the shape, the color fill of the shape should change to a random neon color" Now we're cooking! the threejs shape is now linked with the mediapipe hand tracking. The positions of the fingers are now being used to drive the size and color of the shape in real-time.

Prompt 4: "add some smoothing to the resizing of the sphere, to prevent jittering/glitching of the rapid unintended size changes" Just some clean-up to make the size changes smoother. Voila! Our demo is complete. This was a basic example, but the same principles can be used to create games, interactive websites, and other fun augmented reality applications.

Here's the github repo with the open source code: And here's the live demo:

Hey guys, here's the tutorial I promised I'd make for you! Sorry it took so long, I got distracted on some side quests 😅 Hope this is useful! Any likes / reposts would be appreciated 🙇‍♂️ @benallfree @AlexerSalcido @craigrrob @AIlife2024ka @thechrislacorte @roileean @artofnazar @3DBeeing @ThaboMode @sarabahri @TrentJConley

@threejs built something similar, what do you think?

@threejs Really smooth tracking, nice work! I like the integration with voice mode search. What tech stack did you use for this?

@threejs Cool! I was the PM for MediaPipe for a while, so I'm very happy to see this work. Just FYI friend Uri has been using it to create a theremin & more:

@threejs Thank you for everything you did with MediaPipe. It's a lovely piece of tech. So powerful and easy to get started with it. I've been obsessed with it lately and will continue to learn / explore :) The theremin projects looks amazing, thanks for sharing