Загрузка видео...

Не удалось загрузить видео

На главную

Hand Menu Interaction. The simplest trick to improve the accessibility of floating hand-attached UI panels for spatial interactions is to smooth their movement. It allows us to stabilize them and mitigate a hand tremor due to hand-tracking imperfection and/or physiological condition. (Interestingly, the visual stabilization helps reduce the physical...

Volorf's profile picture

Oleg Frolov

7,030 subscribers

27,218 просмотров • 2 лет назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 9

Фото профиля Oleg Frolov
Oleg Frolov2 лет назад

Download the prototype for Meta Quest devices

Фото профиля vijay verma
vijay verma2 лет назад

Pretty Cool 👌

Фото профиля Oleg Frolov
Oleg Frolov2 лет назад

🙏

Фото профиля Oleg Frolov
Oleg Frolov2 лет назад

The animation choreography depends on the active item. The closest items have zero delay, and then I multiply the distance index by the delay value. #designdetails

Фото профиля Oleg Frolov
Oleg Frolov2 лет назад

Here is the distance index method. Love enums. Initially, I thought to hardcode it but decided to make it analytically which is always fun doing. Yeah, now I can scale the length of the menu infinitely.

Фото профиля Oleg Frolov
Oleg Frolov2 лет назад

A legendary concept!

Фото профиля Future Seeker VR
Future Seeker VR2 лет назад

smooth 👍

Фото профиля Tal
Tal2 лет назад

This looks like the best Hand-menu implementation I’ve seen and intrigues me to built one myself. My only concern: how intuitive do you find it for people how tries VR *for the first time*. Does it take 5 seonds or 1-2 minute to learn? How do you present this mechanic to them?

Фото профиля Oleg Frolov
Oleg Frolov2 лет назад

Thank you! There are no signifiers that there is a menu on the hand, so you should teach users how to invoke it. Therefore, to avoid tutorials, you need to show them the signifiers. It might be little dots on the hand side that would spot the user's curiosity, and they would turn their palm to look at them and invoke the menu by transitioning the dots into menu elements.

Похожие видео

Explored the finger menu with a depth-based layout. Felt fun. Microinteractions/UI motion helped to compensate for the lack of haptic feedback by providing rich visual clues and reinforcing the user's intentions (all these subtle micro-movements). Based on the prototype, here are a few things to consider when you use this interaction pattern: 1) Perspective visual distortion. It might take more time to recognize visually distorted signs when they are arranged by depth. It's a good idea to use well-known sign sequences where the current selected element gives you a hint about the previous and next ones. A user should have an embedded knowledge of how the elements in the menu are sorted. A number selector might work well with it 2) Low range of hand movement in stationary mode. Because you need to keep visual contact with the UI, the useful range of possible hand movement is very limited. You must position your hand in front of your eyes, maintaining a comfortable distance. It restricts the number of items you can have on the menu. Plus, the trigger distance (30-40mm) also reduces the amount of menu elements you can use. Two or three items work the best (four is max). Download APK [Meta Quest] — #spatialdesign #spatialui #spatialux #vr #unity3d #prototyping #handtracking #interactiondesign #virtualreality #productdesign #xr #xrdesign #xrinteractiontoolkit #xrdevelopment #spatialcomputing
0:24

Explored the finger menu with a depth-based layout. Felt fun. Microinteractions/UI motion helped to compensate for the lack of haptic feedback by providing rich visual clues and reinforcing the user's intentions (all these subtle micro-movements). Based on the prototype, here are a few things to consider when you use this interaction pattern: 1) Perspective visual distortion. It might take more time to recognize visually distorted signs when they are arranged by depth. It's a good idea to use well-known sign sequences where the current selected element gives you a hint about the previous and next ones. A user should have an embedded knowledge of how the elements in the menu are sorted. A number selector might work well with it 2) Low range of hand movement in stationary mode. Because you need to keep visual contact with the UI, the useful range of possible hand movement is very limited. You must position your hand in front of your eyes, maintaining a comfortable distance. It restricts the number of items you can have on the menu. Plus, the trigger distance (30-40mm) also reduces the amount of menu elements you can use. Two or three items work the best (four is max). Download APK [Meta Quest] — #spatialdesign #spatialui #spatialux #vr #unity3d #prototyping #handtracking #interactiondesign #virtualreality #productdesign #xr #xrdesign #xrinteractiontoolkit #xrdevelopment #spatialcomputing

Oleg Frolov

22,202 просмотров • 2 лет назад

Redesigned the previous prototype with visual mechanics inspired by Blender. The circular selection interaction pattern is a great way to enhance accessibility and improve accuracy by restricting item selection to a single dimension and, therefore, reducing noise in user input. For a circular menu, one may describe spatial transformations using polar coordinates components: Radius and Angle. I used Radius to set selection availability and Angle to define the selected item. Plus, this particular pattern allows you to expose the names of the items, which might be helpful in cases when the functions of the items are not self-obvious. Feel free to test the prototype on your own and share your thoughts :) #handtracking #xrinteraction #xri #xr #virtualreality #unity3d #prototyping #spatialcomputing #spatialdesign #spatialui #ux #ui #xrdevelopment #spatialprototype #interactiondesign
0:22

Redesigned the previous prototype with visual mechanics inspired by Blender. The circular selection interaction pattern is a great way to enhance accessibility and improve accuracy by restricting item selection to a single dimension and, therefore, reducing noise in user input. For a circular menu, one may describe spatial transformations using polar coordinates components: Radius and Angle. I used Radius to set selection availability and Angle to define the selected item. Plus, this particular pattern allows you to expose the names of the items, which might be helpful in cases when the functions of the items are not self-obvious. Feel free to test the prototype on your own and share your thoughts :) #handtracking #xrinteraction #xri #xr #virtualreality #unity3d #prototyping #spatialcomputing #spatialdesign #spatialui #ux #ui #xrdevelopment #spatialprototype #interactiondesign

Oleg Frolov

38,202 просмотров • 2 лет назад

Prototyped another iteration of a hand-tracking-based BoxelXR toolbar. This time, it's a circular menu spawned in front of a user once the index pinch is triggered. Notes: 1) It works great with four elements, but it's hard to scale for a higher number; 2) The selection interaction is forgiving for spatial positioning errors. This is because a 3D hand span is interpreted through a 1D angle range (a 1D subspace in the polar coordinate system of 2D projection space, which is a subspace of the 3D space of the experience, respectively). This prevents false positives and makes it easy for users to execute. Precise muscle coordination is not required, either. Design tip: Lower the dimensionality of your interaction to reduce cognitive loads and make it more accessible. 3) Using depth-based animations to show different UI states is a great way to leverage the third dimension and add something interesting to the experience. But be careful. When the UI faces a user's face, the contrast in the motion might be low, so use it only as an additional layer. In this prototype, color coding is the primary way to mark changes in a UI state. ⬇️🐙🐱 Download the prototype — #handtracking #xrinteraction #xri #xr #virtualreality #csharp #unity3d #prototyping #spatialcomputing #spatialdesign #spatialui #ux #ui #xrdevelopment
0:25

Prototyped another iteration of a hand-tracking-based BoxelXR toolbar. This time, it's a circular menu spawned in front of a user once the index pinch is triggered. Notes: 1) It works great with four elements, but it's hard to scale for a higher number; 2) The selection interaction is forgiving for spatial positioning errors. This is because a 3D hand span is interpreted through a 1D angle range (a 1D subspace in the polar coordinate system of 2D projection space, which is a subspace of the 3D space of the experience, respectively). This prevents false positives and makes it easy for users to execute. Precise muscle coordination is not required, either. Design tip: Lower the dimensionality of your interaction to reduce cognitive loads and make it more accessible. 3) Using depth-based animations to show different UI states is a great way to leverage the third dimension and add something interesting to the experience. But be careful. When the UI faces a user's face, the contrast in the motion might be low, so use it only as an additional layer. In this prototype, color coding is the primary way to mark changes in a UI state. ⬇️🐙🐱 Download the prototype — #handtracking #xrinteraction #xri #xr #virtualreality #csharp #unity3d #prototyping #spatialcomputing #spatialdesign #spatialui #ux #ui #xrdevelopment

Oleg Frolov

11,746 просмотров • 2 лет назад

Started experimenting with hand-tracking-based interactions for my little voxel editor – BoxelXR. There are a lot of things to tune, but overall it feels good. Though it would require some decent work in sound design to compensate for the lack of haptic feedback.
0:36

Started experimenting with hand-tracking-based interactions for my little voxel editor – BoxelXR. There are a lot of things to tune, but overall it feels good. Though it would require some decent work in sound design to compensate for the lack of haptic feedback.

Oleg Frolov

915,837 просмотров • 9 месяцев назад

Made another prototype to explore how you can create a voxel with hand-tracking. When you push the main voxel, it generates “wireframe” voxels along its path. The interaction itself feels fun and effortless, but it requires generated voxels to be in a wireframe or ghost mode to avoid overlapping with your hand. Tools: XCode + RealityKit + SwiftUI Device: Apple Vision Pro
0:37

Made another prototype to explore how you can create a voxel with hand-tracking. When you push the main voxel, it generates “wireframe” voxels along its path. The interaction itself feels fun and effortless, but it requires generated voxels to be in a wireframe or ghost mode to avoid overlapping with your hand. Tools: XCode + RealityKit + SwiftUI Device: Apple Vision Pro

Oleg Frolov

51,552 просмотров • 7 месяцев назад

#VisionPro pushes for far interactions but is that the best way? With gaze and pinch, #Apple seems to discourage developers from implementing more direct hand interactions into their experiences. There are certainly some advantages but I thought the whole purpose of #VR and #spatialcomputing was to be immersed and feel like you are interacting with #digital objects as if they were real. I am even more convinced after testing this SDK from SpectreXR that takes advantage of the Nvidia physics engine to make hand interactions feel more natural. Would you use more direct interactions like in the video or stick with far gaze+pinch?
0:42

#VisionPro pushes for far interactions but is that the best way? With gaze and pinch, #Apple seems to discourage developers from implementing more direct hand interactions into their experiences. There are certainly some advantages but I thought the whole purpose of #VR and #spatialcomputing was to be immersed and feel like you are interacting with #digital objects as if they were real. I am even more convinced after testing this SDK from SpectreXR that takes advantage of the Nvidia physics engine to make hand interactions feel more natural. Would you use more direct interactions like in the video or stick with far gaze+pinch?

Gabriele Romagnoli

104,043 просмотров • 2 лет назад

🦯 How to help visually impaired people easily navigate through everyday life with the Apple Vision Pro? I know it's not the expected use case of an XR headset, but you can do it with real-time meshing, hand tracking, and spatial audio.
1:09

🦯 How to help visually impaired people easily navigate through everyday life with the Apple Vision Pro? I know it's not the expected use case of an XR headset, but you can do it with real-time meshing, hand tracking, and spatial audio.

Stijn Spanhove

21,819 просмотров • 2 лет назад

⛳️ Tiny Golf - A Very Cool Meta Horizon Start Developer Competition Submission! Mixed Reality games are challenging because they must adapt to real-world spaces with dynamic content, but they don’t have to be complex. Tiny Golf is a great example, using a minimalistic aesthetic and 🙌 hand-tracking pinch-and-pull gestures to launch the golf ball. I hope MR/VR games like this inspire many of you to build, there is just so much opportunity with XR today and 2026 is the year to start! 😉 📌 To get started, you can use: Meta XR All-In-One SDK, or a leaner option: Meta XR Interaction SDK, which pulls in the Meta XR Core SDK package and includes advanced hand-tracking features and passthrough.
0:43

⛳️ Tiny Golf - A Very Cool Meta Horizon Start Developer Competition Submission! Mixed Reality games are challenging because they must adapt to real-world spaces with dynamic content, but they don’t have to be complex. Tiny Golf is a great example, using a minimalistic aesthetic and 🙌 hand-tracking pinch-and-pull gestures to launch the golf ball. I hope MR/VR games like this inspire many of you to build, there is just so much opportunity with XR today and 2026 is the year to start! 😉 📌 To get started, you can use: Meta XR All-In-One SDK, or a leaner option: Meta XR Interaction SDK, which pulls in the Meta XR Core SDK package and includes advanced hand-tracking features and passthrough.

Dilmer 👓🔜 GDC

31,749 просмотров • 8 месяцев назад

This batter was hit by a pitch. The problem is, he stuck his hand out. The umpire said he was struck by the pitch and awarded him first base. The coach came out to argue and said he stuck his hand out. The field umpire came in and told the plate umpire he did stick his hand out. But the plate umpire said or meant to say he didn’t do it intentionally but rather defensively. I had this happen to me at 14 years old. Fast ball right at me and reactively I put my hand up to block it. The ump awarded me first base. What do you think? Do you think he stuck his hand out intentionally or to defend himself from being hit?
0:44

This batter was hit by a pitch. The problem is, he stuck his hand out. The umpire said he was struck by the pitch and awarded him first base. The coach came out to argue and said he stuck his hand out. The field umpire came in and told the plate umpire he did stick his hand out. But the plate umpire said or meant to say he didn’t do it intentionally but rather defensively. I had this happen to me at 14 years old. Fast ball right at me and reactively I put my hand up to block it. The ump awarded me first base. What do you think? Do you think he stuck his hand out intentionally or to defend himself from being hit?

👉M-Û-R-Č-H👈

131,011 просмотров • 1 месяц назад

.Brett Adcock explains why Figure pivoted away from designing their humanoid robot's hands after human hands: "This is extremely important. If we want to solve AGI and get to human intelligence in the physical world—it's all going to start with the hands." "Our first-generation hand is a tendon-driven hand." "The rationale here is that a human hand is like this: most of our motors are in our forearm and little tendons drive all the fingers." "It ended up becoming the wrong engineering choice." "We recently unveiled our high degree of freedom, next-generation hand a month ago. This hand has human-level dexterity, as many joints in the hand as a human."
3:14

.Brett Adcock explains why Figure pivoted away from designing their humanoid robot's hands after human hands: "This is extremely important. If we want to solve AGI and get to human intelligence in the physical world—it's all going to start with the hands." "Our first-generation hand is a tendon-driven hand." "The rationale here is that a human hand is like this: most of our motors are in our forearm and little tendons drive all the fingers." "It ended up becoming the wrong engineering choice." "We recently unveiled our high degree of freedom, next-generation hand a month ago. This hand has human-level dexterity, as many joints in the hand as a human."

sourcery

323,480 просмотров • 20 дней назад

Here is another haptic feedback experiment with hand tracking. This time there are more boxes! Or cuboids, for greater scientific rigor. I tried using different haptic patterns for different hover effects — it worked very well. Really, haptics are a missing piece in the hand-tracking experience to make it feel solid. Tools: Unity3D + C# MR Device: Quest3 Haptic device: HapticLabs's Prototyping Kit Music: PO-12 and Orchid
1:04

Here is another haptic feedback experiment with hand tracking. This time there are more boxes! Or cuboids, for greater scientific rigor. I tried using different haptic patterns for different hover effects — it worked very well. Really, haptics are a missing piece in the hand-tracking experience to make it feel solid. Tools: Unity3D + C# MR Device: Quest3 Haptic device: HapticLabs's Prototyping Kit Music: PO-12 and Orchid

Oleg Frolov

50,326 просмотров • 8 месяцев назад

Hands are as important as the brain. From Shenzhen, Stella Robot has unveiled the PANTHEON HAND 22:a new generation of bionic dexterous hand that combines 22 degrees of freedom with remarkable strength and finesse. It’s a hand that truly understands precision. It can lift 30 kilograms, yet is delicate enough to thread a needle.Each finger can bear 1.5 kilograms, and the entire hand can gently hold a piece of tofu or sense the subtle expansion of a balloon. Powered by a tendon-driven mechanism with integrated actuators inside the forearm, it achieves compact design and natural, human-like movement. From precision assembly and component handling to material transport and human-robot collaboration,the PANTHEON HAND 22 shows what it means for humanoid robots to truly work with their hands.
0:53

Hands are as important as the brain. From Shenzhen, Stella Robot has unveiled the PANTHEON HAND 22:a new generation of bionic dexterous hand that combines 22 degrees of freedom with remarkable strength and finesse. It’s a hand that truly understands precision. It can lift 30 kilograms, yet is delicate enough to thread a needle.Each finger can bear 1.5 kilograms, and the entire hand can gently hold a piece of tofu or sense the subtle expansion of a balloon. Powered by a tendon-driven mechanism with integrated actuators inside the forearm, it achieves compact design and natural, human-like movement. From precision assembly and component handling to material transport and human-robot collaboration,the PANTHEON HAND 22 shows what it means for humanoid robots to truly work with their hands.

CyberRobo

26,867 просмотров • 10 месяцев назад

MIT engineers built an AI wristband that controls robots by reading your hand muscles. It works by using an ultrasound to capture images of the muscles and tendons in your wrist. An AI algorithm then translates those images into the exact position of all 5 fingers, tracking 22 different ways your hand can bend, twist, and extend. In demos, wearers directed a robotic hand to play piano just by moving their own fingers. To put the complexity in perspective, it takes: > 34 muscles > 27 joints > 100+ tendons ...Just to scroll your phone This device captures it all from a band the size of a smartwatch. They tested across 8 volunteers with different hand sizes, tracking everything from sign language letters to holding scissors. The team is now collecting data from hundreds more users to train robots in surgical procedures. Something like this could one day become the universal remote for robots and virtual worlds, and we're way closer than most people expect.
2:40

MIT engineers built an AI wristband that controls robots by reading your hand muscles. It works by using an ultrasound to capture images of the muscles and tendons in your wrist. An AI algorithm then translates those images into the exact position of all 5 fingers, tracking 22 different ways your hand can bend, twist, and extend. In demos, wearers directed a robotic hand to play piano just by moving their own fingers. To put the complexity in perspective, it takes: > 34 muscles > 27 joints > 100+ tendons ...Just to scroll your phone This device captures it all from a band the size of a smartwatch. They tested across 8 volunteers with different hand sizes, tracking everything from sign language letters to holding scissors. The team is now collecting data from hundreds more users to train robots in surgical procedures. Something like this could one day become the universal remote for robots and virtual worlds, and we're way closer than most people expect.

Rowan Cheung

27,999 просмотров • 4 месяцев назад

Can you name the 5 alien limb syndromes? This video shows one of the most famous: inter-manual conflict. Key points: - Did you know it is important to localize where the pathology is likely within the brain and whether you have a disconnection syndrome; anterior vs. posterior and involving or not involving the corpus callosum, parietal or frontal regions - I consider 5 categories though many papers refer to 4; the 5th in a movement disorders clinic is actually the most common; levitating. - 1) diagnostic dyspraxia/inter-manual conflict; one hand performs actions contrary to the other hand - 2) alien hand sign where there is a true feeling that the hand does not belong to the person - 3) Anarchic hand; performs goal-directed activity not desired or under the willfulness of the person - 4) supernumerary hand which is a person feeling like they have an extra limb - 5) levitating arm or leg; levitates without volitional action. Hint on this last one to consider corticobasal syndrome or PSP.
3:01

Can you name the 5 alien limb syndromes? This video shows one of the most famous: inter-manual conflict. Key points: - Did you know it is important to localize where the pathology is likely within the brain and whether you have a disconnection syndrome; anterior vs. posterior and involving or not involving the corpus callosum, parietal or frontal regions - I consider 5 categories though many papers refer to 4; the 5th in a movement disorders clinic is actually the most common; levitating. - 1) diagnostic dyspraxia/inter-manual conflict; one hand performs actions contrary to the other hand - 2) alien hand sign where there is a true feeling that the hand does not belong to the person - 3) Anarchic hand; performs goal-directed activity not desired or under the willfulness of the person - 4) supernumerary hand which is a person feeling like they have an extra limb - 5) levitating arm or leg; levitates without volitional action. Hint on this last one to consider corticobasal syndrome or PSP.

Michael Okun

24,193 просмотров • 2 лет назад

This is one of the simplest ways to be more consistent at the plate. “Eyes behind the ball.” That’s it. - Track it longer. - Let it travel. - Stop trying to hit it out of the pitchers hand and let it come to you. > Save this for when you need it. > Send it to a hitter who does.
1:39

This is one of the simplest ways to be more consistent at the plate. “Eyes behind the ball.” That’s it. - Track it longer. - Let it travel. - Stop trying to hit it out of the pitchers hand and let it come to you. > Save this for when you need it. > Send it to a hitter who does.

Trey Hannam

59,662 просмотров • 3 месяцев назад

📣 This solo developer who currently works at Pixar as a Technical Director is building a VR rhythm game called Sock Puppet Superstar for Quest as an indie project, and over the last week he has been going viral on TikTok! 🤖Features/Tech: - Using Unity, OpenXR, hand tracking, and a custom voice synthesizer built in FMOD - Raise and lower your puppet to control the pitch - Works with hand tracking or controllers - Customize your sock puppet with colors, fabrics, accessories, and voices 📌 Devs can achieve this using the Meta XR Interaction SDK or Unity XR Toolkit (XR Hands) to add full hand support like what’s shown here. 🚀 Share this to give this solo indie dev more visibility! I’ll continue this bi-weekly with more indie devs who deserve the amplification!
0:35

📣 This solo developer who currently works at Pixar as a Technical Director is building a VR rhythm game called Sock Puppet Superstar for Quest as an indie project, and over the last week he has been going viral on TikTok! 🤖Features/Tech: - Using Unity, OpenXR, hand tracking, and a custom voice synthesizer built in FMOD - Raise and lower your puppet to control the pitch - Works with hand tracking or controllers - Customize your sock puppet with colors, fabrics, accessories, and voices 📌 Devs can achieve this using the Meta XR Interaction SDK or Unity XR Toolkit (XR Hands) to add full hand support like what’s shown here. 🚀 Share this to give this solo indie dev more visibility! I’ll continue this bi-weekly with more indie devs who deserve the amplification!

Dilmer

151,220 просмотров • 4 месяцев назад

What happens when you change the hand mid-task? We tested this by modifying the hands mid-rollout and letting the same model keep running. It perceives the new tool, and finds a new trajectory and contact strategy to complete the task. This works because training on mixed data forces the model to condition its behavior on the hand in front of it. This brings it closer to a general understanding of how shapes and contact surfaces interact with the physical world – and which actuation strategy should follow.
1:18

What happens when you change the hand mid-task? We tested this by modifying the hands mid-rollout and letting the same model keep running. It perceives the new tool, and finds a new trajectory and contact strategy to complete the task. This works because training on mixed data forces the model to condition its behavior on the hand in front of it. This brings it closer to a general understanding of how shapes and contact surfaces interact with the physical world – and which actuation strategy should follow.

Generalist

21,929 просмотров • 1 месяц назад

SHE CHARGES $7500 FOR A SITE YOU CONTROL BY WAVING AT YOUR WEBCAM no mouse, no scroll, no clicking anything, the camera tracks your hand and the whole site responds as you move it through the air how she built it: render the base clip in seedance 2 open replit and attach your design reference tell it to wire hand tracking with mediapipe map the hand position to the video in a single prompt still in the same chat, switch it to slide mode or a scroll transform whenever you want she never wrote one line of the tracking code, replit stood up the entire gesture layer from that one instruction meanwhile your local dev still thinks interactive means a button that lights up when the cursor grazes it the exact mediapipe prompt is in the guide below, save it before this becomes the standard
0:33

SHE CHARGES $7500 FOR A SITE YOU CONTROL BY WAVING AT YOUR WEBCAM no mouse, no scroll, no clicking anything, the camera tracks your hand and the whole site responds as you move it through the air how she built it: render the base clip in seedance 2 open replit and attach your design reference tell it to wire hand tracking with mediapipe map the hand position to the video in a single prompt still in the same chat, switch it to slide mode or a scroll transform whenever you want she never wrote one line of the tracking code, replit stood up the entire gesture layer from that one instruction meanwhile your local dev still thinks interactive means a button that lights up when the cursor grazes it the exact mediapipe prompt is in the guide below, save it before this becomes the standard

Bounce

59,981 просмотров • 1 месяц назад

J. Cole speaks on why he’s doing Trunk Sale Tour. 💿 “Hand to hand CD sales, hand to hand interaction, talking to people like how it used to be when we’d drop a mixtape… now that I’m doing it, it’s confirming, thank god I did it. What if I never drop an album again? I’m 50–60 years old, I’m far removed from all of this.. I get to actually see people face to face, look in their eyes, and see the impact.”
0:48

J. Cole speaks on why he’s doing Trunk Sale Tour. 💿 “Hand to hand CD sales, hand to hand interaction, talking to people like how it used to be when we’d drop a mixtape… now that I’m doing it, it’s confirming, thank god I did it. What if I never drop an album again? I’m 50–60 years old, I’m far removed from all of this.. I get to actually see people face to face, look in their eyes, and see the impact.”

Cole World

122,296 просмотров • 6 месяцев назад