Using Presence Platform’s upgraded Hand Tracking API, we introduced Hand Tracking with our most recent update to Myst on the Meta Quest Platform, titled ‘Hands & More’. We’re super excited to finally let folks play Myst on Quest without physical controllers! In this post, we’ll discuss the evolution and iteration of implementing hand tracking in Myst—and in particular, adding more support for it in Unreal Engine 4.27.2.
Guest Article by Hannah Gamiel
Hannah Gamiel is the Development Director at Cyan—the studio behind the original ‘Myst’ games—and helped develop the new ‘Myst (2020)‘ which includes VR support. Originally coming from a purely technical background, she now helps lead production on all titles and manages business & tech efforts at Cyan. She has worked on titles such as ‘Myst’ (2020), ‘The Witness’, ‘Braid, Anniversary Edition’, ‘Obduction’, ‘Firmament’ (coming soon!), and more.
Design Phase & Considerations
Designing Navigation for Hand Tracking
Picture indicating where you’d like to go. You likely thought of pointing, right? That’s why we opted to use a ‘pointing’ method for movement in Myst.
When you’re in teleport mode, you can point to where you’d like to go and the teleport ring appears at your destination. When you ‘un-point’ (by extending the rest of your fingers, or simply pulling your pointer finger back into your palm), the teleport is executed.
When you’re in smooth movement mode, pointing with your free-movement-dominant hand (which can be configured in our controls settings, but is the Left hand by default) will begin smoothly moving you around in the direction you’re pointing.
[GIF no longer available]
When playtesting movement with pointing, we found that hand tracking can sometimes be unreliable with your pointer finger and middle finger when it is occluded by the rest of your hand. The system isn’t sure whether those fingers are fully pointing or fully ‘enclosed’ in your hand. We added a bit of a ‘fudge’ factor to the code to account for more stable movement initiation/execution on this front—which we’ll go into a bit later when we discuss changes made to out-of-the-box Hand Tracking support in Unreal Engine.
Turning
The ‘point’ method doesn’t work for all navigation usages. When it comes to turning, we initially combined pointing with wrist rotation. Comparing the player’s wrist and the camera’s forward vector would indicate the direction of the turn (and how big the turn should be).We tried this initially since it seemed intuitive to keep the ‘pointing’ theme for navigation going between all modes.
Complications arose in comfort tests, however. In playtesting, most players would point forward with their palm facing the ground, as one likely would when attempting to point at something outside of a game as well. Rotating your wrist to the left and right (around the up axis of your wrist) while you have your palm facing the ground is challenging and has a very limited range of motion, especially if trying to turn away from your chest.
This issue is the same even if you asked a player to point at something in front of them with their palms facing inward. You can bend your wrist in toward your body quite a bit, but you won’t get the same range of motion bending your wrist away from your body.
So how did we solve this? We ended up assigning turning to a ‘thumbs-up’ gesture instead of a pointer-finger-pointing gesture.
Imagine giving a thumbs-up. Now turn your wrist right and left. Note that even though you don’t have a huge range of motion it’s still fairly consistent pointing either ‘left’ and ‘right’ with your thumb in this gesture.
This is what we settled on for turning in hand tracking mode. Although pointing with your thumb doesn’t seem like the most intuitive way to turn, it did end up being the most comfortable and consistent way of doing so.
With snap turning, rotating your wrist to the left or right from a thumbs-up position causes a single snap turn to initiate. You then have to return your hand to the ‘center’ (straight up) position in order to reset the snap, and additionally wait for a very short cooldown to occur to initiate a snap turn again.
With smooth turning, turning your wrist while in a thumbs-up position will begin rotating you left or right—once you leave a ‘dead zone’ that prevents a turn from occurring until you pass the threshold.
Handling Conflicts Between Movement & Object Interaction Poses
Of course, pointing a finger is too broad of a gesture to be assumed to only just be used for navigation. People will make the same pointing gesture to press buttons or interact with other things in the world just out of habit or their own expectation. It would be pretty jarring to walk up to (but not right up to) a button, point your finger to press it, and then suddenly (and unwantedly) move closer to it in-game (or initiate a teleport unintentionally)!
The way we prevent movement from occuring while the player may be interacting with something is by preventing any movement code from firing off when the hand making the ‘move’ gesture is within a certain range of an interactable object. This range has been tweaked multiple times to get to a good ‘sweet spot’ based on playtesting.
We’ve found that this sweet spot is around 25 cm from the world space location of the bone of the tip of the index finger. Myst is full of interactive objects of various sizes (everything from small buttons to very large levers) arrayed in both wide-open spaces and narrow hallways, so it took us quite a bit of testing to settle on this number. We initially tried 60 cm (about two feet), but that prevented movement from occurring when players still needed to get closer to an object. Likewise, anything below 25 cm caused undesired player movement to trigger when players were trying to grab or touch an object.
One of our best testing areas was the generator room on Myst Island, where you make your way through a narrow entryway and are then immediately greeted by a panel full of buttons. When the interaction testing area was too large, players were unable to move through the entry and toward the panel because it detected buttons within range of the index finger.
That said, 25 cm is what worked specifically for Myst. Other games may need to adjust this number if they want to implement something similar, with their own criteria in mind.
Designing Object Interactions for Hand Tracking
Right now, all grabbable interactions in Myst are built to work with hand tracking—turning valves, opening doors, pressing buttons, turning book pages, and so on.
The interactions piggy-back off what we had already set up for Myst with Touch controllers. There, pressing the grip button automatically blends the in-game mesh representation of your hand into a ‘grabbed’ pose, either putting your hand into a fist (if empty) or grabbing an object. With hand tracking, we’ve added code that will make a qualified guess as to when you have curled your fingers enough to ‘grab’ something and initiate the same logic as mentioned before.
For example, when you’re using hand tracking and your hand hovers over something that’s grabbable, your hand color turns orange (this is exactly what happens when you don’t use hand tracking in Myst VR as well). When you grab an interactable object by beginning to curl your fingers into a fist, an orange sphere replaces your hand mesh and represents where the hand is attached to the object.
The reason why we went with this method instead of making custom poseable meshes for your hands—or allowing your hands/fingers to appear to physically interact with portions of these objects—is because we wanted the interactions to be at parity with what we offer on the Touch controller side for now.
Pushing buttons works differently though. There’s no need for abstraction since buttons aren’t grabbable objects, and instead we allow you to simply push a button using generated capsule colliders between each of the finger joints on the poseable hand mesh. You can do all sorts of weird and fun things because of this—like using only your pinky or the knuckle of your ring finger to interact with every button in the game, if you really want to.
This implementation differs slightly from the way Touch controllers interact with buttons in the game in that we usually expect players to use the grip button on their controller to set the hand to be a posed ‘finger pointing’ mesh to get an accurate in-game button press on their end. With hand tracking, there’s obviously significantly more flexibility in the pose you can create with your hand, and therefore significantly more ways to press buttons with the same level of accuracy.
Menu/UI Interactions
For interacting with menus, we ended up going with the same interaction paradigm that Meta uses for the Quest Platform: a two-finger pinch between thumb and index finger, on either hand. This can be used both to open our in-game menu and interact with elements in the menu. No sense in reinventing the wheel here when players are already taught to do this in the OS-level menus when they first enable hand tracking on Quest!
Communicating All of This to the Player
Because hand tracking isn’t as common an input on Quest as Touch controllers, and because there may be some people playing Myst for the very first time (or even playing their very first VR game!), we tried to be considerate with how we communicate all of this information about hand tracking to the player. We made sure to include another version of our “controller diagram” specifically tailored to describe hand tracking interactions (when enabled in Myst), and we show the player specialized notifications that tell them exactly how to move around with their hands.
Additionally, we thought it would be vital to remind the player how to have a smooth hand tracking experience, once enabled. The player is notified in Myst’s menu that hand tracking stability is much better if they ensure they’re in a well-lit room and keep their hands within their field of view.
Meta also informs players that these are key to a well-tracked hand tracking environment, but we recognize that some players might jump into a game not having parsed Meta’s notices about this first, so we’ve chosen to remind folks in case they forgot.