We have been using the computer mouse for decades to interact with our technology. Touchscreens brought us a new way to input commands to our gadgets, but they rely on the same fundamental idea of the click. Even the new 3D touch on the iPhone 7 is just an incredibly sophisticated way of using the hand to answer a yes or no question.
Both Michael Buckwald, CEO of Leap Motion and Jim Marggraff, CEO of Eyefluence agree that the future of human computer interaction will be multimodal. Marggraff is doing to the eye what Steve Jobs did to the finger with the mouse. His company builds eye-tracking technology for AR and VR. Buckwald takes it a slightly different direction by leveraging hand motion as a communication tool.
“Everyone now has what would have been a super computer in their pockets 15 years ago,” added Buckwald. “But if you compare that to the ways that we actually use these devices, it’s still essentially binary.”
Humans have a natural desire for communication to be a bidirectional process. We crave haptic feedback and real physical plastic buttons for a reason. However, when we uncover new methods of communication, we indirectly find ourselves with new and unfamiliar feedback. Sometimes this feedback is uncomfortable, like the nausea some of us feel after spending too much time in VR realities, but sometimes it can be beautiful.
“Seventy-80 percent actually report that their brain is telling them they have gotten some haptic feedback,” said Buckwald, when addressing phantom sensations from users of Leap Motion’s Orion.
This is fun for folks looking to escape their nine to five grind to Everest Base Camp, but it becomes incredibly valuable for others with amputations that suffer from phantom pain. VR has served to be a valuable tool in helping these people connect their missing limbs to a brain that is certain they are still present.
Even worse, those with locked-in syndrome cannot communicate easily with the outside world. Communication becomes an extensive and tiring process and many struggle to keep up with the rapid-fire speech most of us take for granted.
“The general notion of pointing at things to navigate through a menu can be done in tens of milliseconds with your eyes versus the time it takes you to use your hands,” added Marggraff.
Just as some people like to talk with metaphors while others leverage humor, human-computer interact won’t play out as a game of winner take all.
“If you want to grab a virtual object and hold it and move it around and look at it from different sides and then you would like to look at that object and find out the volume or maybe change its color and morph it in some way, well you could start with your hands and continue with your eyes,” continued Marggraff.
What we are certain of, is that no matter the input mechanism, communication must be seamless, without delay or interruption. The natural and the realistic go hand in hand with the quality of the experience. Not only will worlds seem more immersive, they will also be less likely to make us puke.
This can have implications for everyone. One day our children will be able to grow up in a world where they are capable of playing with friends on other continents through VR. This is the same hypothesis being explored by companies like AltspaceVR, with entire social networks built around shared experiences.
The availability of content remains a serious limitation in the growth of VR as an entertainment platform. An “iPhone moment” can happen, but only with the propagation of multimodal human computer interaction.