Why positional tracking for mobile virtual reality is so damn hard
AMSTERDAM — Consumers haven’t even had the chance to buy some of the most promising virtual reality devices yet, but the industry is already looking toward a future where all VR is mobile. But if we’re going to get to that point, the industry will need to solve a few key problems.
ARM, the hardware-architecture company responsible for the design of most of the processors inside your smartphone, is working with companies like Oculus VR to deliver efficient sensor data for positional tracking in mobile virtual reality. The goal is to have a device like the Gear VR (which combines the power of a smartphone with specialized VR optical equipment) that can precisely track where you stand in 3D space. Having position-tracking VR available to anyone who owns a competent smartphone would drastically reduce the barrier-to-entry for stunning simulation experiences as well as augmented reality. That’s key to unlocking the potential of a market that could generate $120 billion in revenue by 2020, according to tech advisor Digi-Capital.
But today, solutions like Gear VR have no idea where you’re standing. They can only tell when you twist your head. PC devices, like the $800 HTC Vive, can track your position, but they use external beacons as guideposts. That’s an option for mobile VR, but Oculus chief technology officer John Carmack has said that he is working on creating a way for smartphones to track your position from the inside out. This means that the phone will do all of the processing.
Why this isn’t possible today
To figure out what is holding mobile VR from reaching that point, GamesBeat caught up with ARM developer relations manager Nizar Romdan at the Casual Connect Europe event in Amsterdam last week.
Romdan explained that one of the biggest hurdles is that the sensor chips that would make it easy to track position don’t have a lot of other uses.
“The mobile device is limited by its form factor,” he said. “It’s a five-inch screen and a kinda snug casing. How many sensors can handset makers try to fit in? Everyone wants a gyro, but manufacturers could put in lots of other sensors. However, they won’t do that for a device for the mainstream. It’s not something that everyone really uses. So they will, by default, not put those kinds of sensors in unless it becomes a must.”
Romdan says that the industry is starting to think that VR could make those sensors a “must.” Those extra sensors may start finding their way into next-gen iPhones and Androids as Apple, Samsung, and LG try to figure out how to approach this space.
But even more sensors won’t work
We can build a positional-tracking headset today. Microsoft and Magic Leap have teased prototypes of their respective AR systems, and they are self-contained head-mounted displays that know where you are in relation to the world around you. But we know very little about Microsoft’s HoloLens, which doesn’t have anything close to a release date or price. We know even less about Magic Leap. But those devices were built from the ground up for this kind of functionality, and they are a ways off from reaching a consumer-ready status.
When we’re talking about putting a mobile phone into a VR casing, the challenges compound because even if you add powerful sensors, you’ll run into other limitations.
“The more sensors you have, the more information you have coming into the processors,” said Romdan. “But you still have the same power budget. It takes roughly 3 watts to run a smartphone — maybe 5 watts if it’s a bigger one like the Note. Maybe you’ll rarely get up to 10 watts. But we are not in the 300-to-600-watt level of a PC. So you have to process all the information while your power budget is still the same.”
Phones are not specialized for niche-use cases. They need to sip a tiny amount of power at all times because people switch rapidly from using them for work, then for gaming, and then for exercising, and then for anything else. If the sensors solve one problem but need 100 watts of energy, then you’re only swapping out one problem for another.
A combination of solutions
Despite the difficulties, Carmack, who is one of Oculus’s biggest brains, is working on this. He has said that he is spending “most of his time” on getting positional tracking working with Gear VR. And, considering his history as a hypertalented software engineer, it seems likely that he will look for a hybrid answer that combines the best new hardware with new software algorithms that can make better determinations from the same old data.
Romdan said as much when we asked him how to overcome these obstacles.
“You need to find clever techniques,” he said. “It’s more challenging. But, at ARM, we think challenge is good. It pushes you to think and find a solution and innovate. We’d rather be in a really tough use case than to be in an easier one where we can relax.”
The Casual Connect organization paid for my trip to the event. Our coverage remains fair.
