Apple Patents Gesture, Expression Tracking for Mixed Reality Headset
Click here to read the full article.
Apple’s upcoming mixed reality headset could include a number of sensors to track the eyes, gestures and even facial expressions of its users. The company applied for a patent to track these kinds of inputs, and combine them with information gathered from outward-facing sensors for mixed reality experiences.
The patent application in question, simply titled “Display System Having Sensors,” was first filed in March of this year, and published last week. It describes in detail plans to use a range of sensors to gather data from the wearer of a mixed reality headset.
More from Variety
Apple Design Guru Jony Ive to Depart to Form Independent Firm
Apple Music Tops 60 Million Subscribers, Including Users on Free Trials
Such sensors would make it possible for Apple to more realistically reproduce a user’s facial expression in mixed reality. Apple has already developed facial tracking software for Animoji, the company’s animated AR emoji. Animoji make use of an iPhone’s selfie camera to track facial expressions, and then translate those movements to animation.
The challenge with that approach is that you can’t simply film a user’s face if they’re wearing a headset. That’s why Apple is looking to combine data from separate sensors, including some used for eyebrow and jaw tracking, as well as eye tracking cameras. The latter could also be used for biometric authentication, as the patent application notes. The company also may use cameras for gesture tracking, according to the patent application.
Apple has been working on its own headset for a couple of years now. Cnet reported last year that this headset would combine augmented and virtual reality, meaning that it would overlay virtual objects and worlds over a view of the real world.
The patent application filed last week further outlines how the company is likely going to do that: by capturing the real world with outward-facing cameras, and then displaying these images on a display — an approach that’s very different from the way Microsoft’s and Magic Leap’s augmented reality headsets work. From the patent application:
“In some embodiments, the world sensors may include one or more “video see through” cameras (e.g., RGB (visible light) video cameras) that capture high-quality video of the user’s environment that may be used to provide the user with a virtual view of their real environment.”
Apple has yet to publicly comment on its headset plans, including when it intends to ship any such product. In March, Apple analyst Ming-Chi Kuo estimated that the company may start to produce its headset by Q4 of this year, and then publicly introduce it in 2020.
Sign up for Variety’s Newsletter. For the latest news, follow us on Facebook, Twitter, and Instagram.
