New software could give lip-reading powers to everyone

Giving everyone the ability to comprehend one another might soon be much easier.

A team from the University of Oxford's Department of Computer Science has developed new lip-reading software, LipNet, which they claim is the most accurate of its kind to date by a wide margin. 

SEE ALSO: It's not just you: Siri is getting smarter

The development of the software, which was supported in part by Alphabet's DeepMind AI program, has been detailed in a paper which reports LipNet has bested the existing top marks in lipreading tech accuracy by 13.8 percent. The previous best software and its 79.6 percent mark was already light-years ahead of the efforts of human lip-readers, who averaged 52.3 percent accuracy with the same test.  

Counter to practical logic, the breakthrough is actually in part thanks to a less refined approach to the task — at least in terms of scale. The Oxford team expanded their focus from a speaker's individual words, which every previous system had used, to the larger constructions on the sentence level.

According to the paper, "All existing [lip-reading approaches] perform only word classification, not sentence-level sequence prediction.... To the best of our knowledge, LipNet is the first lip-reading model to operate at sentence-level."

In other words, the software became more effective as it moved closer to the way the human brain best processes this type of visual data. It takes the video of a speaker and instead of honing in on each and every word as a distinct entity, its deep-learning predictive capabilities allow it to place them within a larger context for greater understanding (you can see it in action in the video above).   

A member of the team, Oxford Professor and Google DeepMind scientist Nando de Freitas, has taken to social media to give the general public more context than they might have been able to find in the cut-and-dry jargon of the paper.

First, he clarified that the software has not yet been put to task beyond the baseline test and needs further development: 

More hopefully, he hinted at the great potential LipNet has for practical use: 

Most importantly, this heightened level of accuracy opens up new possibilities. For those who depend on sign language and, to a lesser degree, lip-reading, communication can be extremely challenging.

There are also clear benefits for people in general: Reading lips could potentially become something anyone with a smartphone could do, and voice command systems may become even more accurate with the application of software like LipNet.