Providing access to expertise in the school of Computing Sciences.


Fighting Crime : Computerised Lip-Reading

Business Challenge

Lip-reading has been used in evidence in court but its use has been controversial because human lip-readers are prone to human errors which cannot easily be quantified. Nevertheless there are situations, in particularly noisy environments for example, when lip motions might be the only evidence available so it is important to understand how speakers move their lips, whether they do so consistently, and whether artificially intelligent computers can be built to recognise visual speech.

Our Solution

A group of our computing experts, led by Dr Richard Harvey, are currently developing a computer lip-reading system which will convert videos of conversations into written transcripts. Researchers have identified that each speech sound has a particular facial and mouth position, which can be mapped on video and then processed by software that associates the various orientations with the sounds they produce.

The technology focuses on mapping the facial movements, called 'features', and converting them into signals which the software can process and 'translate' into text.

The University are teaming up with Surrey University, who have built accurate and reliable face and lip trackers. Combining the two technologies will create an innovative and ground-breaking piece of equipment that will be extremely valuable in the fight against crime.

A significant research breakthrough was reported in April 2009. See the Lip-Reading Update study.

Business Outcome

When this technology is finalised, it will be extremely beneficial to law enforcers collecting evidence to convict criminals and terrorists.

Dr Richard Harvey said: "There are often situations where the police cannot get close enough to hear what a suspected criminal is saying. In such situations, a camera could see the lips and a computer might be able to lip-read. To be effective the systems must accurately track the head over a variety of poses, extract numbers, or features, that describe the lips and then learn what features correspond to what text. To tackle the problem we will need to use information collected from audio speech. So this project will also investigate how to use the extensive information known about audio speech to recognise visual speech."

More Detail

For further information download the full Computerised Lip-Reading Case Study and the the Lip-Reading Update from April 2009.