I cannot imagine that an automated lip reading system would be any easier to develop than audio speech recognition. Just as audio speech recognition has detrimental factors like accent, style (fast talking, running words together, etc.), head colds, and sobriety, mechanical speech recognition has to deal with mouth and lip size, whether or not the speaker is buck-toothed (I’m serious here!), or whether or not the speaker just came from the dentist.

Anyone who has seen those old videos of Hitler knows that, at least while making speeches, Hitler was very animated with his mouth and facial expressions, and he opened his mouth very wide. That has to be a significant factor.

I have a friend who is a pretty good lip reader, but she once told another friend that she could not read his lips because, as she put it, when he spoke it looked like he was eating sunflower seeds. That has to be fairly common.

Secondly, does anyone who is well versed in the subject know if anyone is researching whether or not something like Dragon Naturally Speaking can be trained by the profoundly deaf? I would think that if the speaker is consistent, the software could be trained to recognize words even if they are not intelligible to the human ear.

Has anyone seen that awful show on TV with Flava-Flave (spelling?), who wears a clock around his neck for fashion? I imagine a stripped-down laptop (no sound card, minimum video capability, no cd-rom, etc.) with no hinge, having a keyboard on the back of the screen. The user could wear it or carry it. When the user speaks, the trained computer would recognize the sounds even though the audience cannot, and the words would magically appear on the screen. Or to make it smaller and lighter, eliminate the screen and use a voice synthesizer. The keyboard would be needed for unusual words or proper names.

As an added benefit, this might blur the lines (and the animosity) over oralism.

I also envision BlackBerries eventually being able to do this. Imagine a profoundly deaf person walking into a convenience store, and saying “ic curk eek” into his/her trained BlackBerry and “I am deaf. Pack of camel filters, please,” coming out of the BlackBerry.

I dunno what to tell you. You’re the expert on this, not me! So I’ll trust your judgment on this, really.

The video is not captioned, so I have to wait for a hearing friend to help me out.

One thing that I have been able to piece together so far is that Michael Brooke is being quoted, and he *does* have a bunch of publications on ALR and has specifically investigated the possibility of remapping lip movements from video onto a 3D face model that is purported to be lip-readable. This kind of meshes with the bits of the demonstration that are shown in the video. It could be that this was one of the enhancing methods used, with bits and pieces of ALR, with the deaf lipreading experts as a backup and arbiters.

I highly doubt that Frank himself developed the ALR technology. There is nothing to indicate that he has the expertise in this specific area. It is more likely that he was provided with it, and either used it as an application, or he developed the user interface.

But these things do not add up:

- There are no recent scientific publications by Brooke on this topic. Why keep quiet on it if he has been improving it?

- There is no mention of lipreading algorithms that are that good in the scientific literature. It is not very likely that a quantum leap of this magnitude (if it were) would be kept quiet during all that time.

- It is very difficult to find official references to the video. The British channel “Five”, for instance, where it originally aired, has no mention of it anymore. Movie databases do not even identify a publisher.

- It is being mentioned frequently in the context of anti-semites and holocaust deniers.

- I could not find any records of validation of this technique against historical footage with known audio.

I dunno - maybe you’re right, things got lost in translation, but I’m pretty sure of what I saw in that documentary.

Plus, it is pretty hard to conceive that I would never have heard of a Deaf German fellow national if he had indeed manged to develop a breakthrough in automated lip reading - it would have been a huge story.

So, I did some digging, and here is what I can say: Yes, there was a documentary by the British history channel. Yes, there was a deaf German, Frank Hübner, involved. No, he is not developing lip-reading software. He owns a company that offers sign language courses, at http://www.gebaerdenfabrik.de.

Apparently he used his flesh-and-blood lipreading skills to help decipher segments of Hitler’s private videos.

The puzzling - and frankly amazing - thing is how the English news sites could get the information so horribly wrong. A German newspaper has the story much like I wrote up here. It mentions that this guy used a computer to zoom and enhance the images. It is likely that this is the part that the English newspapers and subsequently the majority of the blogosphere misinterpreted as automated lip reading.

If it looks too good to be true, it probably is.

Großvater, großmutter, wo bist du ?

(-:

Maybe someone can get ahold of that guy in Germany and see what he can do for us? ;)