Oculus have released a dedicated Unity plugin which aims to automatically detect and transform an audio stream from speech into virtual character lip movements automatically.
It seems that the substantial investment in research and development at Oculus over the last couple of years is leading to some welcome, if somewhat unexpected developments to come out of their labs. At Unity’s 2016 Vision VR/Summit, the company have unveiled a lip sync plugin dedicated to producing lifelike avatar mouth animations, generated from analysing an audio stream.
The new plugin for the Unity engine analyses a canned or live audio stream, such as microphone captured live voice chat, into potentially realistic lip animations for an in-world avatar.
The system processes the audio stream and created a set of values called ‘visemes’. For those not familiar with the term (and I certainly wasn’t until just now), a viseme is a “gesture or expression of the lips and face that corresponds to a particular speech sound”, according to the documentation accompanying the Plugin release. The practical upshot of incorporating the functionality then, is that your game-world counterparts, whether human or AI controlled, will look as if they’re actually saying what your hearing. At least in theory. The plugin’s documentation further elaborates:
Our system currently maps to 15 separate viseme targets: sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh, and ou. These visemes correspond to expressions typically made by people producing the speech sound by which they’re referred, e.g., the viseme sil corresponds to a silent/neutral expression, PP appears to be pronouncing the first syllable in “popcorn,” FF the first syllable of “fish,” and so forth.
Whilst there are multiple use cases for this technology, along with the recent release of Oculus Social for the Gear VR platform, you have to think that the company’s focus on ‘selling’ the idea of useful human interactivity in virtual reality is pretty strong, given the how tangential this release is to core virtual reality technologies.
You can grab the new plugin for Unity 5 right here. And if any developers out there have a chance to try it out, we’d love to see some examples of it in action.