At GDC 2013 I saw a cool demo at the AMD booth called FaceX. Created by a company called Mixamo, FaceX tracks a player’s face using a standard webcam and can use that data to animate a 3D model in real-time. While this tech could be used as cheap motion capture to animate computer-controller characters, the much more exciting implication would be real-time facial animation for multiplayer games where teammates communicate with voice chat.
FaceX could be an elegant solution to increasing immersion in current multiplayer games that aren’t yet making the leap to full blown VR.
In theory, the system is perfect for a PC gamer. Most likely they’ve already got a microphone/headset to communicate with teammates. If they’re on a laptop, they already have a camera, or if they’re using a desktop, a cam can be picked up on the cheap — either way, they will already be close to the camera for accurate tracking. Developers wouldn’t even need to sync the audio to the facial animation as they are both being delivered in real time. There’s a huge audience of PC gamers to which this technology could be deployed quickly and inexpensively.
Using this tech with console gamers would be a bit more complicated, but not impossible. Current console cameras like the Kinect and PlayStation Eye probably don’t have the necessary resolution to accurately track gamers’ faces as far as the couch. Higher resolution cameras would be an easy fix, and we’ll probably see that from the Kinect 2 and the PS4 Eye.
FaceX tracks more than just mouth and eye movement. It uses the incoming data to convincingly animate cheeks and other subtle face muscles thanks to machine-learning analysis of over 30 test subjects, Mixamo told me.
You can see that it’s a bit jumpy in the video as I turn off to the side to ask the rep questions and look down at my camera, but front-on, where PC gamers are looking all the time, it works quite well, and I would expect it to get better over time.
I can imagine it now: hunkered down in Battlefield 3 with bullets whizzing over my head. The M-COM Station objective is just a few yards away, but the defensive line has me and a teammate suppressed. My teammate, who is communicating with me over voice chat, turns to me and says, “Drop some smoke and head straight for the station, I’ll flank left and take them out,” all the while, I can actually see his mouth moving in real time — and hear the audio emanating from his avatar. This would be significant step forward over simply hearing a disembodied voice coming from nowhere.
Another great place for FaceX would be slower MMORPG games like Second Life or World of Warcraft where players frequently have extended face-to-face interaction through trading, meetings, and general banter. Second Life with FaceX would be fun, but the novelty of seeing a non-human avatar (like a Troll or Orc in World of Warcraft) animated as though you are talking through them would be very cool indeed.
While this tech would be great for virtual reality as well, there are some issues. The near-term future of virtual reality will likely revolve around head mounted displays like the Oculus Rift. This means that the eyes are always going to be obscured to a normal webcam. There was ways around this, like small eye-tracking cameras inside of the HMD, but that adds an extra hardware requirement which quickly begins to shave down the potential audience. There’s also the issue of head-tracking wherein players using head mounted displays will not always be looking directly at a computer monitor in front of them. Thus their faces will sometimes be obscured when they turn away from the camera and tracking would be lost.
But mainstream virtual reality gaming is still a little ways off. Meanwhile, there are tons of games that could benefit from this tech; Mixamo tells me that FaceX will be launched in the next 6 months.
