South Korean TV broadcaster MBC recently aired a Korean language documentary that centers on a family’s loss of their young daughter, seven-year-old Nayeon. Using the power of photogrammetry, motion capture, and virtual reality, the team recreated Nayeon for one last goodbye with the family’s mother, Ji-sung.
Like a typical seven-year-old, Nayeon was a spry, playful kid. Then she suddenly fell ill with hemophagocytic lymphohistiocytosis (HLH), a life-threatening disease of severe hyper-inflammation caused by uncontrolled growth of the body’s white blood cells. To Ji-sung, it all first appeared to be a common cold due to Nayeon’s swelling and fever. Her daughter passed away one month later in the hospital.
Called ‘Meeting You’, the documentary goes on to recreate a series of warm memories from Nayeon’s life. Donning an HTC Vive Pro, Vive trackers, and wireless adapter, Ji-sung revisits a virtual version of a park the family would frequent. Nayeon giggles, and cautiously asks her mom if she’s afraid. She wonders why it’s cold outside. Touching hands, they’re both lifted up to a heavenly realm.
The mother and the virtual simulacrum experience more happy memories together. It’s Nayeon’s birthday, and honey rice sweets, a birthday cake, and her favorite seaweed soup are all there. Ji-sung puts her down to bed for a nap, and plays with her hair as Nayeon precociously bobbles around.
For non-Korean speakers, using YouTube’s auto-generated translation is basically useless for the nine-minute video. Thankfully, the baked-in Korean subtitles were simple to translate via Google’s camera-based app and were remarkably clear too. Translation or not, the power of Ji-sung’s emotions are intensely human, no matter the language.
Putting aside the obvious exploitation factor of reuniting a mother with her deceased child for television viewers—Nayeon even pulls at the heartstrings by telling her father to stop smoking, and her siblings not to fight so much—recreating a deceased loved one in such high fidelity raises some ethical concerns, and they’re ones we simply don’t have clear answers to yet. Whether conjuring virtual doppelgangers of lost loved ones may one day be considered an unnecessary re-traumatization, or a valid coping mechanism to help overcome tragedy, we just can’t say for now.
Personally, all of it was unsettling to me at first glance, and maybe well outside of what I’d consider healthy. Still, it does seem to have helped Ji-sung to some extent, who carries with her a tattoo of her daughter’s birthday as an indelible reminder. At the family home there are pictures of Nayeon all over the place. Every month the family visits Nayeon’s burial place to leave her favorite toys in remembrance. It’s clear the family isn’t running away from the reality, or trying to forget what was very likely one of the worst things to happen to them either, but in the same breath they aren’t holding too tightly onto the past. Before Ji-sung left for the VR experience, she burned letters and offerings to her child, including a shirt that was too warm for her daughter to wear at the hospital. Integrating those last, vivid virtual experiences of Nayeon into her memory serves as a singular, bittersweet goodbye, one you’d never get in a hospital room. Death oftentimes proves to be frightfully uncinematic like that.
After translating each subtitle, my first cynical instincts to label this a blatantly insensitive puppeteering of a dead child are mostly gone. It’s not perfect, but it makes sense to me on some level.
Creating a Virtual Child
Developed by South Korean startup Vive Studios (no relation to HTC’s Vive Studios), the virtual Nayeon was created over the course of eight months using a variety of techniques. Motion capture not only recorded an adult actor’s movements, but also facial expressions, some of which were acted out based on video and photos of the real-world Nayeon.
High resolution photographs were taken in a 3D capture technique called photogrammetry. Nayeon’s little sister, a spitting image of her older sibling, was used as the basis of the character model.
Although not apparent in the video above, the studio also added a degree of liveliness to the character by integrating voice recognition and a basic AI, which would let the pair have a basic conversation. Responses were created based off of family interviews and videos.
In the end, revitalizing the image of a deceased person isn’t exactly new, and a bevy of examples come to mind: Fred Astaire dancing with a Dirt Devil, Tupac holograms, and more recently the litany of deepfakes that make you question whether famous actors are still alive or not. And much like those early CG humans and carefully contorted deepfake masks, Vive Studios’ tailor-made VR experience is no doubt impressive for a short while too, but at this point it’s really no more than a carefully orchestrated funerary rite. It’s that point in the future though when AI is capable of automatically conjuring a person based off a compendium of video and photo that we’re waiting to see. Because whether you like it or not, virtual humans are coming, and I think we’ve just taken one step closer.