Tender Claws, the creators of the award-winning interactive VR narrative Virtual Virtual Reality, premiered a new, site-specific, interactive AR narrative experience at the Sundance New Frontier called TendAR. It was a social augmented reality experience that paired two people holding a cell phone and sharing two channels of an audio stream featuring an fish guiding the participants through a number of interactions with each other and exploring the surrounding environment. The participants were to instructed to express different emotions in order to “feed” and “train” the AI fish. Google’s ARCore technology was used for the augmented reality overlays, the Google Cloud Vision AI API for object detection, as well as early access to some of Google’s cutting-edge Human Sensing Technology that could detect emotional expressions of the participants.
LISTEN TO THE VOICES OF VR PODCAST
Overall, TendAR was a really fun and dynamic experience that showed how the power of AR storytelling lies in doing interesting collaborative exercises with another person, but also becoming aware of your immediate surroundings, context, and environment where objects can be discovered, detected, and integrated as a part of a interaction that’s happen with a virtual character.
I had a chance to talk with Tender Claws co-founder Samantha Gorman to talk about the studios’ approach to experiential design for an open-ended interactive AR experience, the unique affordances and challenges of augmented reality storytelling, their collaboration with interactive storytelling theater group Piehole, the challenges of using bleeding-edge AI technologies from Google, and some of their future plans in expanding this prototype experience into a full-fledged 3-hour, solo AR experience with a number of excursions and social performative components.
Here’s a brief teaser for TendAR:
Gorman said that they’re not planning on storing or saving any of the emotional recognition data on their side, and this is the first time that I’ve ever heard anything about Google’s Human Sensing group. I trust Tender Claws to be good stewards of my emotional data, and their TendAR experience shows the potential of what type of immersive narrative experiences are possible when integrating emotional detection as an interactive biofeedback mechanic.
Mimicking a wide range of different emotional states can evoke a similarly wide range of different emotional states, and so I found that TendAR provided a really robust emotional journey that was a satisfying phenomenological experience. TendAR was also an emotionally intimate experience to share with a stranger at a conference like Sundance, but it demonstrates the power of where AR storytelling starts to shine: creating contexts for connection and opportunities to create new patterns of meaning in your immediate surroundings.
However, the fact that Google is working on technology that can capture and potentially store emotional data of users introduces some more complicated privacy implications that are worth expanding upon. Google and Facebook are performance-based marketing companies who are driven to capture as much data about everyone in the world as possible, and VR & AR technologies introduce the opportunity to capture much more intimate data about ourselves. Biometric data and profiles of our emotional reactions could reveal unconscious patterns of behavior that could be ripe for abuse, or be used to train AI algorithms that reinforce the worst aspects of our unconscious behaviors, to the benefit of others.
I’ve had previous conversations about privacy in VR with behavioral neuroscientist John Burkhardt who talked about the unknown ethical threshold of capturing biometric data, and how the line between advertising and thought-control starts to get blurred when you’re able to have access to biometric data that can unlock unconscious triggers that drive behavior.
VC investor and privacy advocate Sarah Downey talked about how VR could become the most powerful surveillance technology every invented or it could become one of our last bastions of privacy if we architect systems with privacy in mind (SPOILER ALERT: Most of our current systems are not architected with privacy in mind since they’re capturing and storing as much data about us as possible).
And I also talked with VR privacy philosopher Jim Preston who told me about the problems with the surveillance-based capitalism business models of performance-based marketing companies like Google and Facebook, and how privacy in VR is complicated and that it’s going to take the entire VR community having honest conversations about it in order to figure it out.
Most people get a lot of benefit from these services, and they’re happy to trade their private data for free access to products and services. But VR & AR represent a whole new level of intimacy and level of detail of information that is more similar to medical data that’s protected by HIPAA regulations than it is to data that is consciously provided by the user through a keyboard. It’s been difficult for me to have an in-depth and honest conversation with Google about privacy or with Facebook/Oculus about privacy because the technological roadmap for integrating biometric data streams into VR products or advertising business models have still been in the theoretical future.
But news of Google’s Human Sensing group building products for detecting human emotions shows that these types of products are on the technological roadmap for the near future, and that it’s worth having a more in-depth and honest conversation about what types of data will be capture, what won’t be captured, what will be connected to our personal identity, and whether or not we’ll have options to opt-out of data collection.
Here’s a list of open questions about privacy for virtual reality hardware and software developers that I first laid out in episode #520:
- What information is being tracked, recorded, and permanently stored from VR technologies?
- How will Privacy Policies be updated to account for Biometric Data?
- Do we need to evolve the business models in order to sustain VR content creation in the long-term?
- If not then what are the tradeoffs of privacy in using the existing ad-based revenue streams that are based upon a system of privatized surveillance that we’ve consented to over time?
- Should biometric data should be classified as medical information and protected under HIPAA protections?
- What is a conceptual framework for what data should be private and what should be public?
- What type of transparency and controls should users expect from companies?
- Should companies be getting explicit consent for the type of biometric data that they to capture, store, and tie back to our personal identities?
- If companies are able to diagnose medical conditions from these new biometric indicators, then what is their ethical responsibility of reporting this users?
- What is the potential for some of anonymized physical data to end up being personally identifiable using machine learning?
- What controls will be made available for users to opt-out of being tracked?
- What will be the safeguards in place to prevent the use of eye tracking cameras to personally identify people with biometric retina or iris scans?
- Are any of our voice conversations are being recorded for social VR interactions?
- Can VR companies ensure that there any private contexts in virtual reality where we are not being tracked and recorded? Or is recording everything the default?
- What kind of safeguards can be imposed to limit the tying our virtual actions to our actual identity in order to preserve our Fourth Amendment rights?
- How are VR application developers going to be educated and held accountable for their responsibilities of the types of sensitive personally identifiable information that could be recorded and stored within their experiences?
The business models of virtual reality and augmented reality have yet to be fully fleshed out, and the new and powerful immersive affordances of these media suggest that new business models may be required that both work well and respect user privacy. Are we willing to continue to mortgage our privacy in exchange to access to free services? Or will new subscription models emerge within the immersive media space where we pay upfront to have access to experiences similar to Netflix, Amazon Prime, or Spotify? There’s a lot more questions than answers right now, but I hope to continue to engage VR companies in a dialogue about these privacy issues throughout 2018 and beyond.
Support Voices of VR
- Subscribe on iTunes
- Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
