Star Trek’s holodeck has long inspired dreams of digitizing our world into interactive, immersive, playable 3D space. Spatial computing is how we get there. It is how we make a holodeck.
We can already rapidly 3D-map people and places. We can already stream that content to mixed-reality devices. Soon we’ll give up watching TV shows and playing games through distant screens and start engaging with them through glasses or lenses. Our entertainment will become a layer of augmented reality mapped to the specifics of our living room, with the actors dancing around our furniture as if each show were tailor-made to our homes. (For more on spatial computing, see our previous posts Spatial Computing: The Next Generation of Information, or Spatial Computing in a COVID-19 World, or this intro piece from Scientific America prophesizing its rise as well).
However, spatial computing and our future holodecks won’t just be fun and games. Welcoming volumetric-captured performers on the 3D stage of your living room will change how we engage with content, ideas, and each other.
Yes, this is a huge bump in our ability to work and play with colleagues, friends, and family worldwide. Yes, this is yet another variable in our ability to live and work remotely as part of the #futureofwork. More crucially, this is an unprecedented leap for storytelling—for how we craft and share our stories.
Human storytelling has already come so far. We transitioned from oral stories told in real-time around primordial campfires to written tales transmitted through time and space on papyrus, parchment, and paper. In the last century, we rapidly extended this capacity—first via radio and film, then television, and finally to the internet and its power for rapid literary creation and sharing along with our ability to stream and binge content like never before. Each iterative improvement from reading to radio to tv improved the experience of our stories, the fullness and fidelity with which they could be told.
Though it will mean high initial costs, spatial computing will likely influence a financial bonanza for the entertainment industry. By including 3D, music, gaming, and Hollywood industries will be able to transport their performers and influencers directly into your living room, up close and personal and even personalize-able. Beyond mapping the digital theater stage to your layout, this means the possibility of new narrative devices—new ways of engaging the audience to move the story along. It will be fun, at times it will be frivolous. More than anything, it will be powerful and powerful systems can be misused easily.
An Extended Reality War for…Reality
There is considerable evidence that humans build empathy through both context and first-hand experience. Listening to someone tell you about what happened to them is an abstract, second-hand experience. Seeing a still or moving picture is more concrete. Being somewhere and seeing something is mind-blowing and change-making.
It is one thing to see a refugee camp through images and our televisions. It is another thing entirely to be there in person – to experience the sights, sounds, and intimate stories of its inhabitants first hand. Immersive media, like virtual reality and augmented reality, begin to bridge this gap between first- and second-hand experience. Unfortunately, current forms of immersive reality hardware are bulky, cumbersome and have limited content, limiting their value.
But these hardware and content restrictions are soon to be lifted, just as costs go down simultaneously. Major players like Apple and Facebook are racing against startups to replace 2D screens with 3D-capable devices that consume AR and VR content while simultaneously collecting 3D scans of their usage environment to contribute to collaborative 3D maps. These new devices may take the form of glasses, contact lenses, or even more novel forms. Meanwhile, heavyweights at Hollywood studios and new biggies like Niantic (developer of mobile, AR game Pokémon Go) race to create content and explore new mechanisms for engaging with these extended, augmented, virtual, and mixed realities.
These technology trends will change how governments operate. Countries along the Pacific Rim already are incorporating spatial computing into their everyday smart city lives and government policies, leaning spatial computing’s promise of machine-readable, computer vision-driven mechanisms for identity and activity tracking. Australia, China, Japan, Singapore, and South Korea are investing heavily in digital twins, virtual reality, and holography in business and academia—aiming to be the opening players and ultimate arbitrators of these new ubiquitous content forms.
Indeed, immersive media is already ubiquitous for some. Hit VR game Beatsaber boasts over a million sales, with many users jumping on daily to get their sword-ercise. Millions more spend at least an hour or two in Twitch, the baseline for engagement in 2D and emerging 3D realms of human engagement. Immersive media is still in its early days, but just as radio, tv, and mobile before them, immersive will eventually be omnipresent as hardware and production costs go down and quality and accessibility of content go up. And each piece of media you watch in it will be that much more real and empathy building than the content you used to watch on traditional 2D Netflix. It will feel more real. It will be in your living room, in your personal space, performing just for you, up close and personal, while simultaneously linking you up with other audience members to relate to the experience together, digitally.
Empathy is good, but empathy can be misdirected and manipulated by more than storytellers hoping to give you a cultural thrill. A personal holodeck will not just mean we get to experience a next-gen interactive Game of Thrones in immersive 3D around us; it also means we’ll be able to digitize and re-live our current events. Spatial computing is a documentation mechanism for reality itself, and as such, it is the new format for how we will record and share our history.
Imagine if we could 3D-digitize things that have already happened—to be able to walk among the cars in the parade on November 22, 1963 as JFK was assassinated, to stand near the stage on January 1, 1863 as Abraham Lincoln gave the Emancipation Proclamation. It will mean a revolution in primary school education. Preserving our present as future historical documentation also opens the door to its use as misinformation and propaganda.
We’ve seen bumps up in digitization technology yield similar results before. Just after the Russian Revolution, when certain comrades fell out of favor, they were removed from office and official photographs of things that had already happened. North Korea has an interesting habit of re-writing history via Photoshop as well. Humans have a bad habit of trying to erase the past by altering media.
Our courts still struggle with evidentiary rules for photography and video, but we don’t have to explain how photography and video cameras work at this point. Immersive media will require experts to explain new tech to juries in hope that they will rapidly understand and pass appropriate judgement. Point clouds, photogrammetry, and all the other rapid 3D content mechanisms will have to be explained, as will the many different ways these technologies can be manipulated for use in virtual and augmented reality. 2D deepfakes are already a problem; 3D deepfakes of humans and places in immersive reality are going to confuse everyone from juries to the public watching the news play out in their living room.
Events recorded in 3D could be manipulated to satisfy future propaganda and political machinations. To tell our stories in 3D effectively, we must immediately start building technology stacks that limit their misuse. Just as we are beginning to wage tech wars to handle the threats of video deepfakes, we need to begin to apply the same standards, concerns, and care to the wild frontier of 3D data that lies ahead. We will explore it soon enough. And if we’re going to be able to re-live the world’s facts and fictions as immersive experiences, we’re going to need to be able to tell fact and fiction apart.