Over the past 13 or so years, there has been material technical progress. And we do see growing deployment. Today, XR is selectively used in civil engineering and industrial design, in film production, on assembly lines and factory floors. Some schools use VR some of the time in some classes - and the utility a virtual classroom with virtual Bunsen Burners and virtual frogs to dissect, all overseen by an embodied instructor, while you sit beside and make eye contact with your peers, is obvious. VR is also increasingly popular for workplace safety training, especially in high-risk environments such as oil rigs; teaching personnel how, when, and where people look is already having life-saving applications. And on the topic of saving lives, Johns Hopkins has been using XR devices for live patient surgery for more than a year, beginning with the removal of cancerous spinal tumors. If you use a high-end VR headset such as the Varjo Aero (which also requires a physical tether to a gaming-grade PC and costs $2,000) to play a title such as Microsoft Flight Simulator (which operates a 500,000,000 square kilometer reproduction of the earth, with two trillion individual rendered trees, 1.5 billion buildings, and nearly every road, mountain, and city globally), there is the unmistakable feeling the future is near.
The examples listed above are technically impressive, meaningful, and better than ever . But the future was supposed to have arrived by now. In 2023, it’s difficult to say that a critical mass of consumers or businesses believe there’s a “killer” AR/VR/MR experience in market today; just familiar promises of the killer use cases that might be a few years away. These devices are even farther from substituting for the devices we currently use (and it doesn’t seem like they’re on precipice of mainstream adoption, either). There are some games with strong sales—a few titles have done over $100MM—but none where one might argue that, if only graphics were to improve by X%, large swaths of the population would use VR devices or those titles on a regular basis. I strongly prefer doing VR-based presentations to those on Zoom—where I spend 30-60 minutes staring at a camera as though no one else is there. But the experience remains fraught; functionality is limited; and onboarding other individuals is rarely worth the benefit because its participants seem to find these benefits both few and small. When the iPhone launched, Steve Jobs touted it did three distinct things—MP3 player, phone, internet communicator—better at launch than the single-use devices then on the market. The following year, the iPhone launched its App Store and “There’s an App for That” proliferated, with tens of millions doing everything they could on the device. The “killer app” was that it already had dozens of them. (...)
Of course, XR devices will not suddenly replace an existing device category. Hundreds of millions will first use VR/AR alongside their consoles, PCs, and smartphones before tens of millions drop one of the latter for the first – and hundreds of millions will continue to use both longer after (this essay is written on a PC, for example). But the timing of this transition is relevant for those investing. Return to my Johns Hopkins example. After completing the surgery, Dr. Timothy Witham, who is also director of the hospital’s Spinal Fusion Laboratory, likened the experience to driving a car with GPS. I love this analogy because it shows how XR can complement existing devices and behaviors rather than displace them (it also complements reality, rather than disconnecting us from it). Put another way, we drive a car with GPS; we don’t drive GPS instead of a car, and GPS doesn’t replace the onboard computer either. What’s more, many of us travel more often because GPS exists. Dr. Witham also provides a framework through which we can evaluate the utility XR devices. To exist, they need not upend convention, just deliver better and/or faster and/or cheaper and/or more reliable outcomes. But even under these more moderated measures, the future seems far off. GPS began to see non-military adoption in the 1990s, but it took another two decades to mature in cost and quality to become a part of daily life. Furthermore, the mainstream value in GPS was not only in improving commutes but in enabling applications as diverse as Tinder, Siri, Yelp, Spotify, and many others. (...)
Many entrepreneurs, developers, executives, and technologists still believe XR is the future (I do). In particular, these groups believe in AR glasses that will eventually replace most of our personal computers and TV screens. And history does show that over time, these devices get closer to our face, while also more natural and immersive in interface, leading to increased usage too. But why is this future so far behind? Where is the money going? What progress is being made? And most importantly, how many XR winters must come and go before a spring actually leads to summer?
“It Looks Like Wii Sports”
More than half of all households in the United States own a video game console. In almost all cases, this console is the most powerful computing device owned, used, or even seen by the members of that household. This includes those households who own the most recent model of iPad Pro or work in an office with a high-end enterprise PC or Mac. Regardless which one they choose, that video game console is also more affordable than most other consumer or even professional-grade computing devices. It typically costs more, for example, to purchase a comparably powered gaming PC or even to replace the graphics card on an existing PC. This is because consoles benefit from substantial economies of scale, with their manufacturers shipping 50–150MM mostly standardized units over a decade. Purchasing individual components, each one individually packaged, marked-up, and retailed, often with new models released annually, is expensive. Video game consoles are also subsidized, typically by $100–$200, as their manufacturers pursue a razor-and-blades model whereby subsequent software purchases eventually recoup the money lost selling the hardware. No graphics card or monitor manufacturer gets a cut of your Robux or V-Bucks.
Compared to everyday devices, the computing power of a video game console is so great that in 2000, Japan even placed export limitations on its own beloved giant, Sony, and its signature PlayStation 2 console. The government feared that the PS2 could be used for terrorism on a global scale, for instance to process missile guidance systems. The following year, in touting the importance of the consumer electronics industry, U.S. Secretary of Commerce Don Evans stated that “yesterday’s supercomputer is today’s PlayStation.” Evans’s pronouncement was powerful—even though it was arguably backwards; today’s PlayStation is often tomorrow’s supercomputer. In 2010, the U.S. Air Force Research Laboratory built the 33rd-largest supercomputer in the world using 1,760 Sony PlayStation 3s. The project’s director estimated that the “Condor Cluster” was 5% to 10% the cost of equivalent systems and used 10% of the energy. The supercomputer was used for radar enhancement, pattern recognition, satellite imagery processing, and artificial intelligence research.
Yet in many ways, video game consoles have it easy. Consider the PlayStation 5 or Xbox Series X, both top-of-the-line video game consoles released in 2020. These devices are nearly ten pounds and larger than a shoebox—brutal in comparison to other consumer electronics devices, but fine given that these devices are placed inside a media shelving console and never moved. In fact, it’s not fine—it’s an advantage! Because these devices can be large, unsightly, and stationary, Sony and Microsoft get to place large and loud fans inside their consoles, which keep these consoles cool as they perform their intensive calculations, and aid these fans with large intake and exhaust vents. Sony and Microsoft can also keep component costs down because they don’t need to prioritize their size the way a smartphone manufacturer must. And while Sony’s and Microsoft’s consoles are heavy, they, unlike most consumer devices, never need a battery. Instead, they receive constant power from the electrical grid. This reduces the size of the device, as well as the heat it generates, which in turn means that the fan can be smaller, too, and means they can run indefinitely, rather than just a few hours. (...)
This context around consoles is important to keep in mind as we consider VR/AR/MR. It’s common to hear the critique that the experiences produced by these devices look worse than those produced by the consoles of a decade ago that cost half as much at the time. When it comes to visually rendering a virtual environment, VR/AR/MR devices will always fall short of a modern video game console. Always. This is because the “work” performed by these devices is far, far harder while the constraints are far, far greater. (...)
But Does It Play Better
All consumer tech faces tradeoffs and hard problems. But XR devices require so many points of optimization - heat, weight, battery life, resolution, frame rate, cameras, sensors, cost, size, and so on. Zuckerberg’s belief in this device category, placed aside these problems, explains how it’s possible he’s spending $10B+ year after year after year. That money is being sunk into optics, LEDs, batteries, processors, cameras, software, operating systems, and the like. And if Zuckerberg can crack this, with nearly all of his competitors years behind (if they’re bothering at all), the financial returns may be extraordinary. In early 2021, Zuckerberg said “The hardest technology challenge of our time may be fitting a supercomputer into the frame of normal-looking glasses. But it's the key to bringing our physical and digital worlds together.”
The immense difficulty of XR also explains why “the graphics look like they’re from the Wii” is actually a compliment—it’s a bit like saying an adult ran 100 meters as fast as a 12-year-old, even though the adult was wearing a 50-pound backpack and solving math problems at the same time. This defense is separate from whether Meta’s art style is good relative to its constraints. There’s pretty widespread consensus it’s bad. However, it’s not quite fair to compare the graphics of Meta’s avatars or signature products, such as Horizon Workrooms, to those of third party VR titles such as VRChat or RecRoom. This fidelity is available to Meta, but only selectively – as we know, “graphics” are just one part of the computing equation. For example, a two-person meeting in Horizon Workrooms that expands to eight might require a halving of the frame rate or avatar definition or accuracy in eye reproduction, while also draining batteries far faster. Or your avatar—intended to be a representation of you—could look better or worse, more detailed or generic, legged or legless, depending on which application you’re using it in. This gets eerie, distracting, and annoying. (...)
Many people I know believe that absent extraordinary advances in battery technology and wireless power and optics and computer processing, we simply cannot achieve the XR devices that many of us imagine and that would conceivably replace the smartphone or merely (a smaller ask) engage a few hundred million people on a daily basis. Just last December, six years after he told Venture Beat that such devices were five to seven years away, Tim Sweeney told Alex Heath, “Well, I think that augmented reality is the platform of the future. But it’s very clear from the efforts of Magic Leap and others that we need not just new technology but, to some extent, new science in order to build an augmented reality platform that’s a substitute for smartphones. And it’s not clear to me whether that’s coming in 10 years or in 30 years. I hope we’ll see it in my lifetime, but I’m actually not sure about that.”
by Matthew Ball, MatthewBall.vc | Read more:
Image: DALL-E, uncredited
