By Dr. Hwanjo Heo, principal researcher at ETRI
With the release of its first spatial computing device, the Vision Pro, Apple is reshaping the metaverse industry, which was previously focused on virtual reality (VR). The company envisions a next-generation computing experience where digital content seamlessly blends with the real world. Meanwhile, Meta, a leader in the VR headset market, has introduced full color passthrough in its latest Meta Quest headsets. This feature offers a high-fidelity, real-time representation of the physical world. Additionally, Meta recently unveiled its first augmented reality (AR) glasses, named Orion, designed to provide a more comfortable and integrated user experience.
Before this shift in the metaverse industry, popular VR applications like VRChat aimed to offer a surreal and immersive experience distinctly separate from users’ real-world surroundings. Ironically, this separation acted as a safety boundary for users, as virtual worlds were not easily confused with reality, and events in these virtual spaces rarely impacted real life. However, as spatial computing applications increasingly integrate with the physical world, trustworthiness has emerged as a critical consideration. This raises an important question: What are the essential ingredients for trustworthy spatial computing?
Privacy protection is undoubtedly the most critical requirement for this new computing paradigm. Spatial computing devices, or mixed-reality headsets, are worn by users and equipped with numerous high-fidelity sensors. For example, the Apple Vision Pro features 23 sensors, including a dozen cameras, capturing not only the user’s surroundings but also tracking facial expressions and eye movements with remarkable precision. Industry leaders emphasize the need for robust privacy protections, and Apple has highlighted its commitment to user privacy, particularly with its eye-tracking feature showcased during the Vision Pro’s launch. However, reports indicate that sensitive information may still be vulnerable through Vision Pro’s eye tracking [1]. Given the nature of a wearable device equipped with numerous sensors, the importance of privacy cannot be overstated [2].
The integration of digital content into the real world has led to a shift in the “killer applications” of the metaverse industry. For years, games and entertainment have dominated the use of Meta Quest headsets. Recently, Meta introduced its first metaverse operating system, Meta Horizon OS, aimed at extending mixed-reality experiences to third-party hardware designed for fitness and productivity [3]. This spatial computing paradigm is beginning to integrate powerful headsets into workplaces such as offices, factories, hospitals, and even battlefields, where human lives are at stake. Consequently, even a simple denial-of-service attack that slightly degrades video processing performance could result in serious, irreversible harm, particularly during critical remote surgical operations. Ensuring secure and reliable operation of devices and services while managing the heavy workloads required for an immersive user experience presents significant technical challenges [4].
Lastly, spatial events triggered by interactions between digital and real-world environments should be recorded and traceable through a reliable provenance system. Trustworthy spatial computing necessitates mechanisms that ensure the integrity and accountability of relevant events, especially given the unprecedented connection between the digital and analog worlds. Blockchain is frequently viewed as a trust anchor for both environments, although significant performance improvements will be needed to accommodate the scale of future spatial computing [5]. In South Korea, the Electronics and Telecommunications Research Institute (ETRI) is conducting a project titled “Trustworthy Metaverse: Blockchain-enabled Convergence Research,” funded by the Ministry of Science and ICT. This project addresses all three critical research areas—privacy, security, and blockchain scalability—to establish a foundation of trustworthiness for future spatial computing and metaverse applications.
[1] https://www.wired.com/story/apple-vision-pro-persona-eye-tracking-spy-typing/
[2] https://arxiv.org/pdf/2207.13176
[3] https://www.meta.com/blog/quest/meta-horizon-os-open-hardware-ecosystem-asus-republic-gamers-lenovo-xbox/
[4] https://engineering.fb.com/2023/09/12/security/meta-quest-2-defense-through-offense/
[5] https://cointelegraph.com/learn/what-is-metaverse-in-blockchain
About the author
Dr. Hwanjo Heo is a principal researcher at the Electronics and Telecommunications Research Institute (ETRI). He earned his Ph.D. from Korea Advanced Institute of Science and Technology (KAIST) and his master’s degree from Purdue University. His research interests include the security of computer systems, encompassing blockchain, computer networks, and artificial intelligence systems.
