Imagine you’ve planned a sightseeing vacation with a friend, but your flight gets canceled and now there’s no way to get to your destination in time. If that happened today, you’d be looking at a few days sitting home with the FOMO kicking in. In a future where hybrid augmented reality-virtual reality (AR-VR) experiences are widespread, though, that missed flight wouldn’t have to ruin the vacation. Your friend could don an AR rig, you could put on your VR headset, and the two of you could still share the experience.  

To make these shared experiences not just possible, but also as intimate as a virtual experience can be, it’s essential to have AR-VR technology that can accurately produce a digital twin space with real-time responsiveness. The wheels are in motion on this exciting possibility.

The opportunity of hybrid experiences

Connection with other humans, even in a very connected digital world, can be hard to come by when busy schedules, unexpected complications, and a million other aspects of life get in the way. Innovations in AR-VR hybrid experiences will eventually be able to help you get morning coffee with that old friend you’ve been trying to connect with for three months or even share an amazing concert experience with someone halfway across the planet.  

Hybrid augmented reality-virtual reality scenarios create experiences in which at least one participant is in a physical space with an AR device, and at least one more is in a digital twin of the same physical space with a VR device. In these hybrid scenarios, participants (AR and VR) can communicate with each other. But instead of simply being a livestream, AR-VR hybrid experiences will need head-mounted devices (HMD) with a system server that maintains a real-time dynamic aspects' state of a physical space and a digital twin of the physical space. The dynamic aspects' state will include locations and orientations of AR users in the physical space and VR users in the digital twin made possible by an image analyzer configured to derive dynamic parameters from video frames captured by the AR HMD, like weather, lighting, people/crowds, significant objects and any other parameters that bring real-world conditions into the virtual world.

Shared experiences are about movement. By perfecting the way we model the dynamic aspects of the physical space and how the VR participant can perform locomotion (in the digital twin of the physical space), we can bring a more visceral sense of “togetherness” to an experience. Once the participants – AR and VR users alike – can enjoy seamless interaction, we can take shared experiences to a new level.  

This is why, at Adeia, we’re exploring methods by which the dynamic aspects (e.g., locomotion, people and object modeling) of both the physical and digital space can be accounted for to develop better representations of shared experiences.  

The future will be filled with hybrid experiences similar to this. At Adiea we are exploring how shared experiences like the one we just described could and should work, and then expanding the methods, techniques and inventions in our extended reality (XR) and immersive R&D portfolio to make those experiences as rich and fulfilling as possible.

Hybrid experiences create unique opportunities for locomotion in the digital twin world. A VR user, for example, may use self-locomotion, or perform locomotion guided by a selected AR user. This allows a VR user to “accompany” an AR user even as the AR user moves through the real world and the VR user moves while experiencing the digital twin.

Since locations of users are kept on a global level, the system is aware of the locations and orientations of all users, whether in the physical space, or traversing the digital twin. In short, with immersive apps, your VR friend may be sitting on their bed, but your AR rig makes it appear as if your friend is walking with you; when you turn your head, you see a representation of them there. As the sensory capability of HMDs and trackers improves, the shared experience will go beyond the impression of mere presence – even facial expressions and body poses may be faithfully translated between the worlds.

Powering the future of immersive applications with L4S

Streaming requires the data capacity of high bandwidth, but bandwidth alone cannot tackle the challenges of hybrid experiences. The content in immersive apps is constantly adapting to user input, which makes solving latency issues a challenge. Platforms like Metaverse are the frontier of the AR-VR hybrid experience, which is why Adeia is heavily involved with Metaverse Standards Forum (MSF) Labs Initiative on providing empirical evidence of the benefits of L4S: Low Latency, Low Loss, Scalable Throughput.

L4S is a sender-receiver protocol, which when used in conjunction with a capable network, enables an ultra-low latency, high throughput “priority lane.” The protocol has been built to allow for fair use of available internet capacity in instances of competing traffic flows. Introduced as an IETF (Internet Engineering Task Force) proposal, it is being implemented and tested in the real world by service providers such as Comcast and T-Mobile.

We believe future immersive apps will benefit from wide adoption of L4S. To demonstrate the benefits, the MSF Labs Initiative created a testbed and demo platform. P-Net Labs, a member organization of the MSF, was selected to run tests on a 5G network that simulates real-world conditions to measure performance to validate L4S compliance as well as measure its performance benefit to applications and platforms, ultimately resulting in a better lag-free user experience. As a single, one-stop evaluation platform providing real-world validation, the Labs Initiative is an ideal way for developers to optimize applications for L4S without depending on fragmented ISP testing.  

It could then also offer developers clear instructions, best practices, and support for how to optimize their applications in the evaluation environment.  

By speeding the adoption of L4S, we can push all immersive experiences, including the hybrid experience ecosystem forward. The goal is for all of us to be able to plug into each other as easily as possible — it will be about people and connection, not just what headset you own.