It’s been three years since the groundbreaking virtual concert, ABBA Voyage, launched in London. Featuring the iconic line-up in full digital rendering, the show takes audiences back in time to enjoy the band singing, dancing, and seemingly interacting across a stage in 3D to a 3,000-capacity arena.
Similar technology has proved truly popular for other artists, unleashing endless creative potential for live concerts. In recent years, Madonna has performed alongside five dancing holograms of herself, bringing added theatre to her live shows, while an immersive Whitney Houston hologram tour gave fans an opportunity to honour the musical icon in digital form.
Volumetric content capture, synchronised recording from multiple camera angles, has brought new real-time entertainment possibilities like this to life, allowing innovative experiences to be made possible. But what about the next phase of this real-time technology that allows people to communicate or engage with fully realised three-dimensional forms in real life?
Volumetric video processing makes this a reality, by enabling the viewer to explore a remote or virtual environment without the constraints of a specific viewing angle or position. This added depth and perspective brings greater intimacy and more meaningful interaction to real-time applications – across the entertainment industry, but also increasingly, in professional and personal realms.
Even better, volumetric video doesn’t require a major software or hardware overhaul as it operates on existing standards and devices supporting low-latency E2E streaming.
How does volumetric video work?
Nokia has developed the world’s first standards-based real-time volumetric video communication system. It is based on visual volumetric video-based coding (V3C) and MPEG immersive video (MIV) standards and leverages existing 2D video coding tools. This makes it a cost-efficient and scalable solution that avoids the bitrate requirements and lack of standardisation usually associated with volumetric video.
The setup is straightforward: a 4- or 8-camera setup provides the detail needed for a fully immersive 3D content on a head-mounted display (HMD). Meanwhile, a 2-camera setup (potentially incorporated into laptops and/or monitor devices) opens up new possibilities for everyday use cases such as online training and remote collaboration.
Previous implementations have relied on the delivery of pre-encoded media, but the implemented real-time transport protocol (RTP) in V3C’s payload format defines protocol level support for real-time volumetric video delivery. Depth sensing with time-of-flight techniques makes environmental mapping possible and allows for people or objects to be segmented out of their surroundings without the need for a green-screen studio or chroma-keying. The system can reach extremely low glass-to-glass latencies of around 160 milliseconds (ms), falling comfortably within the parameters for conversational interaction and below the average latency of traditional teleconferencing services.
The key use cases for volumetric video
Any application that would benefit from more personalised and engaging remote interaction can be enhanced with volumetric video.
Hybrid working habits have become commonplace in the last five years, enabled by significant improvements in video conference technology. However, 2D video images (and the cartoon avatars used in metaverse environments) continue to pose limitations on personal interaction. With volumetric video, you will be able to get a complete rendering of a potential new hire, or a keynote speaker, with all their non-verbal body language and mannerisms, providing a far more meaningful level of communication.
With volumetric video, music, sports or comedy fans can stand (virtually) side-by-side with their idols, making for more immersive experiences. Golf fans will be able to discover the secret to a World Championship holder’s swing – with real-time analysis from every angle, they can get up onto the tee from just inches away.
Online education is another sector which will benefit greatly from improved visual depth perspective. Perhaps a guitar teacher wants to observe their pupil’s hand positioning and finger placement on the fretboard. This comprehensive viewpoint could help the teacher to remotely identify and correct any nuances that may be hindering the student’s playing ability. This is a clear progression from the limitations that come with current 2D teleconferencing tools.
In a professional environment, volumetric video allows a group of people to enter a shared 3D space in extended reality, interacting with three-dimensional CAD models in real-time. This is possible because volumetric video captures the object live, without lengthy preprocessing stages to create the 3D model. In this rendered reality, engineers, architects, or set designers could perform a dynamic examination of the model from every conceivable angle, ensuring a comprehensive understanding of spatial relationships and bridging the gap between virtual planning and tangible construction.
Imagine taking part in engine maintenance training where the engine as well as the maintenance work is volumetrically captured and shared live. The practical applications for volumetric video communication are vast.
Shaping the future of broadcast and media
Volumetric video reflects the progress made in developing and applying standards for multimedia systems, and is anticipated to contribute significantly to the future evolution of XR. The best way to understand volumetric video is to experience it for yourself. As this technology moves from proof-of-concept to commercial reality, it is designed to inspire future innovators to create new experiences for the world to enjoy as volumetric video becomes part of everyday communication.
