Fans in the UK and Japan are being offered a brand new way to experience this summer’s FIFA World Cup, driven by FIFA’s 3D live tracking data.
Immersiv.io, based in Paris, is working with the BBC in the UK and DAZN in Japan to deliver an immersive 3D experience.
The company was cofounded by Stéphane Guérin and Emmanuelle Roger in 2016, both engineers and sports fans, with the vision to reinvent the sports fan experience with the use of immersive technologies.
“Immersiv.io started building its 3D platform (ARISE) three years ago, when skeleton data was starting to be tested and deployed by major leagues,” Roger tells TVBEurope.
The platform has already been tested for multiple sports, including ice hockey, tennis, basketball, American Football, and on multiple platforms (mobile, web, Meta Quest, Apple Vision Pro, Samsung Galaxy XR).
During a game, the live skeleton data is ingested into the platform, which is then transformed into an interactive live 3D recreation of the game. “The skeleton data consists of multiple coordinates for each player, which are sent multiple times per second, and describe the actual movement of the player,” explains Roger. “This is captured by cameras set up in the stadiums; this part is not handled by Immersiv.io but by tracking providers who are contracting with leagues/competitions.”
The data is then mapped onto avatars of the players, which are animated in the 3D environment to reproduce the live game. “We’ve developed a pipeline to generate the players’ (and referees’) avatars at scale based on their headshots and height/weight.
“In parallel, we’re ingesting the live audio from the broadcaster, synchronising the audio track with the 3D recreation. We have developed dedicated technologies to automatically synchronise video or audio with skeletal and statistical data.”
The workflow employs both artificial intelligence and machine learning, both to generate the player avatars and in the synchronisation of the video/audio with the data. “This part is critical for the experience and often very complex to do because the production workflows for data and video/audio are completely different.
“We have also developed processes to improve and fix the data we are receiving from the data providers. As this data is captured live, there are sometimes inconsistencies, players not tracked correctly, or some jitter. We use AI tools to improve the 3D recreation as much as possible.”
In the BBC’s 3D environment, fans can choose what they want to watch, from the broadcast view, tactical view, third-person player PoV, and first person POV.
“To give more insights to the users, we’re overlaying metrics/indicators on top of the 3D playback depending on the selected PoV (player in possession, pass possibilities, distance to the goals, formation lines),” adds Roger.
Once the final whistle is blown, the post-match experience (3D highlights and full replay) is instantly available. Because both involve replaying the data received during the game there is no additional processing required. “It is up to the BBC or DAZN to choose how quickly they’d like the game to switch from live to post-match status.”
For fans in the UK, the BBC’s 3D Experience offers both full-match replays and highlights via desktop or the BBC Sport app. In Japan, DAZN is using the technology during its half-time and post-match coverage, clipping multiple 3D replays of the same action from different angles.
Asked if she believes this is the future of watching sport, Roger says she believes the 3D recreation will improve over the next few years, with the progress of technology and AI. “Our vision is that it will eventually become as realistic as video and will transform the way the matches are produced and consumed by fans!”
