On Sunday night Liam Payne will open the British Academy Film Awards alongside his digital doppelganger.
But first, fans will be able to see Liam’s avatar in a world-first real-time augmented reality mobile experience, created using EE’s 5G network. Liam’s performance will be recorded live and beamed to fans in the UK on their mobile phones wherever they are via AR app The Round.
Pip Brignall and Thomas Winsor, founders of The Round, developed the app as a way to enable live event producers and creatives to build and distribute their productions using augmented reality, and they describe it as a venue for live holographic performances.
“Our background is in theatre, so our ‘in’ for immersive tech was very much from a theatre perspective, and we started to explore how virtual reality and augmented reality could be used to enhance that live performative storytelling format,” Brignall tells TVBEurope. “When we first started working on The Round it really came out of Covid shutting down all the theatres. We realised that a lot of the obstacles that we were facing could be solved by creating this one platform. We wanted to explore ways that dramatic storytelling could evolve into the audience’s personal space. They’re inviting the drama to them rather than going to it.”
The duo were taking part in Digital Catapult’s 5G Accelerator Programme, experimenting on how they could utilise that technology to improve the platform and develop it further when the marketing team at EE found them. “5G is a huge part of EE’s immediate and future roadmap and they always try to do a big exciting campaign around the BAFTAs and they were looking for something that would be a world first. They got in touch and asked us what would be possible,” explains Brignall.
“It was quite funny. They saw the video of our prototype and emailed saying they wanted to do that, and we said ‘well that’s what we’re already doing’,” adds Winsor.
EE brought Payne on board, and the project developed from there. The AR version of the singer will be kept under wraps until 6.45pm on Sunday evening, but the app is available to download now with an AR holding animation that viewers can use to help them familiarise themselves with the process. Payne will perform exclusively for The Round at 6.45pm on Sunday before taking to the stage alongside his avatar to open the BAFTAs.
The avatar itself has already been created using photogrammetry. “We captured Liam’s image from every single angle you could imagine in a huge rig, and took lots of photos of his face and then we’ve turned that into the avatar,” explains Winsor.
The images were then turned into a mesh, which was converted into an avatar and touched up in Maya. “We’re using really innovative facial capture technology as well so when we were doing the photogrammetry shoot we got Liam to recreate the face shapes that he makes when he’s singing,” says Brignall. “When we were creating his avatar we had ready-made facial positions that we know are similar to what he does, so we then pre-programmed these blend shapes which were from an iPhone just strapped to his head. It tells the avatar that Liam is now doing blend shape number six and the face moves into blend shape number six. It has a series of blend shapes depending on what he’s doing. That is all transmitted in real time so that once we’ve rigged the avatar it moves and behaves in the ways that Liam himself does, his digital face is programmed to make the same facial positions. When it’s in real time that information makes the avatar come to life.”
A slightly enhanced version of the avatar is being used in the TV performance. “There are some limitations at this point as to what level of fidelity you can put an AR avatar into a mobile phone because they have a hard ceiling on what they can process. So we’ve created an ‘improved’ version of the avatar that people will see on the TV broadcast,” says Winsor.
Part of the process has also involved Payne pulling on the dreaded mo-cap suit. But Winsor and Brignall say he’s been up for anything. “We had a very early session with him when we went to our motion capture studio and we got him in a mo-cap suit, and had a play and started loosely blocking it out. At that point we captured some test data so that we could run that through our app, and we did the photogrammetry session in the afternoon,” adds Winsor.
“We had another session about three weeks later where we had our first version of the avatar so he got to see what it was all about and move his avatar around. In that session we tested everything was working. We did a dry run the week before last in the studio we’re using to stream the performance from, and then we’ve got him for one more day.”
So what does Payne make of his digital doppelganger? “He seems to love it,” says Winsor. “It’s a strange experience, someone controlling their own avatar in augmented reality. In the studio there’s a big screen where he can see his avatar and he’s controlling it and it’s really cool, and then when you see it in augmented reality, when you see a true 3D object in your space that you can move around, it’s very different. The last test we did with him, his management team were there and they all went crazy because they could move it around and see his back! They were really excited to be able to go and look at the back of a character, the place they would never normally see. It feels intimate, personal.”
Once Sunday is over, Brignall and Winsor’s focus will move to The Round’s the next stage of development. While the EE project will only be available in the UK, the app itself is available across Europe, and there are plans for more international roll outs. “We’ve got some very exciting projects that we can’t talk about,” teases Winsor. “We’re looking to get as many content partners on board as possible. In our first year it’s all about working with the best people we can find. Working with great talent, great artists to create incredible work and get people on board.
The second part will be a year of R&D on creation software that allows live event producers to build an entire production and operate it all in real time, whether that be a scene change, lighting change, sets etc. “We’re not expecting to deliver that until the end of 2022,” says Winsor. “All the learnings we take from our content projects will be fed back into that software until we make it just right to release.”
The Round app is available to download here.