When Riot Games wanted to add an eye-catching augmented reality extravaganza to the Pro League Finals event based on their League Of Legends game characters, the company decided to partner with The Future Group and their innovative Pixotope mixed reality platform.
League of Legends can be described without exaggeration as an internet phenomenon. With over 100 million monthly players, the game’s creators, Riot Games, generate annual revenues in excess of $1.5 billion. The team-based strategy game enables players to choose a “champion” to play as. With a unique back story and specific skills, each champion helps drive player affiliation in the same way that real-world sports players attract a fan base.
The venue for the live esports event was Shanghai’s 18,000-seat Mercedes-Benz Arena and the pinnacle of the augmented reality content featured Akali, one of League of Legends’ most popular champions, singing and dancing with real dancers on stage, before being interviewed live in front of the packed arena. The Future Group’s Pixotope systems were to be found at the heart of the AR action.
Augmenting Akali
Kevin Scharff from Riot Games explains why they wanted Akali to visit the real world via augmented reality at the live finals event: “We felt there was an opportunity to utilise AR to bring a live, real-time performance of one of our champions, which became the live interview of Akali with Candice, one of the event hosts. We found that the most compelling cases for AR character use had taken place in live, unscripted expressions. There is something magical and authentic when a virtual character interacts ‘in the moment’ that goes beyond a fully scripted and pre-recorded expression. AR allowed us to marry the virtual character to the real world.”
It is the extraction of game characters into the real world that is so important to the diversified IP expressions and fans experience for Riot Games. Thomas Tang from the Riot Games IP & Creative team explains, “Riot has a great reputation of being a trendsetter for game and esports music. Over the course of 10 plus years, the company has built many music legacies including the hit champ MV Get Jinxed and phenomenal worlds anthem Legends Never Die. Creating a super edgy virtual idol band called K/DA and launching its first live performance on our Worlds Final Championship was another big bet the company made in 2018.”
Sophie Xiao from Riot adds, “Virtual idols are very popular in China, and the technology and people’s understanding of virtual idols are maturing super-fast day by day. We believe Riot is uniquely positioned to take another leap of faith and do some more experimental AR music endeavour in 2020. The goal is simple, keep building up our virtual idols and make them feel truly ‘real’ and authentic, as if they are real human beings. Real time motion capture and live AR is not only visually stunning but also highly interactive and emotionally engaging. This is the less travelled path we want to take this time.”
She continues, “Therefore, bringing Akali, the most popular champion from the virtual band K/DA back to the LPL Summer Final stage, seemed to be a great opportunity to further deepen our IP and technology, celebrate our hard-core fans, and hopefully reach more pan-entertainment audience in China.”
In reality, Akali is a CG character from a game and so, even though augmenting a CG character to shots from the live event doesn’t sound that difficult, there is a great deal involved. The level of interaction with show lighting, free moving cameras, recorded and live motion capture, as well as facial expression capture with voice lip-sync, led to a uniquely complicated scenario to achieve the end results.
Kevin Scharff continues, “To complement the Riot team’s efforts in developing the character assets for the event, we called in The Future Group, who were such fantastic partners at Worlds 2018 in Korea, to help bring the AR experience together.”
The Future Group is the developer of mixed reality platform Pixotope. However, they are more than just a software engineering company. They retain expertise across all the various disciplines that are called into play when creating an augmented reality event. This enables them to work closely with their customer’s creative departments to bring technically complicated ideas to life, while also having the capacity to suggest alternative routes and how to achieve the maximum impact from ever-advancing technology.
For the Riot Games event, The Future Group deployed a team of five augmented reality experts, including chief creative officer, Øystein Larsen and senior product specialist, David Stroud, to assess and solve the various challenges.
The Future Group specified and installed four Pixotope augmented reality platforms. Each system generated real-time augmented material and combined it with a live action background. There were three Pixotope systems connected to three different camera feeds, plus one more in reserve as a live backup. Each Pixotope contained a digital model of the Akali character and stage assets, as well as a virtual tracked camera which matched its real-world counterpart.
It’s all about the lighting
A significant challenge to solve when adding an augmented character is to match the lighting of the background scene. As Larsen explains, “Failure to achieve this accurately leads to characters that look ‘stuck on’ and not embedded into the real-world scene. The specific challenge for this event was that Akali was to perform a dance song under pulsing show lighting that synchronised with a music track. Added to this, projected images were mapped to the arena floor.”
Scharff adds, “The Future Group utilised their expertise to perfectly immerse our character and performance assets to feel fully integrated into the live experience.”
David Stroud, Pixotope product specialist, explains, “We used high dynamic range (HDR) site photography to accurately map the real-world lighting rigs to the virtual world created within Pixotope, where the digital Akali would be lit before being composited over the live camera shots. The arena lighting was pre-programmed and driven by timecode, as were the virtual lights within Pixotope, to ensure synchronicity.”
A secondary function of lighting is the creation of shadows. The human eye is curiously adept at calling out “fake” images where shadows have not been correctly simulated. To avoid this, Pixotope enabled the use of “ray traced” shadows for Akali’s interview. Computationally time-consuming ray tracing is normally the domain of long-render-time post production. However, thanks to Pixotope’s full use of the Unreal Game Engine, together with The Future Group’s bespoke custom rendering pipeline, these highly accurate shadows could be produced in real time. Since the show was broadcast at 59.94 frames per second, challenging mathematics had to be completed and composited over every single frame of the live action in less than 17 thousandths of a second.
Expanding more on the subtleties of shadows, Larsen says, “Apart from shadows that fall on the floor, there are also what are known by game creators and CG artists as ‘self-shadows’. These are shadows that are cast from one part of a CG model onto another part of the same model. While these significantly increase the photo-real attributes of a generated image, they are often the first casualty when ensuring that images can be produced in real time. However, with Pixotope’s efficient render pipeline, Akali’s baseball cap correctly and accurately cast self-shadows over ‘her’ face and reacted perfectly to the lights as her character moved, changing the direction of the shadows in respect to the light sources.”
Bringing Akali to life
One of the key aspects of this production was that Akali had to move realistically. Specifically, she first had to dance in sync with real-world dancers and then later, had to respond to a live interviewer and audience. The dance moves were directed and choreographed by the Riot team and then motion captured in Los Angles by Animatrik Film Design. The movement data was combined with the Akali character model within Pixotope and produced in real-time to a timecode reference to ensure synchronisation with the music track. For the interview sequence, Pixotope replaced the pre-recorded dance motion capture data with a live motion capture stream derived from an actress performing off stage. This enabled Akali to generate moments of interactivity and spontaneity which were so sought after.
Apart from the full body motion capture, Cubic Motion’s Persona system streamed facial capture data live from the off-stage actress, who was also providing the voice for Akali. Of course, the arena was a very noisy environment so in fact the body and face motion capture, together with audio recording, took place some 100 metres away in a sound proofed anechoic room. Pixotope combined both the body and facial motion capture feeds to drive the geometry of the CG Akali model. This in fact required some specific intervention from The Future Group’s technicians. When making games, motion capture data is recorded, cleaned up and stored within the game to be recalled during game play. With live events such as this, the motion capture data needs to directly drive the CG geometry live.
David Stroud reveals that the complex motion capture setup was extensively tested by The Future Group alongside all the creative and technical partners in Manchester, UK for several days prior to the live event. “We built the systems in Manchester and were joined by the actress who would provide both the voice and actions for Akali. Together with the creatives from Riot, we helped her perform the appropriate level of expressiveness and to get used to working in isolation away from her interviewer. Also, we were able to work through and solve the various technical challenges, such as synchronising body and facial motion capture data together with vocal audio.”
Kevin Scharff adds “We worked together across three continents for this experience – in Los Angeles for the original motion capture of the dance beat, in Shanghai for casting, modelling, location scouting and pre-visualisation, in Manchester for a technical test and capture of the live interview, and finally in Shanghai for the live event. The key to the success was the collaboration and seamless integration of Riot teams in LA and China with world class partners in The Future Group, Animatrik and Cubic Motion.”
Akali in the real world
Having completed the lighting and motion setups for Akali, the next challenge was to ensure that she would be rendered from a viewpoint that precisely matched the carefully choreographed live-action dancers. Øystein Larsen states, “In fact, there were three different live action shots to match; one from a moving camera mounted on a jib arm, one on a free roaming Steadicam and one high above the arena. Each camera feed was sent to a dedicated Pixotope system, along with camera positional, directional and lens information, encapsulated in a data stream provided by camera tracking specialist, Stype.”
Each of the three Pixotope systems then created a matching camera inside their respective virtual scenes to “photograph” the digital, moving, lit Akali model and composite her perfectly onto each of the respective camera feeds. Those feeds were then sent to the broadcaster’s production switcher, just as normal camera feeds would otherwise have been. The resulting programme was broadcast both externally as well as shown on giant screens within the arena so that the attending audience could enjoy the augmented reality experience.
Expect the unexpected
Live broadcast and events inevitably throw up last minute and unexpected challenges to solve. As the venue was sponsored by Mercedes Benz, two of their cars adorned the stage and periodically, went for a spin – dislodging and scattering precisely laid out AR tracking markers in the process. Kevin Scharff remembers, “I can still hear our AR camera lead Luka from Stype screaming in anxiety each time those cars started moving around and him scrambling to recalibrate the AR setup each time. But you know what was magical? All along the way, the team, Rioters and incredible partners, were total professionals throughout it all.”
The Future Group’s David Stroud also recalls a moment that demonstrates the unpredictability of live events. “When it came to the live dance performance, the Steadicam operator chased and framed the augmented Akali character perfectly. But then after the event, he seemed quite considerably annoyed, which was odd given how well it went. It turned out that the augmented reality video feed to his monitor somehow became disconnected from the signal we were providing, so he couldn’t see the character he was supposed to be framing at all. He shot the whole performance by remembering Akali’s movements from the rehearsals!”
Stroud adds, “It is so important in live events and broadcast situations to have technology and people that can react quickly to evolving issues. The Pixotope system we make and the expertise we have is the product of many adventures in AR production. There’s always some unexpected element that needs to be accommodated.”
A highly successful event
The event was incredibly successful, generating millions of views via broadcast and lasting social media engagement. The live interview with Akali, complete with real time ray-tracing at 60 FPS, was ground breaking.
Kevin Scharff states, “The Future Group was critical to the success of the event in the use of their Pixotope technology. They ensured the core of the experience – the fantastic and unexpected arrival of Akali who then fully integrated into a killer dance with live performers and the magical illusion of Akali and Candice having a post-match interview shined front and centre.
“The magic trick was not the point though. What I love about The Future Group’s implementation of AR is that it met our goals of creating this fantastic immersive player experience of having a virtual idol perform at the biggest regional League of Legends event in China. From the visual smoke effects to the matched lighting of her glow stick, they ensured Akali looked and felt present at the event. They also solved every crazy hurdle that came upon this project, all with total professionalism and a great sense of humour. They are a stacked team of creative and technical wizards, that is for sure.”
Larsen concludes, “It was a privilege for us at The Future Group to be so central to such an amazing event. Riot Games are a great client. They are bold and inventive and allow us to really push the envelope as to what is possible in live AR, while also helping to lay the foundations as to what incredible things come next.”
