It may be 16 years old, but this year’s Strictly Come Dancing has been as popular as ever. Viewers have taken to social media to express their joy at the show that’s brought sparkle back to our screens, and helped lift everyone’s spirits.
It’s not been easy to bring the ballroom back in the year of the pandemic, and the production team deserve all the plaudits they’ve received for their hard work to get the show on air. But rather than sit on their laurels, Strictly’s production team have been particularly innovative by bringing augmented reality into the mix. From the racing cars in week one, to the elephant that appeared during Bill Bailey’s Quickstep, augmented reality has featured every week during the live shows.
The use of AR has been a real team effort, involving both the lighting and audio teams, as well as companies, Mo-Sys and Potion Pictures. While this year is the first time AR has been employed in the show, it’s something the team has been considering for a while. “We’ve previously used perspective in the floor to create the illusion that the dancers are standing on top of a lighthouse or wedding cake, or skyscraper,” explains Potion Pictures’ managing director David Newton, who also serves as the show’s graphics designer.
“That’s been really effective but the big drawback is you can’t move the camera. Likewise, we’ve had concepts where the production team wanted to place something in the scene, and we’ve done chess pieces, falling glitter and bats, but again you can’t move the camera. Using AR was a chance to expand on those concepts and bring everything together a little bit more.”
Newton was asked by Strictly’s producers to look into the possibility of AR, and chose to use Unreal Engine as they were already starting to use it for real time rendering. “Mo-Sys’s name kept coming up in relation to AR and camera tracking, and Epic Games said Mo-Sys have a plugin that works great with their software and we were comfortable with using Unreal so it all sort of added up.”
Everyone involved came together for a demo day at BBC Studioworks’ Studio D at BBC Elstree Centre. “They already had the little reflective stickers in the ceiling of Studio D that are critical to the tracking system so it was a logical place to do the demo,” explains James Uren, technical director at Mo-Sys.
Once the producers were happy with the demo and test days, planning got underway for the series proper. Eagle-eyed viewers will have noticed AR has featured in at least two dances each week. Amazingly, the planning for each dance only begins days before we see it on screen. “We pretty much always start on a Monday. We’ll have a meeting about it the previous Friday, and then on Monday we’ll start the work for that week’s show,” says Newton.
“One thing we were really keen on from the beginning was that the technology shouldn’t drive the concepts and the creatives. We wanted the creative team to come to us with the concepts in the same way they have previously done with the screen graphics. During the demo day we demonstrated some of the things that they could possibly do. Most weeks it’s very much led by the creative team. The show’s creative director Jason Gilkison and producers that work with him come to us with a concept and then it’s up to us to try and make that work in AR.”
Using AR looks great on screen, but there’s always the possibility that the couples will end up dancing right through it. How do the team ensure that doesn’t happen? Newton cites Clara Amfo’s recent jive, which featured an AR record player. “The first draft of the record player had it much further down stage, we thought the original starting position was going to be camera right. So we sort of changed how a record player works to actually have the arm on the other side. But it all changed, and it went further upstage and there were a few more meetings about where it was going to go, what colour it should be, what side the arm was going to be, how many letters of complaint we would get if the records spun in the wrong direction,” he laughs.
“One of the good things about using Unreal is that it’s a real-time renderer so you can make those changes quickly. It’d be great to have all the information uploaded at the beginning of the week and that not change but the realistic thing, especially this year on a show like Strictly, is that it’s a creative process and everything is going to continue to develop during the week. Using Unreal lets us react to that, without it taking 12 hours to render. At this point I should pay tribute to my colleague Joe Phillips, who’s the lead designer at Potion Pictures and has done all the Unreal work on Strictly this series.”
As well as the graphics, Strictly’s lighting is a key component of the augmented reality employed in the show. David Bishop, the show’s lighting director, says it’s been interesting to explore the relationship between lighting and AR. “In my mind there are two routes with AR, you have to either make something that is real, and therefore it has to appear on the screen as being absolutely real, or you have to make something that’s very clearly an ethereal dimension and isn’t meant to be real,” explains Bishop. “For example, if you’re inside an AR-created house, as we’ve used this series, and there’s a light bulb inside then the person that’s standing inside that house needs to be lit as though it’s coming from that fake light. That means I have to find a real light in the same direction, which sort of does the same job.
“On the flip side, when an AR object is in the studio it needs to appear like it’s lit with my lighting rig, so it’s reflecting the same angles and casting shadows as everything else in the studio,” he continues. “One major thing with the elephant that gave it that extra dimension was giving it a shadow, which we created in AR. We added a follow spot that you can see in frame pointing at the elephant, which is what we would do if a real elephant walked into the studio.
Bishop continues: “The tricky thing about that is that our spot ops can’t see the things they’re trying to point the lights at. So they’d be pointing the spotlight in one direction and then I would be saying left a bit, down a bit. It’s that sort of workflow that’s become quite new to us but it is absolutely key, getting the lighting angles right is what’s making the AR even more believable, and that’s certainly something that’s improved throughout the series.”
Another key area where AR has impacted Strictly’s usual production workflows is sound and this is where it gets really complicated! Andy Tapley, BBC Studioworks’ sound and technology manager and Strictly’s joint sound supervisor explains. ‘’AR introduces a nine frame (360 milliseconds) video delay into the transmission video output, so sound obviously has to match this for synchronisation. When in AR mode we effectively work in two time zones; time zero on the studio floor where everyone hears the live orchestra, and AR time zone in the control suites.
In the Production Control Room it’s essential for the director, PA and vision mixer to see and cut shots to the post-render camera outputs and delayed sound, but it’s also essential that everyone listening to talkback on the studio floor does not have the confusion of hearing delayed programme spill from the PCR speakers. To get around this the audio team cut the PCR speakers, and headsets are worn when in AR mode.
“There’s also an added complication of working out how and when to switch in and out of AR mode to avoid seeing a jump in vision and delayed/un-delayed sound,” continues Tapley. “We’ve done this by not just switching into AR mode for the dance, but for a whole studio sequence – bookended by VT play-ins where we can effectively hide the switch over. It’s a complicated workflow, but it’s working really well.’’ He also adds “a lot of the credit for working this out goes to my very clever joint sound supervisor colleague, Richard Sillitto.’’
“Andy, Richard and I talked in the beginning about how this would work and on several occasions, I had to ask him to stop because my brain was hurting,” laughs Bishop. “What’s really interesting from a lighting point of view is that in the room you’ve got different people trying to achieve different things, the augmented reality graphics team and my three programmers all sit in the room with me. My programmers have to work at time zero, no delay at all, so that the lighting change happens with the music, but I have to look at the show in delay so that I know what’s being seen at home. So within the same room it can get really complicated!”
We know audiences are enjoying this new aspect of Strictly’s Saturday night shows. What are the dancers themselves making of it? “I think the skyscraper that we created for HRVY and Janette has had the biggest and the best reaction so far,” says Newton. “Janette was quite heavily involved. We had to be very specific early on and say if you want to dance on a skyscraper these are the limits, otherwise you’re going to fall off. We were all talking pretty much every day, back and forth, as they were rehearsing and working out little bits of when stuff was going to happen. That’s been the thing that I’ve seen the most positive reactions online towards. A lot of people talk about the elephant as well.”
The technology has certainly been embraced by viewers, 62 per cent of respondents to a recent online poll by Newsround said it really brings the dances to life. Does that mean we’ll see it again when Strictly returns in 2021? “Strictly has always been the kind of show that’s pushed things forward and likes to try new things,” says Newton. “Right now, we’re very much thinking about one week at a time, and this year has been so completely different from any other. Having worked on the show for 10 years this year has been so very, very different. Hopefully next year we’ll continue to evolve. We’ve done a lot of hard work now in terms of working out how to use AR and how to integrate it into the show.”
