The brand new season of The Crown has employed deepfake technology for the first time on a sequence involving Erin Doherty’s Princess Anne.
The sequence involved replacing the face of an equestrian athlete who stood in for Doherty. For several shots in which Anne is taking part in a show jumping competition, the team at Framstore, who worked on the sequence, employed a deep fake technique to achieve the replacement.
Framestore’s deputy head of 2D and the project’s 2D Supervisor Ollie Bersey tells TVBEurope they opted to use deepfake because “it was going to do the best job in the in the shortest time.”
He adds, “The deep fake isn’t suitable for everything, but it seemed to fit the bill. We had lots of images that we could feed into the machine learning computer. There was no shortage of reference of the faces we were replacing and the faces we wanted to regenerate, so it seemed the best approach. It’s quite an exciting new approach that we wanted to test.”
Bersey adds that he thinks deepfake technology will continue to develop for film and TV productions. “When we first started looking at this, there were a number of limitations in resolution available, and these were kind of improving as we went along,” he explains. “There is no reason why this isn’t going to become a much bigger thing. We’re using it on on other projects, we’ve used it in commercials. It’s quick, and it’s successful. It’s going to be used and it’s an important part of the future. In The Crown, we did actually have to get in there with good old fashioned 2D comping and make further adjustments to the face because the performance of the stunt double wasn’t quite going to achieve the correct output.”
Framestore once again worked on multiple shots and sequences for season four providing VFX work on 230 shots, from creating digital environments and crowd work to a photoreal CG stag. Back in 2006, the company created the stag featured in The Queen. However, for The Crown it was a whole new piece of work. “Internally we didn’t have a stag asset that we could kind of lean into or pick up and work up from there,” says Ahmed Gharraph, Framestore’s head of CG.
“I think partly what helped win us the job is that last year we delivered a commercial with a photoreal reindeer in it. Reindeers and stags are roughly of a similar ilk, antlers and furry and whatnot. I guess that project kind of gave the client faith that we can make these photo real creatures, as well as our history of creature work in projects such as His Dark Materials. Creating photoreal creatures is in our blood. When this sequence came in we were particularly excited about it, because The Crown is not a visual effects heavy show. It’s embedded in history and embedded in the real world so there was no room for error. That made it a really exciting challenge because ultimately we were aiming to create something that should hopefully fool the audience into thinking what they’re looking at is real footage.”
In order to create the stag, some of the Framestore team visited Richmond Park in London to hang out and observe the animals. “Really it was endless amounts of reference, just constantly finding pictures and videos. It’s not something that you just study at the start and you go, right, I know what the stag looks like, now I’m going to go off and make it work,” adds Gharraph. “As the shots and the asset develops, we seek out new references and find little subtle nuances of how that anatomy is put together, how the stag might step in a certain way. It’s an iterative and subtle process.”
“If you look at it in terms of the workflows, it’s storyboarded, and then there’s a reference pass of that, and then there’s an animation block of that where you develop animation cycles, and you put them in the shot,” adds Ross Burgess, head of animation. “Then you speak to the client to see if that’s what they were thinking for pacing and timing. It’s like a piece of sculpture, you just chipping away at the blocks until you finally get there. On this project it was actually a very organic process, there was lots of iterations, but it only got better as we chipped away at it.”
“There’s a particular shot that kept me awake at night,” says Burgess. “It’s when he gets shot. We were battling with the idea that what usually happens when stags get shot is that they go down, they collapse and never get back up again. Obviously, that would just ruin the story of what we’re trying to do because our stag gets up and runs off. So we were asking ourselves how we were going to take the realism of what a stag would actually do, and giving it a sense that it could get back up again. What we had to do in animation to kind of force that was a real challenge, because it’s not something that there was any reference for. We had to traditionally keyframe it and it was very difficult.”
Other sequences the Framestore team worked on that they found challenging include turning 21st century Manchester into 1980’s New York. “New York was quite a challenge,” says Standish Millennas, Framestore’s VFX producer. “There was a lot going on in our new New York scene. We had to make the streets of Manchester look like New York. The set dressing was very good, up to a certain level, but we’re looking straight down what’s meant to be avenues of New York so there was an awful lot to fill in.”
“We did a lot of crowd work, which was very challenging,” adds Bersey. “We used partical sprite techniques to populate areas like the Sydney Opera House. There’s lots of crowds in there. We built crowds of 15,000 people out of very few sprites, employing various techniques, colour correcting their wardrobe, retiming them. randomising them adding lots of flags and stuff. We only had a handful of sprites for those shots and we made them go a long way.”
Framestore has worked on The Crown right from season one. How would the team say technology has developed during that time? “I wouldn’t say it’s the technology that has changed so much as the shots that The Crown production team are taking on have become bigger,” explains Bersey. “The conventional work that we’ve done for them, that kind of largely 2D based extension of sets, things like that, hasn’t really changed and the techniques used to execute them haven’t really changed. They’ve become more efficient and we’re doing them at a higher resolution. We’re certainly adopting new techniques for the crowd scenes and stuff like that. But they’re willing to take on stuff like the creature now and that is that is new ground for The Crown.”
Millennas adds, “I think in season one The Crown production team weren’t too experienced with visual effects, and as the seasons have gone on, their requests have become bigger and more confident as they realise that they can put their faith in visual effects. They’ve just got more and more confident in what their VFX supervisor has told them we can achieve without it looking like a big visual effects spectacular. The biggest development throughout the seasons is that now when they’re on set they’re keeping visual effects more in the forefront of their mind, rather than it being an afterthought.”
Finally, has the team at Framestore begun planning for season five? “We know that on the client side they’re starting to talk about what they might need,” says Millennas. “I think we’ve just been given a heads up, but we haven’t actually gone into anything in any kind of detail at all. We’ve only just finished season four!”
“We get lot of pride out of the show when it comes on air and you hear how people talk about it and how much people love it. A lot of people talk about the production quality of the show, it’s up there as a benchmark for really good production values. So you definitely want to have your name associated with that.”
