ILM’s StageCraft volume system can create anything from rainy, dark streets in Gotham City to the sandy landscape of Tatooine. The company’s executive producer Chris Bannister talks to TVBEurope about how virtual production has developed since ILM first used it over 20 years ago, and where it’s going next
How long has ILM been working with virtual production, and what prompted that first attempt?
Industrial Light & Magic has been innovating in the virtual production space since 2001 when we developed the first real-time, in-camera, on-set previs system for Steven Spielberg’s A.I. Artificial Intelligence. Since then, we have been developing a robust set of tools in this space which in 2018 were incorporated into a platform we call ILM StageCraft. StageCraft encompasses all of ILM’s virtual production and performance capture tools including our advanced LED stage technology.
How has that work developed over the years?
Filmmakers come to ILM for the creative collaboration, world class artistic talent, and technological innovation. Our innovation is spurred by the needs of the filmmakers we collaborate with and we are always taking their feedback and suggestions and incorporating them into our solutions so we can meet the creative challenge while allowing the filmmakers to work in the way that makes them most comfortable.
What does ILM StageCraft actually do?
ILM StageCraft is a platform that encompasses a suite of virtual production tools and our class-leading performance capture solutions. Perhaps the most well known component is StageCraft LED, sometimes referred to as ‘The Volume’. In 2018, working with The Mandalorian creator and executive producer, Jon Favreau, along with our technology partners Epic Games, Lux Machina, Fuse, and Profile Studios, we created a system that combined a large-scale installation of LED screens, real-time rendering technology, camera tracking, ILM colour science, and the skill of ILM’s artists to create a new production paradigm. At its core, the technology provides a means to take any environment, be it real, imagined, or any combination of the two, and display it on LED screens for in-camera capture during principle photography. It has proven to be a game-changer in terms of production efficiency but also has a positive impact on sustainability and of course creative workflows.
In terms of performance capture, ILM’s tools such as Flux, and Anyma and Medusa which were initially developed by Disney Research Studios and now advanced by ILM allow us to offer our clients flexible solutions to high-resolution facial and body performance capture depending on the needs of their particular project or performer. We’ve developed solutions for use both on set during principle photography as seen in films and series such as Terminator Dark Fate, The Irishman and The Witcher, and off set performance capture as used on such projects as Aladdin, and Aquaman.
How has virtual production changed since you first began to use it?
We continue to evolve our virtual production technology and workflows on every project we undertake. The most substantial change is the power of the hardware that is available now versus what the state of the art looked like in 2001. Processors are immensely more powerful now and we are able to run dozens of systems in parallel to create a single environment. GPU’s have also come a long way and with the latest and greatest NVIDIA RTX technology we are able to create environments and effects that were unthinkable in those early years. Virtual production is now routinely used on a wide variety of project types, not just large visual effects-driven projects. We work on projects ranging from music videos, brand films, commercials, broadcast promos, episodic series, to feature films and they all leverage virtual production in different ways.
Are there any major differences between working with virtual production for a film project compared to a TV production?
In the current landscape, the experiences for film and TV are closer together. The key component of the service that we offer with ILM StageCraft is the people with the experience to help find the right virtual production approach for any project. In addition to our work on some of the Lucasfilm television projects, we have a close partnership with Disney Television Studios. We have provided virtual production solutions for features, television, commercials, music videos, and more.
Can you talk us through all the productions where you’ve used virtual production, and how you’ve used it?
We’ve used virtual production technology on a variety of projects including The Mandalorian and The Book of Boba Fett, where over half of the series were shot on our StageCraft volumes, The Midnight Sky for which we built a bespoke StageCraft volume to accommodate the interior set for the Barbeau Laboratory, and recently The Batman.
How involved has ILM StageCraft been in developing the technology side of virtual production as you’ve employed it more and more?
The ILM R&D team along with Lucasfilm’s Advanced Development Group (ADG) partners with our StageCraft teams to keep the feedback loop going and a constant flow of innovations coming in response to production demands. We are always looking to streamline the system and incorporate the specific tools that a given filmmaker might request on their project.
How do the creatives that you work with react to virtual production? Are they all onboard immediately?
As you might expect, each project and each creative team is different. Some creatives are more firmly rooted in traditional production and thus more hesitant to jump fully into virtual production. That said, once we demonstrate the capabilities and benefits of virtual production and answer all of their questions they’ve all gotten onboard and had a great experience. For us the key has been to set realistic expectations and educate our clients on how they can make the technology work for their production. Virtual Production is not a silver bullet that solves all production woes, to work effectively you need an experienced team who knows the strengths of the approach and where it may falter so you can plan your production accordingly.
How involved do they get in learning about the technology?
At ILM we don’t create technology for technology’s sake, we create tools to help filmmakers bring their vision to life. Some filmmakers are technologically savvy and want to understand every facet of how the systems work, others may not be as interested in the inner workings but rather what the technology allows them to achieve. We seek to put tools into the hands of artists and filmmakers that allow them to focus on the art of storytelling and the craft of filmmaking and we let the technology fade into the background. The technology is production-hardened and our teams’ job is to make sure we bring great ideas to the table, collaborate with our clients and keep the technology working so the filmmakers can focus on achieving their creative goals.
How does virtual production impact the post production process?
Virtual production as a whole has a tremendous impact on any production. Oftentimes a fair amount of work that would have traditionally been done during the post production phase of a project is pulled up to the pre-production phase. One example would be CG environments that need to be created ahead of shooting on a virtual production stage. Those environments are built, scouted, and lit all before principal photography begins on the scenes involving them. Having the environments available on the day allows for much better creative since the director can see exactly what the have to work with, the DoP can frame and light shots in context, and actors don’t have to imagine what a setting will look like as they did with traditional green or bluescreen setups but rather they can focus their energy on the performance itself.
What’s been the toughest challenge you’ve faced working with virtual production?
Frankly, other than partnering with filmmakers to share how best leverage the technology, the hardest challenge is making key creative decisions on a much larger world scale than you would with a traditional set that fits in a sound stage. The virtual environment could be infinite in scale and detail.
Are there any downsides to working with the technology?
While some aspects of virtual production can be applied to any type of project, shooting on an LED volume is not an ideal for every scenario. For example, if a scene is meant to be an exterior setting at high noon a filmmaker would likely be better off filming outdoors, on location, or a backlot to capture that type of intense lighting environment as it can be very difficult to achieve on a stage. Also, as a filmmaker you want all of your department heads to buy into the process. There are numerous decisions that have to be made up front and you want the key individuals and their departments to be aware of the process so there aren’t any surprises on the day.
What’s next for virtual production?
We are just scratching the surface of how virtual production can be a significant contributor to filmmaking and with each production creatives get more bold in the use of the technology and how our artists can support their vision. Things are about to get very exciting!
