Virtual production is no longer a new concept for the media and entertainment industry. Having started life as the latest toy in Hollywood, the technology is now being used on everything from Coronation Street to Uber’s TV adverts.
But it’s not cheap. And this is the focus of BBC Research & Development’s Osmia VP project, led by Graeme Phillipson and Hell Raymond-Hayling. Virtual production isn’t a new area of research for BBC R&D; the team has been investigating its potential for several years, but now the aim is to look at how it can be deployed on a wider range of productions without increasing the budget.
Osmia VP has just been demonstrated at IBC, but began life as part of EMERALD, an EU and UKRI-funded accelerator programme to design tools that lessen the energy usage in the media production sector. Phillipson and Raymond-Hayling put virtual production at the centre of their research. “For productions which can justify the great startup cost, the appeal of virtual production is that if you can swap out some of your workflows to do more studio shooting and less location work, then you can bring production costs down,” explains Raymond-Hayling. “Equivalently, you’re also going to bring down the cost of the energy usage. We found that while it varies depending on the programme, a third to half of the cost and the carbon contributions are travel. So, our aim for that project was to work on making virtual production more accessible.
“With the Osmia project, the idea was that in theory we can make all these savings for energy and costs, but if we could bring them to more productions, then that is meeting the aim of reducing the environmental impacts of TV production. So that’s where that project came from.”
As an end-to-end workflow, Osmia VP covers creating virtual sets from stills of a real location, live camera tracking and rendering the virtual environment in-studio.
The background asset can be “anything”, according to Raymond-Hayling, but the pair have been particularly focusing on Gaussian splats to demonstrate how virtual backgrounds could be done cheaply and quickly, by existing production staff. “Gaussian splats are an active area of research, and new models and techniques are being released all the time. To produce a virtual background, we would choose the most recent and suitable technique to train a splat from 2D images of a real location. Once you’ve got a Gaussian splat, it can be imported into 3D rendering software to become a virtual background.”
The team created their own Gaussian splat renderer, driven by camera coordinates that are produced live. “You can also use your mouse as normal to explore the 3D world, and we’ve adapted it to do off-axis projection, which is the key to aligning the foreground and background”
Learnings from robotics
Raymond-Hayling describes the camera tracking module as being like a “robot vacuum cleaner”. It looks around for landmarks and ascertains where it is based on what it can see.
“Either those landmarks are installed like in the Mo-Sys system where markers are installed on the ceiling, or there’s the Sony Ocellus system which uses natural features; it looks around the studio and just sees whatever it can. Our basic idea is the same, the tracking module scans the environment and then makes a judgment about where it is relative to what’s in front of it. And if the system’s calibrated against the screen, then you can report an appropriate camera coordinate to the renderer.”
To deliver the calibration, the pair turned to established mathematics and algorithms from the field of robotics, notably SLAM: Simultaneous Localisation and Mapping. “It’s what a Mars rover would have on board, which is sent off to explore Mars and builds a map of its surroundings,” they explain. “We sat down with some relevant papers and spent a long time with our heads in lots of mathematics and computer code.
“There’s a reason that no one has done this because these implementations for the most part are academic,” they continue. “There is a lot of engineering needed to make these components work together. We decided that we would combine an off-the-shelf renderer, SLAM, and Gaussian splats. But what we’ve been doing is adapting these things to connect together into an end-to-end system we’ve produced.”
This is where their role as public service media researchers is imperative, adds Raymond-Hayling. The team is working to implement various existing technologies for public service use cases. “We were adapting robot control software, connecting it to an open-source renderer and using Gaussian splats to make the first version of Osmia. People are getting excited about it, and we are now in the phase of stress testing and replacing functional components to work towards a live production trial.”
Asked if the technology employs any form of artificial intelligence, Raymond-Hayling states, “Yes, multiple types”. The project components are a mix of classical machine learning and computer vision, together with Gaussian splatting, which uses more modern AI techniques. “The reason that we can do this project is because of the advances in AI that have allowed Gaussian splatting research to accelerate,” they add.
“This project isn’t built on any generative AI models. That’s really important to us. It means users will own the IP related to the backgrounds created with Gaussian splatting. This is because a Gaussian splat is essentially a 3D reconstruction of a collection of 2D stills, which you have full ownership of.
“The SLAM algorithms are AI techniques which are specifically designed to work on lightweight hardware because they are designed for robots and small microprocessors. So it’s a really smart, low-cost AI system, but it’s not generative AI at all.”
Plans for the future
Osmia is currently under active development, with the first version available as a demonstration. The BBC Natural History Unit has been involved in development, helping Phillipson and Raymond-Hayling on a consultancy basis. But they are keen to find partners to work on a production trial. “The way we work in BBC R&D is we don’t like to make something complete to then give people; we develop through partnerships. We can communicate what it is and what it can do, and then we try to partner a prospective end user, so that we can iterate together, and know how to best focus our energy and resources”
There are already plans being drawn up for Osmia VP’s future. “Our goal is to put this out into the ether and see what people do with it,” explains Raymond-Hayling. “Are we open to working with people from other companies and production companies? At the moment, we are open to trialling this with anyone who makes programmes for the BBC. But we intend to make this more widely available when it’s matured.”
Click the link to watch a video explaining more about Osmia VP: https://www.bbc.co.uk/programmes/p0nz393w
