SHV stereo 3D experiments24 March 2011
Read TVBEurope’s in-depth report on how BBC R&D experimented with Super Hi-Vision capture of stereoscopic content with the ultimate aim of seeing if SHV could be used for 3D programme creation. A unique method of 2D-3D conversion is also being tested.
“With SHV we are targeting higher resolution 3D stereo for cinema and ultimately the ability to create an SHV programme in 3D,” explains lead technologist Oliver Grau, stressing that this is a research project only.
Last September BBC R&D used the rare chance to get their hands on an SHV camera to conduct experiments with 3D image processing. The results were part of its ongoing i3DLive project which aims to develop tools for multi-camera capture of live action and to enable a virtual camera to synthesise views from any angle.
Two tests were made. The first arrayed two HD studio cameras on either side of the SHV camera in a narrow baseline configuration. The HD cameras were ramped up to 59.94fps to match the frame rate of the SHV, the signals were genlocked and a few minutes of recording made.
The 16 single HD SDI streams from the SHV were stored on a local NHK-developed device. Up to 4 minutes uncompressed video can be captured and up to 2hrs of video stored compressed onto a solid-state flash system based on P2 developed by NHK. After recording the playout was transferred to the BBC’s own experimental uncompressed HD ingest system. In post production the images were automatically processed, by a bespoke software algorithm, into stereo 3D.
According to the BBC, this software finds corresponding image patches in the left and right view (similar to motion detection). The closer an object is to the camera the further apart are the corresponding image patches in the right and left view. Unfortunately finding the corresponding patches is not an easy task, says Grau.
“When you reconstruct 3D information from 2D images one of the problems you have is a loss of information, the main one being occlusion,” says Grau. “If you use the original SHV 2D material as a left eye and then create a right eye image from that material for processing into 3D there will be objects in the foreground, obscuring part of the background visible in the other camera for this parts the depth cannot be determined.”
The BBC’s approach differs from 2D-3D conversion which usually works from one camera. Instead it uses stereo matching, or 3D reconstruction, from multiple cameras which can be done completely automatically.
“By using three HD cameras – especially a very high resolution SHV capture – there’s a better chance of being able to ‘fill in’ that missing visual information,” Grau explains.
A second test, using a wide base line configuration, arrayed ten HD cameras in slightly higher positions than the SHV camera. The results were used to create a 3D computer model of the live action which can be viewed from any angle – even those that would previously have been impossible without a camera appearing in shot.
“From a typical stereo setup you can generate only a profile of the scene (image plus depth information) so that if you move more than 90degrees around the virtual scene in a computer graphic program you will reach the outer limits of the model and see that it’s not solid,” he says. “By using a wide baseline we’re able to generate a 3D model which allows for free-viewpoint video – you can freely move your viewpoint around the scene as it plays out.”
Piero, the BBC developed sports analysis package uses currently one camera image in freeze-mode to visualize sport incidents. Future generations of such a product could be enhanced using the wide baseline model, explained Grau. The SHV camera of course provides a startling level of detail, of faces for example, which conventional HD cameras can’t capture.
“One aim is to see if we can generate 3D and special effects such as crane shots and steadycam without the actual SHV camera moving,” says Grau. “Another outcome we are researching is whether we can automate the process of 3D conversion from one high-res camera, with additional lower quality cameras. With single channel filming you can use the conventional acquisition and production pipeline until conversion in post production.”
A key partner in the i3DLive project is The Foundry which has a keen interest in developing 3D post tools. Its Nuke compositor is already widely used by facilities for 2D to 3D conversion.
Grau says BBCR&D also took away greater knowledge of the SHV unit, noting that because of the sensor’s size and wide field of vision “You tend to look everywhere and expect the scene to be sharp and in focus everywhere. So you have to stop down the iris to obtain a larger depth of field. That has clear implications on light levels, but it is actually in favour of good 3D effects.”