The technology team at Netflix has publicly released two research papers regarding its work in exploring the use of AI in video editing.
On their blog, the team revealed they have been working on two models:
- Vera, a layered video diffusion model that generates only what needs to change as separate edit layers while leaving the rest of the video untouched.
- Void, an inpainting model for video object and interaction deletion. It performs physically plausible inpainting in complex scenes: rather than just removing an object, it reconstructs the scene as if the object were never there.
Netflix first revealed details of Void, which it developed in collaboration with Sofia University, in April.
Vera works by generating an edit layer and alpha matte after being given a source video and text instructions. The layers are then seamlessly composed with the original footage to produce the final edited result, said the blog.
“While Vera and Void show promising early results, reaching production-ready quality will require addressing several limitations,” said Netflix.
The research paper for Vera is available here and Void here.
