Netflix has developed an automated quality control (QC) method to detect pixel-level artefacts in videos.
Designed to reduce the need for manual review, the solution identifies bright spots known as hot or lit pixels enabling intervention at an early stage in the production process before costly and time-consuming correction at a later stage becomes necessary.
Pixel-level errors generally fall into two categories; hot or lit pixels which appear as single frame bright spots and dead (stuck) pixels which do not respond to light. Having previously carried out work to detect dead pixel errors, Netflix has turned its focus to hot pixels, which are harder to flag manually.
The company developed a neural network capable of pinpointing pixel-level artefacts in real-time, at scale and with near-perfect recall rates. Detection requires the identification of small-scale, fine features in large images and the ability to differentiate between artefacts and naturally bright pixels with features that resemble artefacts, such as small lights, catch lights and other spectacular reflections.
To achieve this, Netflix designed a model to process large-scale inputs at full resolution rather than downsampling them in pre-processing, ensuring pixel-level errors remain detectable. Five consecutive frames at a time are analysed, providing the network with the temporal context needed to establish whether a bright object is intentional or a glitch. A continuous-valued map of pixel error occurrences at the input level is produced for every frame. In training, the maps are optimised by diminishing dense, pixel-wise loss functions.
During interference, the algorithm binarises the model’s outputs using a confidence threshold before performing connected component labelling to locate error clusters. Error locations are accurately mapped and reported, with the entire process taking place in real-time on a single GPU.
To create realistic training samples, a synthetic pixel error generator was developed to mimic real-world artefacts. These were superimposed onto frames from across the Netflix content catalogue. While synthetic data was essential for training, the model required multiple cycles of fresh, real-world footage to ensure precision. Iterative refinements took place as follows:
- Inference: Run the model on previously unseen footage without any added synthetic hot pixels.
- False Positive Elimination: Manually review detections and zero out labels for false positives, which is easier than labelling hot pixels from scratch.
- Fine-tuning and Iteration: Fine-tune on the refined dataset and repeat until convergence.
A process of continuous refinement aims to reduce false positives, preserving high sensitivity while minimising the number of false alerts which inherently occur given the volume of content being processed.
The need for hours of painstaking manual review has been minimised, said Netflix, with reviews now potentially taking place in minutes. The company is continuing to refine its capabilities through real-world deployments and working with partners to increase understanding of how pixel errors impact the viewing experience.
