For 40 years, video encoding has used traditional codecs, from MPEG 1 and 2 through the H.264 HD transition, then into the 4K era with H.265 and beyond, with VVC/H.266. Now, AI is sweeping the broadcast world, with pure AI or neural video coding on the horizon. InterDigital’s acquisition of AI compression pioneer Deep Render last October shows that the industry is paying attention and perhaps seeking to secure a foothold in the emerging field of AI-native codecs.
“Video compression has been a phenomenal success story up to now, video is everywhere and it works, and there are huge financial stakes involved,” states Thomas Guionnet, fellow research engineer at Ateme. “The progression of AI has been fast; 40 years of video compression history has been caught up in 5 years. One cannot ignore the potential of that. AI can either help us solve problems we couldn’t solve yet, like video content analysis, or address problems we can already solve, like video compression. The balance between the two is where it becomes interesting.”
Hybrid
Predictive machine learning compression techniques have been around for decades in the encoding field, evolving into what is now often described as AI. But in the short to medium term, few voices are claiming a full displacement of traditional codecs by AI. A hybrid model seems a more likely alternative, wrapping AI services around classic encoding.
Gwendal Simon, senior director of technology at Synamedia, doesn’t foresee a full replacement of the classic codec in the short to mid-term. “The value of AI currently lies in providing capabilities to make the streaming experience better. AI in pre-processing allows frame rate selection and quality improvement in specific areas of a picture. In post-processing, it delivers visual quality assessment going beyond VMAF in the live context”, he explains.
This allows Synamedia to adapt content visually, tailor its encoder to specific scenarios, and identify areas where quality can be increased. “It also supports the bitrate ladder for adaptive bitrate streaming. AI can help find the right trade-off between quality for end-users and cost for customers.”
AI-driven super resolution is another example of how artificial intelligence could enhance the video delivery process. At IBC last September, Beamr partnered with Nvidia to demonstrate a real-time AI-powered video pipeline that upscales HD content to 4K, using AI super resolution to predict and generate new pixels.
“You’re benefiting from the advantages of the classic codec, like being able to remove temporal and spatial redundancy as well as entropy coding, but can conveniently give a different experience at the edge with AI,” says Tamar Shoham, CTO at Beamr. “This addresses a gap in the market where content providers seek to offer premium quality without having to roll out 4K infrastructure at every venue. In the future, this may not be relevant but right now, you have legacy content and infrastructure, and AI can provide that 4K experience using efficient compression.”
Hallucinations
The use of AI in encoding raises a number of questions, fidelity being one. “There is a philosophical question around fidelity and similarity: when is the loss or addition of information within a picture important? Before deeming hallucination a problem, you must first decide what you want,” says Guionnet.
In a live environment, it’s hard to win trust and easy to lose it, warns Ian Wagdin, VP of technology and innovation at Appear. “There’s a lot of work going on around C2PA and authentication to ensure we’re not seeing AI-generated content on news output, for example. This raises questions about how authentic content is once it’s been through several layers of AI processing.”
Hallucination is an obvious barrier for the broadcast and film industries. This demands a careful examination of AI algorithms, according to Shoham: “One of the reasons we went with Nvidia for the super resolution is that its algorithm is more conservative, reducing the risk of hallucination in the encoding. At IBC, people remarked how solid and safe the end result looked and we now have customers examining advanced features.”
Walled gardens and power
Interoperability and support for codecs all the way down the chain, rather than an end-to-end encode by a single manufacturer, is an important consideration, according to Wagdin. “You need to be able to inspect, understand and decode the codec at every point in the chain,” he adds. “If you’re using new techniques, you need to be sure that at every point where you want to intercept that content, you can decode it. Getting there with AI takes a lot of compatibility all the way down the chain; you have to be strictly adherent to standards.”
The processing power required to do AI encoding will also be GPU-heavy, especially in fast-moving live action scenarios like sports. These are challenges that already exist with current codecs, but could be exacerbated with AI. “Can content be processed quickly enough? Is the processing power available where and when it is needed, particularly for remote production?” ponders Wagdin. “These solutions might be expensive to deploy in a world where people are already cautious about using GPU-heavy applications in cloud workflows.”
For Simon, the biggest barrier to AI adoption in encoding could lie within the decoder ecosystem. Any changes requiring the decoder on devices at the receiving end to do AI computation would need strong justification. Resource consumption within the decoder is also an issue. “This will have an impact on the number of channels we can encode at the same time in a given machine. It’s a cost to be considered.”
Slow transitions
The transition to pure AI encoding could happen if it delivers serious value, but right now, the benefits are not tangible. Research has been published on neural codecs, but none are commercially available yet, according to Guionnet. “JPEG AI, standardised in 2025 as the first purely neuronal codec, is an image codec. The transition to video introduces a new level of complexity: maintaining a specific frame rate presents significant additional challenges.”
AI might be slower to impact compression than in other areas, but it is still the future of video compression, he adds. “Right now we have great codecs, but we are hitting a glass ceiling that pure AI encoding will smash. Currently, the structure of a video encoder is sequential; there are things we cannot parallelise, and AI allows that. Things won’t be less complex, but will go faster.”
Ultimately, when considering a full replacement of classic codecs with AI algorithms, it’s important to think about how that video content is to be used, says Shoham. “In scenarios like autonomous vehicles, with massive amounts of video ingest from the different cameras and a 100 per cent AI oriented workflow, pure AI video compression is a tool that could possibly work,” she explains. “In classic fields, where you want to preserve every detail of the video and its artistic intent, I think we will be sticking to classic video encoding for many years to come.”
- This article originally appeared in the May/June issue of TVBEurope, available to download here.
