Amazon Web Services has unveiled a fully managed AI service that can turn live and on-demand video into vertical formats in real time.
AWS Elemental Inference uses an agentic AI application that analyses video and automatically applies the right optimisations at the right moments.
The service aims to help broadcasters get their content out to where viewers are watching, particularly social media sites like TikTok or Instagram.
“The key thing that we’re really excited about is the fact that we’re able to do this in low enough latency for live streaming, and that has not been done before,” Regina Rossi, head of product at AWS, tells TVBEurope.
“A lot of our customers have viewers who are watching content primarily on their mobile phones, and traditional ways of producing video content in a horizontal format don’t work as well for that kind of viewing behaviour.”
AWS Elemental Inference is able to convert horizontal video to vertical with around 5-10 seconds of latency, allowing audiences to watch live content on their phones as well as TV. “The reason for the latency is that broadcasters need to have the context of what’s happening in the scene. We want to be able to have enough video segments so that we can correctly attribute what is happening in the scene, so that either it’s a key moment we can identify for our customers’ clips, or it’s somewhere that we can say, this is the saliency mapping for the action that’s happening on the screen, and we can provide that as the appropriate vertical format,” explains Rossi.
The service works with Elemental MediaLive or MediaConvert for on-demand. It uses multimodal AI, enabling it to analyse both audio and video. “Part of what Inference does is derive the metadata that’s going to be meaningful for the user. So, in the example of the vertical output, we provide the XY coordinates for each of the frames back to the encoder, and the encoder then provides the video output in both the horizontal and the vertical format.”
The service has already been trialled by major broadcasters, including FOX Sports and NBC in the United States, although AWS is keen to stress it’s not just for sport and can be used for news on entertainment shows. Rossi says feedback from AWS’ customers has helped in the development of Inference. “If you look at the generation of viewers who are primarily watching content on social media and TikTok, they’re familiar and used to that vertical format. The idea here is that we want to be able to help our customers be able to provide similar scrolling experiences for their customers.
“In the past, for clips and highlights, you could do it after the fact, but then you have a delay and you lose out on any kind of viral moments that might be happening. This is able to solve that problem.”
Big Blue Marble has been named as a launch partner for AWS Elemental Inference, with the service integrated into the company’s cloud-native Cloud Video Kit platform.
