AI-enabled ABR

[gehirndienst]

Hello!

I'm a research fellow currently working on ABR for real-time data using Deep Learning algorithms. Our focus is on developing real-time streaming extensions for systems like this one: https://github.com/hongzimao/pensieve.

To me, Oven software appears to be a straightforward yet very powerful environment for implementing a client-server or peer-to-peer system, where ABR (live transcoding on the sender side) is controlled by a Deep Learning-based algorithm.

To that, I have a simple question: how does OvenMedia (sender) determine when it's time to switch the video quality? In other words, which parts of the code on the sender and on the receiver sides (e.g., maybe from OvenLiveKit) are responsible for collecting network statistics such as throughput, latencies, etc., and making decisions on the next quality level based on those statistics? I would like to implement IPC scripts that support DL-based quality selection. Specifically, these scripts would analyze the collected statistics over a certain period and signal when it is necessary to switch to a specific video quality level. The idea is to deliver the video streams from an autonomous car's sensors to the operator in a "smart way".

Apologies for the basic question; my expertise lies mostly on the backend side, and I'm not very familiar with JavaScript...

Best regards,
Nikita

[getroot]

This question is more of an industry standard than an OME feature.

The [OvenMediaEngine -> Player] section is as follows.

OvenMediaEngine's WebRTC publisher uses REMB to change the rendition on the OvenMediaEngine server side with the estimated bandwidth notified by the client.

In OvenMediaEngine's LLHLS, players measure bandwidth estimation with their own algorithms along with various conditions (eg resolution) and change renditions.

The [OvenLiveKit --(webrtc)--> OvenMediaEngine] section is as follows.

It uses transport-cc bandwidth estimation, and the browser's WebRTC library changes the encoding quality (this is a little different from general ABR). This is done in the Webrtc Native Code below OvenLiveKit, so OvenLiveKit cannot change the algorithm. (Only min, max can be set)

[gehirndienst]

Hello @getroot !

Thank you for your response. I have two questions, and I would greatly appreciate it if you could take the time to clarify them as they will provide a comprehensive answer to my initial question.

1. [OvenMediaEngine -> Player] direction:
   Could you please indicate the specific methods in the code on both the player and server sides where the player measures the bandwidth and notifies the OvenMediaEngine? I'm interested in understanding how the player communicates this information. It seems that this notification process is the same for both WebRTC and LLHLS. If I can replace this notification with an external AI-based decision and make slight adjustments to how the OvenMediaEngine receives this notification to apply to perform the appropriate transcoding then, it would basically solve the task.

2. [OvenLiveKit --(webrtc)--> OvenMediaEngine] direction:
   Am I correct in assuming that OvenMediaEngine supports media grabbing in formats other than WebRTC? For example, if I deploy OvenMediaEngine on a device where video grabbing occurs (using a laptop's webcam streaming 4k UHD as an example), can I simply send the raw stream and have OvenMediaEngine handle the encoding? In this scenario, if the previous point is done, the encoding process would be solely controlledby my AI model in a form of periodic notifications with decisions coming from player. Or does OvenMediaEngine strictly require WebRTC as the input data format?

Thank you very much for your time and efforts!

Best regards,
Nikita