Use OriginMapStore for OvenMedia Edge to pull stream from RTSP server (Clustering)

[khaled-du]

Is your feature request related to a problem? Please describe.
Will be nice to have the ability to use OriginMapStore to pull stream from RTSP server based on a Redis key value.

According to the documentation OriginMapStore only being used with OVT for now.

Can this be done?

[getroot]

If you are the developer of the rtsp server and develop the server to register its own stream key and url to redis, it will be possible.

[khaled-du]

@getroot Thanks for your quick reply :)

We tried to setup this but it is failing.

Logs:
`[2022-12-08 03:11:24.929] I [SPRtcSig-T3333:238136] Signalling | rtc_signalling_server.cpp:231 | New client is connected: <ClientSocket: 0x7f99ef40cc10, #4, Connected, TCP, Nonblocking, 184.163.29.87:63955>
[2022-12-08 03:11:24.984] I [SPRtcSig-T3333:238136] Publisher | publisher.cpp:186 | Try to pull stream from local origin map: [#default#edm/khaled_123]
[2022-12-08 03:11:24.984] E [SPRtcSig-T3333:238136] Orchestrator | orchestrator.cpp:642 | Could not find Origin for the stream: [#default#edm/khaled_123]
[2022-12-08 03:11:24.984] I [SPRtcSig-T3333:238136] Publisher | publisher.cpp:190 | Try to pull stream from origin map store: [#default#edm/khaled_123]
[2022-12-08 03:11:24.985] I [SPRtcSig-T3333:238136] Orchestrator | orchestrator.cpp:549 | Trying to pull stream [#default#edm/khaled_123] from provider using URL: PullProvider
[2022-12-08 03:11:24.987] I [SPRtcSig-T3333:238136] RtspcStream | rtspc_stream.cpp:356 | SDP : v=0
o=- 0 0 IN IP4 127.0.0.1
s=Stream
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=J00AKKtAPAET8s1AQEBAgA==,KO48gA==; profile-level-id=4D0028
a=control:trackID=0
m=audio 0 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/48000/2
a=fmtp:96 profile-level-id=1; mode=AAC-hbr; sizelength=13; indexlength=3; indexdeltalength=3; config=1190
a=control:trackID=1

[2022-12-08 03:11:24.988] I [SPRtcSig-T3333:238136] Provider | stream.cpp:59 | RTSPCProvider Application has started to play [khaled_123(2277)] stream : RTSP://10.9.0.5:8554/edm/khaled_123
[2022-12-08 03:11:24.988] I [SPRtcSig-T3333:238136] Provider | stream.cpp:49 | #default#edm/khaled_123(2277) has been started stream
[2022-12-08 03:11:24.988] E [SPRtcSig-T3333:238136] OriginMapClient | origin_map_client.cpp:66 | <edm/khaled_123> stream is already registered.
[2022-12-08 03:11:24.988] W [SPRtcSig-T3333:238136] Provider | application.cpp:117 | Reject to add stream : failed to register stream to origin map store
[2022-12-08 03:11:24.988] E [SPRtcSig-T3333:238136] Provider | application.cpp:239 | Could not add stream : #default#edm/khaled_123(2277)
[2022-12-08 03:11:24.988] E [SPRtcSig-T3333:238136] Provider | provider.cpp:153 | RTSPCProvider Application could not create [khaled_123] stream.
[2022-12-08 03:11:24.988] E [SPRtcSig-T3333:238136] Orchestrator | orchestrator.cpp:565 | Could not pull stream [#default#edm/khaled_123] from provider: PullProvider
[2022-12-08 03:11:24.989] E [SPRtcSig-T3333:238136] WebRTC Publisher | webrtc_publisher.cpp:461 | Cannot find stream (#default#edm/khaled_123)
[2022-12-08 03:11:24.989] E [SPRtcSig-T3333:238136] Signalling | rtc_signalling_server.cpp:322 | Cannot find stream [#default#edm/khaled_123]
[2022-12-08 03:11:25.012] I [SPRtcSig-T3333:238136] Signalling | rtc_signalling_server.cpp:365 | Client is disconnected: HttpConnection(0x7f99ef40c210) : WebSocket <ClientSocket: 0x7f99ef40cc10, #4, Connected, TCP, Nonblocking, 184.163.29.87:63955> TLS(Disabled) (#default#edm / khaled_123, ufrag: local: (N/A), remote: (N/A))`

Edge Config:

<?xml version="1.0" encoding="UTF-8"?> <Server version="8"> <Name>OvenMediaEngine</Name> <Type>edge</Type> <IP>*</IP> <PrivacyProtection>false</PrivacyProtection> <StunServer>stun.l.google.com:19302</StunServer> <Modules> <HTTP2> <Enable>true</Enable> </HTTP2> <LLHLS> <Enable>false</Enable> </LLHLS> <P2P> <Enable>false</Enable> </P2P> </Modules> <!-- Settings for the ports to bind --> <Bind> <Providers> <RTSPC> <WorkerCount>1</WorkerCount> </RTSPC> </Providers> <Publishers> <WebRTC> <Signalling> <Port>3333</Port> <TLSPort>3334</TLSPort> <WorkerCount>1</WorkerCount> </Signalling> <IceCandidates> <IceCandidate>*:30000-40000/udp</IceCandidate> </IceCandidates> </WebRTC> </Publishers> </Bind> <VirtualHosts> <VirtualHost include="VHost*.xml" /> <VirtualHost> <Name>default</Name> <Distribution>edm-stream-edge-nodes</Distribution> <Host> <Names> <Name>*</Name> </Names> </Host> <OriginMapStore> <RedisServer> <Host>10.9.0.6:6379</Host> <Auth>*****</Auth> </RedisServer> </OriginMapStore> <Applications> <Application> <Name>edm</Name> <Type>live</Type> <OutputProfiles> <OutputProfile> <Name>passthrough</Name> <OutputStreamName>${OriginStreamName}</OutputStreamName> <Encodes> <Video> <Bypass>true</Bypass> </Video> <Audio> <Bypass>true</Bypass> </Audio> </Encodes> </OutputProfile> </OutputProfiles> <Providers> <RTSPPull /> </Providers> <Publishers> <AppWorkerCount>1</AppWorkerCount> <StreamWorkerCount>1</StreamWorkerCount> <WebRTC> <Timeout>30000</Timeout> <Rtx>false</Rtx> <Ulpfec>false</Ulpfec> <JitterBuffer>false</JitterBuffer> </WebRTC> </Publishers> </Application> </Applications> </VirtualHost> </VirtualHosts> </Server>

[getroot]

Sorry, it doesn't mean it's possible now, it means it will be possible in the future. (As it only requires a very simple fix) I'll work on this in the future when I have time.

[zt1983811]

@getroot I just hit the same wall as @khaled-du, I am wondering if you have the timeline to fix this stuff?
In fact, I am a little interesting in what the simple fix that you have in mind. I would like to create PR to fix this if possible.

Also by just go over fast the code I figured in orchestrator.cpp , function

CommonErrorCode Orchestrator::RegisterStreamToOriginMapStore(const info::VHostAppName &vhost_app_name, const ov::String &stream_name)

will return error if they end up not get client->Register(app_stream_name, ovt_url)
I am wondering if we should put a check that if the stream already exist, we always return success. Like that the error should not happened in above logs.

It may not be the best place to do it, please instruct if you have ideas on where it would be the best place to add this logic and I will raise a PR from there.

Thanks a lot

[getroot]

@zt1983811 Thanks for the suggestion. But it's easier for me to develop myself than to explain. I look forward to your next PR on other topics!

[zt1983811]

@getroot it has been more than a month (year :-)))), I am wondering if you have timeline for this because I am currently eager to get it work. If you are too busy, please just explain short what need to be done there so that I can wrap up a quick PR to solve this problem because I would expect more people will blocked by this. Thanks

[ilterpehlivan]

Hi,
I would like to also see this functionality in. we have RTSP connections and trying to build HA setup. @getroot I appreciate if you could give an ETA for this functionality ?
Thanks

[getroot]

@zt1983811 @ilterpehlivan

I haven't tested this yet, but maybe you can pull the RTSP from the OriginMapStore with the following temporary fix:

OvenMediaEngine/src/projects/base/provider/application.cpp

Line 118 in 9d64404

return false;

Comment out the above line. (return false;)

Please note that this is not a bug or problem. This is a new feature. Supporting an external rtsp origin server was not in my original plan as I originally developed this to be able to automatically scale the origin server on the cluster.

In order to officially support this, I have to think about how to handle it well. For example, there is a way to provide an option to turn on/off the automatic registration of streams in the OriginMapStore. I'll have to think about if there isn't a better way than that.

And this feature has a lower priority than other features on my roadmap. I will consider this later.

[getroot]

I updated the OriginMapStore to support RTSP URLs as well. You can test it with the latest master branch.

[khaled-du]

Thanks @getroot the change works fine.

[khaled-du]

@getroot Can we configure the Redis password as env variable? I tried using ${env:REDIS_PASS} but seems it is not working.

[getroot]

${env::} not working in Server.xml is another issue. But in my environment it seems to work fine.

Your example seems to have a different format.
Try editing it with ${env:REDIS_PASS:abcdef} .