ABR with GPU also use CPU #1125

Pernifloss · 2023-03-09T11:12:31Z

Pernifloss
Mar 9, 2023

Hello,

Do you know why the CPU usage of OME with ABR is so high?

It seems my gpu is doing the encoding and decoding, but the cpu is running at 70%.

CPU usage is proportional to the number of incoming stream, and stay high for the whole duration of the stream.

My config :
I have OME running on a docker on Ubuntu 22.04.1 LTS

i built the docker based on [v0.15.1]:
docker build --file Dockerfile -t airensoft/ovenmediaengine:dev --build-arg GPU=TRUE build was sucessfull.

I run it with !
docker run -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video -d -p 1935:1935 -p 3333:3333 -p 9000:9000 -p 8081:8081 --gpus all airensoft/ovenmediaengine:dev

my Server.xml :

<?xml version="1.0" encoding="UTF-8" ?>

<Server version="8">
    <Name>OvenMediaEngine</Name>
    <Type>origin</Type>
    <IP>*</IP>
    <PrivacyProtection>false</PrivacyProtection>
    <StunServer>stun.ovenmediaengine.com:13478</StunServer>

    <Modules>
        <HTTP2>
            <Enable>false</Enable>
        </HTTP2>

        <LLHLS>
            <Enable>true</Enable>
        </LLHLS>

        <P2P>
            <Enable>false</Enable>
            <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
        </P2P>
    </Modules>
    <Managers>
        <API>
            <AccessToken>your_access_token</AccessToken>
        </API>
    </Managers>
    <Bind>
        <Managers>
            <API>
                <Port>8081</Port>
                <WorkerCount>1</WorkerCount>
            </API>
        </Managers>
        <Providers>
            <RTMP>
                <Port>${env:OME_RTMP_PROV_PORT:1935}</Port>
                <WorkerCount>1</WorkerCount>
            </RTMP>
        </Providers>

        <Publishers>
            <LLHLS>
                <Port>${env:OME_LLHLS_STREAM_PORT:3333}</Port>
                <WorkerCount>1</WorkerCount>
            </LLHLS>
        </Publishers>
    </Bind>

    <VirtualHosts>
        <VirtualHost include="VHost*.xml"/>
        <VirtualHost>
            <Name>default</Name>
            <AdmissionWebhooks>
                <ControlServerUrl>xxx</ControlServerUrl>
                <Timeout>3000</Timeout>
                <Enables>
                    <Providers>rtmp</Providers>
                </Enables>
            </AdmissionWebhooks>
            <Host>
                <Names>
                    <Name>*</Name>
                </Names>
            </Host>

            <Applications>
                <Application>
                    <Name>webtv</Name>
                    <Type>live</Type>
                    <OutputProfiles>
                        <HardwareAcceleration>true</HardwareAcceleration>
                        <OutputProfile>
                            <Name>bypass_stream</Name>
                            <OutputStreamName>${OriginStreamName}</OutputStreamName>
                            <Playlist>
                                <Name>For LLHLS</Name>
                                <FileName>abr</FileName>
                                <Rendition>
                                    <Name>Bypass</Name>
                                    <Video>bypass_video</Video>
                                    <Audio>bypass_audio</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>HD</Name>
                                    <Video>video_720</Video>
                                    <Audio>bypass_audio</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>SD</Name>
                                    <Video>video_576</Video>
                                    <Audio>bypass_audio</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>VSD</Name>
                                    <Video>video_160</Video>
                                    <Audio>bypass_audio</Audio>
                                </Rendition>
                            </Playlist>
                            <Encodes>
                                <Audio>
                                    <Name>bypass_audio</Name>
                                    <Bypass>true</Bypass>
                                </Audio>
                                <Video>
                                    <Name>bypass_video</Name>
                                    <Bypass>true</Bypass>
                                </Video>

                                <Video>
                                    <Name>video_720</Name>
                                    <Codec>h264</Codec>
                                    <Bitrate>2024000</Bitrate>
                                    <Framerate>30</Framerate>
                                    <Height>720</Height>
                                    <BypassIfMatch>
                                        <Height>lte</Height>
                                    </BypassIfMatch>
                                    <Preset>slower</Preset>
                                </Video>
                                <Video>
                                    <Name>video_576</Name>
                                    <Codec>h264</Codec>
                                    <Bitrate>1024000</Bitrate>
                                    <Framerate>30</Framerate>
                                    <Height>576</Height>
                                    <Preset>medium</Preset>
                                </Video>
                                <Video>
                                    <Name>video_160</Name>
                                    <Codec>h264</Codec>
                                    <Bitrate>524000</Bitrate>
                                    <Framerate>30</Framerate>
                                    <Height>160</Height>
                                    <Preset>faster</Preset>
                                    <ThreadCount>1</ThreadCount>
                                </Video>
                            </Encodes>
                        </OutputProfile>
                    </OutputProfiles>
                    <Providers>
                        <RTMP/>
                    </Providers>
                    <Publishers>
                        <AppWorkerCount>1</AppWorkerCount>
                        <StreamWorkerCount>8</StreamWorkerCount>
                        <LLHLS>
                            <OriginMode>true</OriginMode>

                            <ChunkDuration>0.5</ChunkDuration>
                            <PartHoldBack>1.5</PartHoldBack>
                            <SegmentDuration>6</SegmentDuration>
                            <SegmentCount>10</SegmentCount>
                            <CrossDomains>
                                <Url>*</Url>
                            </CrossDomains>
                        </LLHLS>
                    </Publishers>
                </Application>
            </Applications>
        </VirtualHost>
    </VirtualHosts>
</Server>

root@server:/# nvidia-smi
Thu Mar  9 10:53:22 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla M6            On   | 00000000:21:00.0 Off |                  Off |
| N/A   50C    P0    26W / 100W |    359MiB /  8192MiB |     17%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    124731      C   ...ngine/bin/OvenMediaEngine      356MiB |
+-----------------------------------------------------------------------------+
root@server:/# nvidia-smi dmon
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    26    50     -    14     2     6     9  2301   721
    0    26    50     -    14     2     5     9  2301   721
    0    26    50     -    15     2     6     9  2301   721
    0    27    50     -    15     2     6     9  2301   721

cpu : Model name: Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz

top :

root@server:/home/lary# top
top - 10:54:31 up 5 days, 18:49,  4 users,  load average: 0.94, 0.82, 0.74
Tasks: 364 total,   1 running, 363 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.0 us,  1.4 sy,  0.0 ni, 95.6 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  32102.8 total,  25971.4 free,   1342.2 used,   4789.2 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  30289.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                        
 124731 root      20   0   49.4g 543128 283716 S  70.8   1.7   1:31.31 OvenMediaEngine                                                                                                                                                
  80885 root      20   0 2944508  92296  53080 S  21.6   0.3   5:49.48 dockerd                                                                                                                                                        
 124709 root      20   0  720756   9992   7752 S  10.0   0.0   0:13.28 containerd-shim                                                                                                                                                
    470 root     -51   0       0      0      0 S   1.0   0.0   6:28.50 irq/135-nvidia                                                                                                                                                 
 124861 root      20   0   16904  10372   8524 S   0.7   0.0   0:00.02 sshd                                                                                                                                                           
    975 root      20   0   82960   4200   3672 S   0.3   0.0   2:21.38 irqbalance                                                                                                                                                     
 124860 root      20   0   10776   4108   3244 R   0.3   0.0   0:00.04 top                                                                                                                                                            
 124862 sshd      20   0   15424   5592   3976 S   0.3   0.0   0:00.01 sshd                                                                                                                                                           
      1 root      20   0  167700  13380   8460 S   0.0   0.0   0:50.09 systemd                                                                                                                                                        
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.30 kthreadd                                                                                                                                                       
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                                                                                                                                                         
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                                                                                                                                                     
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 slub_flushwq                                                                                                                                                   
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns                                                                                                                                                          
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri                                                                                                                                    
     11 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                                                                                                                                                   
     12 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_                                                                                                                                                
     13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace                                                                                                                                                
     14 root      20   0       0      0      0 S   0.0   0.0   0:02.28 ksoftirqd/0                                                                                                                                                    
     15 root      20   0       0      0      0 I   0.0   0.0   0:32.83 rcu_sched                                                                                                                                                      
     16 root      rt   0       0      0      0 S   0.0   0.0   0:01.90 migration/0                                                                                                                                                    
     17 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/0                                                                                                                                                  
     19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                                                                                                                                                        
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                                                                                                                                                        
     21 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/1                                                                                                                                                  
     22 root      rt   0       0      0      0 S   0.0   0.0   0:02.35 migration/1                                                                                                                                                    
     23 root      20   0       0      0      0 S   0.0   0.0   0:01.44 ksoftirqd/1                                                                                                                                                    
     25 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0H-kblockd                                                                                                                                           
     26 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2                                                                                                                                                        
     27 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/2                                                                                                                                                  
     28 root      rt   0       0      0      0 S   0.0   0.0   0:02.36 migration/2                                                                                                                                                    
     29 root      20   0       0      0      0 S   0.0   0.0   0:01.40 ksoftirqd/2

docker stats :

CONTAINER ID   NAME                CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O     PIDS
f337523550c3   optimistic_fermat   73.32%    316.1MiB / 31.35GiB   0.98%     92.8MB / 2.58MB   0B / 81.5MB   41

Keukhan · 2023-03-09T13:53:33Z

Keukhan
Mar 9, 2023
Maintainer Sponsor

@Pernifloss

Maybe, that GPU has a limit on the number of encoders that can be used simultaneously. If the GPU encoder allocation fails, it will automatically switch to a openh264(s/w) codec.

Please share the full log. Then I can analyze the exact cause.

Thanks.

0 replies

Pernifloss · 2023-03-09T14:07:15Z

Pernifloss
Mar 9, 2023
Author

Thanks for you answer @Keukhan
The stats i provided on my first post are for only 1 incoming stream.

Here are docker logs on start , and on stream start.

[2023-03-09 14:03:30.667] I [OvenMediaEngine:1] Config | config_manager.cpp:239  | Trying to set logfile in directory... (/var/log/ovenmediaengine)
[2023-03-09 14:03:30.668] I [OvenMediaEngine:1] Config | config_manager.cpp:261  | Trying to load configurations... (origin_conf/Server.xml)
[2023-03-09 14:03:30.708] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:23   | OvenMediaEngine v0.15.1 () is started on [f337523550c3] (Linux x86_64 - 5.15.0-67-generic, #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023)
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:27   | With modules:
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:28   |   FFmpeg 5.0.1
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:29   |     Configuration: --prefix=/opt/ovenmediaengine --extra-cflags='-I/opt/ovenmediaengine/include -g -I/usr/local/cuda/include' --extra-ldflags='-L/opt/ovenmediaengine/lib -L/usr/local/cuda/lib64 -Wl,-rpath,/opt/ovenmediaengine/lib' --extra-libs=-ldl --enable-gpl --enable-nonfree --enable-shared --disable-static --disable-debug --disable-doc --disable-programs --disable-avdevice --disable-dct --disable-dwt --disable-lsp --disable-lzo --disable-rdft --disable-faan --disable-pixelutils --enable-zlib --enable-libopus --enable-libvpx --enable-libfdk_aac --enable-libopenh264 --enable-openssl --disable-nvdec --disable-nvdec --disable-vaapi --disable-vdpau --disable-cuda-llvm --disable-cuvid --disable-ffnvcodec --enable-cuda-nvcc --enable-cuda-llvm --enable-libnpp --enable-nvenc --enable-nvdec --enable-ffnvcodec --enable-cuvid --disable-everything --disable-fast-unaligned --enable-hwaccel='cuda,cuvid' --enable-encoder='libvpx_vp8,libopus,libfdk_aac,libopenh264,mjpeg,png,h264_nvenc,hevc_nvenc' --enable-decoder='aac,aac_latm,aac_fixed,h264,hevc,opus,vp8,h264_nvdec,hevc_nvdec,h264_cuvid,hevc_cuvid' --enable-parser='aac,aac_latm,aac_fixed,h264,hevc,opus,vp8' --enable-network --enable-protocol=tcp --enable-protocol=udp --enable-protocol='rtp,file,rtmp,tls,rtmps' --enable-demuxer='rtsp,flv,live_flv,mp4' --enable-muxer='mp4,webm,mpegts,flv,mpjpeg' --enable-filter='asetnsamples,aresample,aformat,channelmap,channelsplit,scale,transpose,fps,settb,asettb,format,scale_cuda,hwdownload,hwupload,hwupload_cuda'
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:30   |     libavformat: 59.16.100
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:31   |     libavcodec: 59.18.100
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:32   |     libavutil: 57.17.100
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:33   |     libavfilter: 8.24.100
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:34   |     libswresample: 4.3.100
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:35   |     libswscale: 6.4.100
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:36   |   SRT: 1.4.4
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:37   |   SRTP: libsrtp2 2.4.2
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:38   |   OpenSSL: OpenSSL 3.0.7 1 Nov 2022
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:39   |     Configuration: compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:40   |   JsonCpp: 1.9.3
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | banner.cpp:41   |   jemalloc: 5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c
[2023-03-09 14:03:30.714] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:72   | Server ID : 7f75bbd2-c900-4c95-ba7f-e729bc5db2eb
[2023-03-09 14:03:31.172] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:81   | Resolved public IP address (89.39.149.193) from stun server (stun.ovenmediaengine.com:13478)
[2023-03-09 14:03:31.187] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:111  | Trying to create MediaRouter...
[2023-03-09 14:03:31.187] I [OvenMediaEngine:1] MediaRouter | mediarouter.cpp:40   | Mediarouter has been started.
[2023-03-09 14:03:31.187] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:114  | Trying to create WebRTC Publisher...
[2023-03-09 14:03:31.188] W [OvenMediaEngine:1] WebRTC Publisher | webrtc_publisher.cpp:105  | WebRTC Publisher is disabled by configuration
[2023-03-09 14:03:31.189] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:115  | Trying to create LLHLS Publisher...
[2023-03-09 14:03:31.190] I [OvenMediaEngine:1] Publisher | publisher.cpp:19   | LLHLS Publisher has been started.
[2023-03-09 14:03:31.191] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:116  | Trying to create HLS Publisher...
[2023-03-09 14:03:31.191] W [OvenMediaEngine:1] HLS | hls_publisher.cpp:38   | HLS Publisher is disabled by configuration
[2023-03-09 14:03:31.192] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:117  | Trying to create MPEG-DASH Publisher...
[2023-03-09 14:03:31.192] W [OvenMediaEngine:1] DASH | dash_publisher.cpp:36   | DASH Publisher is disabled by configuration
[2023-03-09 14:03:31.192] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:118  | Trying to create Low-Latency MPEG-DASH Publisher...
[2023-03-09 14:03:31.193] W [OvenMediaEngine:1] LLDASH | cmaf_publisher.cpp:39   | LLDASH Publisher is disabled by configuration
[2023-03-09 14:03:31.193] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:119  | Trying to create OVT Publisher...
[2023-03-09 14:03:31.194] W [OvenMediaEngine:1] OVT | ovt_publisher.cpp:39   | OVTPublisher is disabled by configuration
[2023-03-09 14:03:31.194] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:120  | Trying to create File Publisher...
[2023-03-09 14:03:31.195] I [OvenMediaEngine:1] Publisher | publisher.cpp:19   | FilePublisher has been started.
[2023-03-09 14:03:31.195] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:121  | Trying to create MpegtsPush Publisher...
[2023-03-09 14:03:31.196] I [OvenMediaEngine:1] Publisher | publisher.cpp:19   | MPEGTSPushPublisher has been started.
[2023-03-09 14:03:31.196] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:122  | Trying to create RtmpPush Publisher...
[2023-03-09 14:03:31.197] I [OvenMediaEngine:1] Publisher | publisher.cpp:19   | RTMPPushPublisher has been started.
[2023-03-09 14:03:31.198] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:123  | Trying to create Thumbnail Publisher...
[2023-03-09 14:03:31.199] W [OvenMediaEngine:1] Thumbnail | thumbnail_publisher.cpp:79   | ThumbnailPublisher is disabled by configuration
[2023-03-09 14:03:31.199] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:126  | Trying to create Transcoder...
[2023-03-09 14:03:31.199] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:129  | Trying to create WebRTC Provider...
[2023-03-09 14:03:31.200] W [OvenMediaEngine:1] WebRTC Provider | webrtc_provider.cpp:140  | WebRTCProvider is disabled by configuration
[2023-03-09 14:03:31.201] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:130  | Trying to create MPEG-TS Provider...
[2023-03-09 14:03:31.201] W [OvenMediaEngine:1] MpegtsProvider | mpegts_provider.cpp:147  | MPEGTSProvider is disabled by configuration
[2023-03-09 14:03:31.201] I [OvenMediaEngine:1] MpegtsProvider | mpegts_provider.cpp:39   | Terminated Mpegts Provider module.
[2023-03-09 14:03:31.201] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:131  | Trying to create SRT Provider...
[2023-03-09 14:03:31.202] W [OvenMediaEngine:1] SrtProvider | srt_provider.cpp:49   | SrtProvider is disabled by configuration
[2023-03-09 14:03:31.202] I [OvenMediaEngine:1] SrtProvider | srt_provider.cpp:39   | Terminated SRT Provider module.
[2023-03-09 14:03:31.202] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:132  | Trying to create RTMP Provider...
[2023-03-09 14:03:31.203] I [OvenMediaEngine:1] RTMPProvider | rtmp_provider.cpp:138  | RTMPProvider is listening on *:1935
[2023-03-09 14:03:31.203] I [OvenMediaEngine:1] Provider | provider.cpp:37   | RTMPProvider has been started.
[2023-03-09 14:03:31.204] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:133  | Trying to create OVT Provider...
[2023-03-09 14:03:31.204] I [OvenMediaEngine:1] Provider | provider.cpp:37   | OVTProvider has been started.
[2023-03-09 14:03:31.205] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:134  | Trying to create RTSPC Provider...
[2023-03-09 14:03:31.206] I [OvenMediaEngine:1] Provider | provider.cpp:37   | RTSPCProvider has been started.
[2023-03-09 14:03:31.206] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:135  | Trying to create File Provider...
[2023-03-09 14:03:31.207] I [OvenMediaEngine:1] Provider | provider.cpp:37   | FileProvider has been started.
[2023-03-09 14:03:31.207] I [OvenMediaEngine:1] OvenMediaEngine | main.cpp:142  | All modules are initialized successfully
[2023-03-09 14:03:31.208] I [OvenMediaEngine:1] Monitor | monitoring.cpp:72   | OvenMediaEngine(7f75bbd2-c900-4c95-ba7f-e729bc5db2eb) ServerMetric has been started for monitoring - 2023-03-09T14:03:31.207+00:00
[2023-03-09 14:03:31.209] I [OvenMediaEngine:1] Monitor | server_metrics.cpp:55   | Create HostMetrics(default/7f75bbd2-c900-4c95-ba7f-e729bc5db2eb/default) for monitoring
[2023-03-09 14:03:31.210] I [OvenMediaEngine:1] Orchestrator | orchestrator_internal.cpp:733  | Trying to create an application: [#default#webtv]
[2023-03-09 14:03:31.210] I [OvenMediaEngine:1] Monitor | host_metrics.cpp:57   | Create ApplicationMetrics(#default#webtv/7f75bbd2-c900-4c95-ba7f-e729bc5db2eb/default/#default#webtv) for monitoring
[2023-03-09 14:03:31.211] I [OvenMediaEngine:1] MediaRouter | mediarouter_application.cpp:50   | [#default#webtv(0)] Created Mediarouter application. worker(1)
[2023-03-09 14:03:31.211] I [OvenMediaEngine:1] MediaRouter | mediarouter.cpp:83   | Created Mediarouter. app(#default#webtv)
[2023-03-09 14:03:31.212] I [OvenMediaEngine:1] Publisher | application.cpp:165  | LLHLS Publisher Application has created [#default#webtv] application
[2023-03-09 14:03:31.212] W [OvenMediaEngine:1] Publisher | publisher.cpp:70   | File publisher is disabled in #default#webtv application, so it was not created
[2023-03-09 14:03:31.212] W [OvenMediaEngine:1] Publisher | publisher.cpp:70   | MPEGTSPush publisher is disabled in #default#webtv application, so it was not created
[2023-03-09 14:03:31.212] W [OvenMediaEngine:1] Publisher | publisher.cpp:70   | RTMPPush publisher is disabled in #default#webtv application, so it was not created
[2023-03-09 14:03:31.213] I [OvenMediaEngine:1] Transcoder | transcoder_application.cpp:31   | Created transcoder application. app(#default#webtv)
[2023-03-09 14:03:31.381] I [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:60   | Supported NVIDIA CUDA hardware accelerator. hw.pixfmt: 117, sw.pixfmt : 23
[2023-03-09 14:03:31.381] I [OvenMediaEngine:1] Transcoder | transcoder.cpp:86   | Transcoder has created [config][#default#webtv] application
[2023-03-09 14:03:31.381] I [OvenMediaEngine:1] Provider | application.cpp:46   | RTMPProvider has created [#default#webtv] application
[2023-03-09 14:03:31.381] W [OvenMediaEngine:1] Provider | provider.cpp:93   | OVT provider is disabled in #default#webtv application, so it was not created
[2023-03-09 14:03:31.381] W [OvenMediaEngine:1] Provider | provider.cpp:93   | RTSP Pull provider is disabled in #default#webtv application, so it was not created
[2023-03-09 14:03:39.629] I [SPRTMP-t1935:12] Provider | stream.cpp:49   | Unknown/(100) has been started stream
[2023-03-09 14:03:39.629] I [SPRTMP-t1935:12] RTMPProvider | rtmp_provider.cpp:189  | A RTMP client has connected from <ClientSocket: 0x7feba08fc610, #26, Connected, TCP, Nonblocking, 95.178.88.86:62445>
[2023-03-09 14:03:40.007] I [SPRTMP-t1935:12] AccessController | access_controller.cpp:183  | AdmissionWebhooks queried xxx whether client 95.178.88.86:62445 could access rtmp://89.39.149.193:1935/webtv/stream. (Result : Allow Elapsed : 145 ms)
[2023-03-09 14:03:40.202] I [SPRTMP-t1935:12] MediaRouter | mediarouter_application.cpp:245  | [#default#webtv/stream(100)] Trying to create a stream
[2023-03-09 14:03:40.202] I [SPRTMP-t1935:12] Monitor | application_metrics.cpp:57   | Create StreamMetrics(stream/7f75bbd2-c900-4c95-ba7f-e729bc5db2eb/default/#default#webtv/stream/i) for monitoring
[2023-03-09 14:03:40.202] I [SPRTMP-t1935:12] MediaRouter | mediarouter_application.cpp:343  | [#default#webtv/stream(100)] Stream has been created
[2023-03-09 14:03:40.203] I [SPRTMP-t1935:12] Transcoder | transcoder_stream.cpp:61   | [#default#webtv/stream(100)] Transcoder stream has been started
[2023-03-09 14:03:40.248] I [InboundWorker:13] MediaRouter | mediarouter_application.cpp:380  | [#default#webtv/stream(100)] Stream has been prepared 
[Stream Info]
id(100), msid(0), output(stream), SourceType(Rtmp), RepresentationType(Source), Created Time (Thu Mar  9 14:03:39 2023) UUID(7f75bbd2-c900-4c95-ba7f-e729bc5db2eb/default/#default#webtv/stream/i)

        Video Track #0: Public Name(Video_0) Variant Name(Video) Bitrate(4.00Mb) Codec(1,H264,Auto) BSF(H264_AVCC) Resolution(1920x1080) Framerate(30.00fps) KeyInterval(0) BFrames(0) timebase(1/1000)
        Audio Track #1: Public Name(Audio_1) Variant Name(Audio) Bitrate(160.00Kb) Codec(6,AAC,Auto) BSF(AAC_RAW) Samplerate(48.0K) Format(s16, 16) Channel(stereo, 2) timebase(1/1000)
        Data  Track #2: Public Name(Data_2) Variant Name(Data) Codec(0,Unknown,Auto) BSF(ID3v2) timebase(1/1000)
[2023-03-09 14:03:40.249] I [InboundWorker:13] Transcoder | transcoder_stream.cpp:306  | [#default#webtv/stream(100)] Output stream has been created. [#default#webtv/stream(3838898584)]
[2023-03-09 14:03:40.266] I [InboundWorker:13] MediaRouter | mediarouter_application.cpp:245  | [#default#webtv/stream(3838898584)] Trying to create a stream
[2023-03-09 14:03:40.267] I [InboundWorker:13] Monitor | application_metrics.cpp:57   | Create StreamMetrics(stream/7f75bbd2-c900-4c95-ba7f-e729bc5db2eb/default/#default#webtv/stream/o) for monitoring
[2023-03-09 14:03:40.267] I [InboundWorker:13] MediaRouter | mediarouter_application.cpp:343  | [#default#webtv/stream(3838898584)] Stream has been created
[2023-03-09 14:03:40.267] I [InboundWorker:13] Transcoder | transcoder_stream.cpp:106  | [#default#webtv/stream(100)] Transcoder stream has been prepared
[2023-03-09 14:03:40.333] I [Dech264NV:20] Transcoder | decoder_avc_nv.cpp:233  | [#default#webtv/stream(100)] input stream information: [video] h264 (High 4.0), cuda, 1920x1080 [SAR 1:1 DAR 16:9], 30.000 fps, 3906 kbps, timebase: 1/60, frame_size: 0
[2023-03-09 14:03:40.337] I [Dech264NV:20] Transcoder | transcoder_encoder.cpp:65   | [#2] hwardware acceleration of the encoder is enabled. The library to be used will be Auto
[2023-03-09 14:03:40.368] I [Dech264NV:20] Transcoder | transcoder_encoder.cpp:65   | [#1] hwardware acceleration of the encoder is enabled. The library to be used will be Auto
[2023-03-09 14:03:40.400] I [Dech264NV:20] Transcoder | transcoder_encoder.cpp:65   | [#3] hwardware acceleration of the encoder is enabled. The library to be used will be Auto
[2023-03-09 14:03:40.579] I [Dech264NV:20] Transcoder | filter_rescaler.cpp:166  | Rescaler is enabled for track #0 using parameters. input: video_size=1920x1080:pix_fmt=23:time_base=1/1000:pixel_aspect=1/1 / outputs: fps=fps=30.00:round=near,hwupload_cuda,scale_cuda=1024:576,hwdownload,settb=1/90000
[2023-03-09 14:03:40.737] I [Dech264NV:20] Transcoder | filter_rescaler.cpp:166  | Rescaler is enabled for track #0 using parameters. input: video_size=1920x1080:pix_fmt=23:time_base=1/1000:pixel_aspect=1/1 / outputs: fps=fps=30.00:round=near,hwupload_cuda,scale_cuda=1280:720,hwdownload,settb=1/90000
[2023-03-09 14:03:40.897] I [Dech264NV:20] Transcoder | filter_rescaler.cpp:166  | Rescaler is enabled for track #0 using parameters. input: video_size=1920x1080:pix_fmt=23:time_base=1/1000:pixel_aspect=1/1 / outputs: fps=fps=30.00:round=near,hwupload_cuda,scale_cuda=284:160,hwdownload,settb=1/90000
[2023-03-09 14:03:40.938] I [OutboundWorker:14] MediaRouter | mediarouter_application.cpp:380  | [#default#webtv/stream(3838898584)] Stream has been prepared 
[Stream Info]
id(3838898584), msid(0), output(stream), SourceType(Transcoder), RepresentationType(Source), Created Time (Thu Mar  9 14:03:40 2023) UUID(7f75bbd2-c900-4c95-ba7f-e729bc5db2eb/default/#default#webtv/stream/o)
        >> Origin Stream Info
        id(100), output(stream), SourceType(Rtmp), Created Time (Thu Mar  9 14:03:39 2023)

        Video Track #0: Public Name(Video_0) Variant Name(bypass_video) Bitrate(4.00Mb) Codec(1,H264,Passthrough) BSF(H264_AVCC) Resolution(1920x1080) Framerate(30.00fps) KeyInterval(0) BFrames(0) timebase(1/1000)
        Video Track #1: Public Name(Video_0) Variant Name(video_720) Bitrate(2.02Mb) Codec(1,H264,nvenc) BSF(H264_ANNEXB) Resolution(1280x720) Framerate(30.00fps) KeyInterval(0) BFrames(0) timebase(1/90000)
        Video Track #2: Public Name(Video_0) Variant Name(video_576) Bitrate(1.02Mb) Codec(1,H264,nvenc) BSF(H264_ANNEXB) Resolution(1024x576) Framerate(30.00fps) KeyInterval(0) BFrames(0) timebase(1/90000)
        Video Track #3: Public Name(Video_0) Variant Name(video_160) Bitrate(524.00Kb) Codec(1,H264,nvenc) BSF(H264_ANNEXB) Resolution(284x160) Framerate(30.00fps) KeyInterval(0) BFrames(0) timebase(1/90000)
        Audio Track #4: Public Name(Audio_1) Variant Name(bypass_audio) Bitrate(160.00Kb) Codec(6,AAC,Passthrough) BSF(AAC_RAW) Samplerate(48.0K) Format(s16, 16) Channel(stereo, 2) timebase(1/1000)
        Data  Track #5: Public Name(Data_2) Variant Name(Data) Codec(0,Unknown,Passthrough) BSF(ID3v2) timebase(1/1000)
[2023-03-09 14:03:40.940] I [OutboundWorker:14] LLHLS Publisher | llhls_stream.cpp:188  | LLHlsStream has been created : stream/3838898584
OriginMode(true) Chunk Duration(0.50) Segment Duration(6) Segment Count(10)
[2023-03-09 14:03:40.941] I [OutboundWorker:14] Publisher | stream.cpp:204  | LLHLS Publisher Application application has started [stream(3838898584)] stream (MSID : 0)

Thanks

0 replies

Keukhan · 2023-03-10T00:06:00Z

Keukhan
Mar 10, 2023
Maintainer Sponsor

@Pernifloss

Thank you for your quick reply.

Fortunately, looking at the logs, all decoder, rescaler and encoders are using the GPU. so, I need to analyze which section uses CPU resources in the Docker environment.
It takes time to analyze the cause. I'll put it in my work plan. wait for the good news.

Thanks

0 replies

Keukhan · 2023-03-10T01:38:22Z

Keukhan
Mar 10, 2023
Maintainer Sponsor

@Pernifloss

Could you share the CPU usage per thread of the OvenMediaEngine process?

You can check it with the command below
top -H -p [PID]

Thanks

0 replies

Pernifloss · 2023-03-10T07:16:18Z

Pernifloss
Mar 10, 2023
Author

@Keukhan

Here you go :

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
142390 root      20   0   49.7g 415264  85048 S  35.2   1.3   1:47.61 Dech264
142391 root      20   0   49.7g 415264  85048 S  14.3   1.3   0:45.71 Dech264
142392 root      20   0   49.7g 415264  85048 R  13.3   1.3   0:46.16 Dech264
142394 root      20   0   49.7g 415264  85048 S  13.3   1.3   0:45.74 Dech264
142393 root      20   0   49.7g 415264  85048 S  13.0   1.3   0:45.71 Dech264
142409 root      20   0   49.7g 415264  85048 S   7.6   1.3   0:17.74 Dech264
142411 root      20   0   49.7g 415264  85048 S   7.6   1.3   0:16.70 Dech264
142397 root      20   0   49.7g 415264  85048 S   6.0   1.3   0:18.45 Ench264
142412 root      20   0   49.7g 415264  85048 S   5.6   1.3   0:17.01 Rescaler
142419 root      20   0   49.7g 415264  85048 S   5.3   1.3   0:12.09 Rescaler
142396 root      20   0   49.7g 415264  85048 S   4.3   1.3   0:12.78 Ench264
142403 root      20   0   49.7g 415264  85048 S   4.3   1.3   0:14.27 Dech264
142404 root      20   0   49.7g 415264  85048 S   4.3   1.3   0:13.07 Dech264
142417 root      20   0   49.7g 415264  85048 S   4.3   1.3   0:09.40 Dech264
142410 root      20   0   49.7g 415264  85048 S   4.0   1.3   0:15.90 Dech264
142418 root      20   0   49.7g 415264  85048 S   4.0   1.3   0:10.64 Dech264
142405 root      20   0   49.7g 415264  85048 S   3.7   1.3   0:13.69 Rescaler
142402 root      20   0   49.7g 415264  85048 S   3.0   1.3   0:13.90 Dech264
142416 root      20   0   49.7g 415264  85048 S   2.7   1.3   0:10.69 Dech264
131754 root      20   0   49.7g 415264  85048 S   2.3   1.3   0:27.73 SPLLHLS-t3333
142398 root      20   0   49.7g 415264  85048 S   2.3   1.3   0:08.16 Ench264
131759 root      20   0   49.7g 415264  85048 S   2.0   1.3   0:29.47 AW-LLHLS0
131756 root      20   0   49.7g 415264  85048 S   1.3   1.3   0:38.82 SPRTMP-t1935
131758 root      20   0   49.7g 415264  85048 S   1.3   1.3   0:14.18 OutboundWorker
142395 root      20   0   49.7g 415264  85048 S   1.0   1.3   0:03.63 Dech264
131751 root      20   0   49.7g 415264  85048 S   0.3   1.3   0:19.77 SPDefUdp
131757 root      20   0   49.7g 415264  85048 S   0.3   1.3   0:06.14 InboundWorker
142423 root      20   0   49.7g 415264  85048 S   0.3   1.3   0:00.11 StreamWorker
142424 root      20   0   49.7g 415264  85048 S   0.3   1.3   0:00.81 StreamWorker
131694 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:01.93 OvenMediaEngine
131752 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 DQConnectionCB
131753 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:02.03 SRT:GC
131755 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.97 DQHTTPTimer
131760 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 cuda-EvtHandlr
131761 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:30.12 cuda-EvtHandlr
131762 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:19.73 SPAPISvr-t8081
131769 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:19.77 SPDefTcp
142399 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142400 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142401 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142406 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142407 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142408 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142413 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142414 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142415 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.00 Dech264
142420 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.10 StreamWorker
142421 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.11 StreamWorker
142422 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.10 StreamWorker
142425 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.44 StreamWorker
142426 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.19 StreamWorker
142427 root      20   0   49.7g 415264  85048 S   0.0   1.3   0:00.11 StreamWorker

Thanks again!

4 replies

zapf2000 Mar 13, 2023

Did you apply the nvidia patch? nvidia limits the maximum numbers of nvenc jobs on their budget cards. after applying the patch, it's unlimited.
https://github.com/keylase/nvidia-patch

Maypeur Mar 13, 2023

Dear zapf2000,

Tesla M6 are not subject to session limit :
https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
Tesla M6 Maxwell (2nd Gen) 5th Gen 1 2 2 Unrestricted
Even with only one stream transcoded, CPU is overused.
Without transcoding, CPU usage from OvenMediaEngine is below 3%.

zapf2000 Mar 14, 2023

Dear zapf2000,

Tesla M6 are not subject to session limit :
https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
Tesla M6 Maxwell (2nd Gen) 5th Gen 1 2 2 Unrestricted

Even with only one stream transcoded, CPU is overused.
Without transcoding, CPU usage from OvenMediaEngine is below 3%.

Yes, you are right. It's a professional card that has no restrictions.
However, it seems that there is no GPU support on your OME.
Normally the process should look like this:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
187867 root 20 0 9162880 961884 225372 S 2,7 2,0 0:05.71 Dech264NV
187877 root 20 0 9162880 961884 225372 S 2,7 2,0 0:01.72 Rescaler
3536 root 20 0 9162880 961884 225372 S 1,3 2,0 11:47.08 SPRTMP-T1935
187871 root 20 0 9162880 961884 225372 S 1,3 2,0 0:00.83 Ench264NV
3584 root 20 0 9162880 961884 225372 S 1,0 2,0 2:47.30 AW-LLHLS0
3535 root 20 0 9162880 961884 225372 S 0,7 2,0 34:12.04 SRT:RcvQ:w1
3583 root 20 0 9162880 961884 225372 S 0,7 2,0 1:28.55 AW-WebRTC0
187869 root 20 0 9162880 961884 225372 S 0,7 2,0 0:00.43 Encopus
3516 root 20 0 9162880 961884 225372 S 0,3 2,0 0:41.56 SPICE-U10002
3581 root 20 0 9162880 961884 225372 S 0,3 2,0 0:42.93 InboundWorker
3582 root 20 0 9162880 961884 225372 S 0,3 2,0 1:37.41 OutboundWorker
3585 root 20 0 9162880 961884 225372 S 0,3 2,0 1:13.14 AW-HLS0
3586 root 20 0 9162880 961884 225372 S 0,3 2,0 1:00.30 AW-DASH0
3587 root 20 0 9162880 961884 225372 S 0,3 2,0 1:21.78 AW-OVT0
187878 root 20 0 9162880 961884 225372 S 0,3 2,0 0:00.16 StreamWorker
187880 root 20 0 9162880 961884 225372 S 0,3 2,0 0:00.12 StreamWorker
3478 root 20 0 9162880 961884 225372 S 0,0 2,0 0:03.54 OvenMediaEngine

It's strange because you logfile sais, cuda is used.

Maypeur Mar 14, 2023

This is what i expected about CPU usage when using GPU Transcoding.
What is strange is that nvidia-smi dmon show enc and dec usage.
Without hardware encoding it is 0%.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ABR with GPU also use CPU #1125

{{title}}

Replies: 5 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

ABR with GPU also use CPU #1125

Pernifloss Mar 9, 2023

Replies: 5 comments · 4 replies

Keukhan Mar 9, 2023 Maintainer Sponsor

Pernifloss Mar 9, 2023 Author

Keukhan Mar 10, 2023 Maintainer Sponsor

Keukhan Mar 10, 2023 Maintainer Sponsor

Pernifloss Mar 10, 2023 Author

zapf2000 Mar 13, 2023

Maypeur Mar 13, 2023

zapf2000 Mar 14, 2023

Maypeur Mar 14, 2023

Pernifloss
Mar 9, 2023

Replies: 5 comments 4 replies

Keukhan
Mar 9, 2023
Maintainer Sponsor

Pernifloss
Mar 9, 2023
Author

Keukhan
Mar 10, 2023
Maintainer Sponsor

Keukhan
Mar 10, 2023
Maintainer Sponsor

Pernifloss
Mar 10, 2023
Author