Clicking "Watch in Browser - MP4" works w/ 8GB RAM but crashes calibre-web.service w/ 2GB RAM / memory (MP4 is 1.0GB; users see "502 Bad Gateway" due to OOM crash)

[holta]

1h41m video (https://youtu.be/vbXu7WaifAo) is downloadable to IIAB even on VM's with 1.5GB RAM or less, and thumbnail/cover/poster seems to appear properly every time. However:

• Clicking "Watch in Browser - MP4" (top-right of "Book Details" pop-up) consistently causes calibre-web.service to crash, on regular VM's with 2GB or less.
• Whereas VM's with a gigantic amounts of RAM (e.g. 8GB) do allow clicking "Watch in Browser - MP4" to play the video.

Service crashing needs to be solved not just mitigated.

Hiding the problem is not a healthy option in my opinion — e.g. at this time I'm personally not in favor of partial solutions like these 2 proposed workarounds:

• PR [Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback or download-to-client [SEE ALSO #37 & #53] #51
• PR Restrict NGINX (public?) access to "large" files only (e.g. .mp3, .mp4, .webm) in /library/calibre-web — for PR #51 OOM workaround #57

Basic diagnostics below, towards getting us on the road to a proper fix:

root@box:~# cat /etc/issue
Ubuntu Noble Numbat (development branch) \n \l

root@box:/usr/local/calibre-web-py3# git log --oneline | head
e83c81e5 Merge branch 'janeczku:master' into master
4c64662b README.md: "Internet-in-a-Box does not support Docker"
2f7413cb Merge pull request #75 from deldesir/deldesir-tornado
93791b3a Increase Buffersize to 10GB
25fb8d93 Merge remote-tracking branch 'nl/master'
b456263d Merge pull request #72 from deldesir/deldesir-mp4-cover
525cd2fe Support case insensitive ext
bc0ebdb7 Updated Dutch translation.
cdfd9b31 Merge pull request #71 from deldesir/deldesir-description
04306ba2 Pull description

root@box:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       372Mi       1.5Gi        21Mi       162Mi       1.5Gi
Swap:             0B          0B          0B

root@box:~# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       8.7G  5.8G  3.0G  67% /

root@box:~# cat /var/log/calibre-web.log
[2023-12-24 17:32:39,037]  INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2023-12-24 17:32:39,037]  INFO {cps:177} Starting Calibre Web...
[2023-12-24 17:32:39,044] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2023-12-24 17:32:39,045] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2023-12-24 17:32:39,079] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2023-12-24 17:32:39,096]  WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(

[2023-12-24 17:32:39,240] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2023-12-24 17:32:39,512]  INFO {apscheduler.scheduler:181} Scheduler started
[2023-12-24 17:32:39,512]  INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2023-12-24 17:32:39,642] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2023-12-24 17:32:39,643] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2023-12-24 17:32:39,643] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2023-12-24 17:32:39,721] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2023-12-24 17:32:39,726] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2023-12-24 17:32:39,768] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2023-12-24 17:32:39,799] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'html2text'
[2023-12-24 17:32:39,809] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2023-12-24 17:32:39,810] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2023-12-24 17:32:39,911]  INFO {cps.server:267} Starting Tornado server on :8083
[2023-12-24 17:34:58,223] DEBUG {cps.web:1410} You are now logged in as: 'Admin'
[2023-12-24 17:34:58,243] DEBUG {cps.ub:82} Login and store session : 8b1f3e026c99c26da2710a803e79fd8c3731814f5a5be1b048a9411edb469756fa361953799483455066116aa7c0ded030ea40dafe80a1633b5985cd2bc19327
[2023-12-24 17:34:58,244] DEBUG {cps.ub:84} Found stored session: 8b1f3e026c99c26da2710a803e79fd8c3731814f5a5be1b048a9411edb469756fa361953799483455066116aa7c0ded030ea40dafe80a1633b5985cd2bc19327
[2023-12-24 17:35:05,991] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download https://www.youtube.com/watch?v=vbXu7WaifAo
[2023-12-24 17:35:05,992]  INFO {cps.tasks.download:28} Starting download task for URL: https://www.youtube.com/watch?v=vbXu7WaifAo
[2023-12-24 17:35:05,992]  INFO {cps.tasks.download:37} Subprocess args: ['lb-wrapper', 'https://www.youtube.com/watch?v=vbXu7WaifAo']
[2023-12-24 17:35:05,996]  INFO {cps.tasks.download:50} /usr/local/bin/lb

[2023-12-24 17:36:12,661]  INFO {cps.editbooks:382} Requested files list: ['/library/downloads/calibre-web/Youtube/MMTV_NEWLOOK/URGENT_PANIQUE_CHEZ_FATSHI_COU.P_D_ETAT_EN_COURS_SOULEVEMENT_POPULAIRES_DANS_LA_DIASPORA_15.90k_[vbXu7WaifAo].mp4']
[2023-12-24 17:36:12,662]  INFO {cps.editbooks:388} Processing file: <_io.BufferedReader name='/library/downloads/calibre-web/Youtube/MMTV_NEWLOOK/URGENT_PANIQUE_CHEZ_FATSHI_COU.P_D_ETAT_EN_COURS_SOULEVEMENT_POPULAIRES_DANS_LA_DIASPORA_15.90k_[vbXu7WaifAo].mp4'>
[2023-12-24 17:36:12,663] DEBUG {cps.uploader:371} Temporary file: /tmp/calibre_web/a47764e9ecae51b20547d3f41ae2b1d0
[2023-12-24 17:36:15,488] DEBUG {cps.helper:543} Moving title: /tmp/calibre_web/a47764e9ecae51b20547d3f41ae2b1d0 to /library/calibre-web/MMTV NEWLOOK/URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT EN COURS, SOULÈVEMENT POPULAIRES DANS LA DIASPORA (1)/URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT EN COURS, SOULÈVEMENT POPULAIRES DANS LA DIASPORA - MMTV NEWLOOK
[2023-12-24 17:36:15,509] DEBUG {cps.services.worker:91} Add Task for user: Admin - Upload URGENT PANIQUE CHEZ FATSHI,COU.P D&#39;ÉTAT EN COURS, SOULÈVEMENT POPULAIRES DANS LA DIASPORA
[2023-12-24 17:36:15,517]  INFO {cps.tasks.download:79} Successfully sent the list of requested files to http://192.168.0.173/books/meta
[2023-12-24 17:36:24,632] DEBUG {cps.web:1577} Start video watching for 1
[2023-12-24 17:36:24,634]  INFO {cps.web:1214} Serving book: URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT E - MMTV NEWLOOK

root@box:~# systemctl status calibre-web.service
× calibre-web.service - Calibre-Web
     Loaded: loaded (/etc/systemd/system/calibre-web.service; enabled; preset: enabled)
     Active: failed (Result: oom-kill) since Sun 2023-12-24 17:36:33 EST; 25min ago
   Duration: 3min 54.843s
    Process: 3550 ExecStart=/usr/local/calibre-web-py3/bin/python3 /usr/local/calibre-web-py3/cps.py -p /library/calibre-web/config/app.db (code=killed, signal=KILL)
   Main PID: 3550 (code=killed, signal=KILL)
        CPU: 30.887s

Dec 24 17:32:38 box systemd[1]: Started calibre-web.service - Calibre-Web.
Dec 24 17:36:32 box systemd[1]: calibre-web.service: A process of this unit has been killed by the OOM killer.
Dec 24 17:36:33 box systemd[1]: calibre-web.service: Main process exited, code=killed, status=9/KILL
Dec 24 17:36:33 box systemd[1]: calibre-web.service: Failed with result 'oom-kill'.
Dec 24 17:36:33 box systemd[1]: calibre-web.service: Consumed 30.887s CPU time.

root@box:~# journalctl -u calibre-web
Dec 24 17:32:38 box systemd[1]: Started calibre-web.service - Calibre-Web.
Dec 24 17:36:32 box systemd[1]: calibre-web.service: A process of this unit has been killed by the OOM killer.
Dec 24 17:36:33 box systemd[1]: calibre-web.service: Main process exited, code=killed, status=9/KILL
Dec 24 17:36:33 box systemd[1]: calibre-web.service: Failed with result 'oom-kill'.
Dec 24 17:36:33 box systemd[1]: calibre-web.service: Consumed 30.887s CPU time.

root@box:~# hostname -I
10.181.233.208 192.168.0.173 10.8.0.46 fd42:fb6b:9c93:d2f1:5054:ff:fefe:fe43

root@box:~# hostname -A
t6.lxd box box t6.lxd

root@box:/library/downloads/calibre-web# tree
.
├── Youtube
│   └── MMTV_NEWLOOK
│       └── URGENT_PANIQUE_CHEZ_FATSHI_COU.P_D_ETAT_EN_COURS_SOULEVEMENT_POPULAIRES_DANS_LA_DIASPORA_15.90k_[vbXu7WaifAo].mp4
└── survey.db

3 directories, 2 files

root@box:/library/calibre-web# tree
.
├── MMTV NEWLOOK
│   └── URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT EN COURS, SOULÈVEMENT POPULAIRES DANS LA DIASPORA (1)
│       ├── URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT E - MMTV NEWLOOK.mp4
│       └── cover.jpg
├── config
│   └── app.db
├── metadata.db
└── metadata_db_prefs_backup.json

4 directories, 5 files

root@box:~# cat /var/log/xklb.log
2023-12-24 17:35:06 - Running command: lb tubeadd /library/downloads/calibre-web/survey.db https://www.youtube.com/watch?v=vbXu7WaifAo --verbose && lb dl /library/downloads/calibre-web/survey.db --prefix /library/downloads/calibre-web --write-thumbnail --format best --format-sort 'tbr~1000' --video https://www.youtube.com/watch?v=vbXu7WaifAo --verbose
2023-12-24 17:36:12 - Download completed successfully.

root@box:~# lb --version
2.2.192

root@box:~# yt-dlp --version
2023.11.16

Related:

• Some videos download as overweight (giant) files that (1) are not appropriate for schools, and (2) crash Calibre-Web with "502 Bad Gateway" #37
• Serve videos [using Video.js ?] #46
  • PR Add Video.js #49
• Give everyone access to all books, intentionally ignoring Calibre-Web ACL accounts/authorization? #53
  • PR [Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback or download-to-client [SEE ALSO #37 & #53] #51
  • PR Restrict NGINX (public?) access to "large" files only (e.g. .mp3, .mp4, .webm) in /library/calibre-web — for PR #51 OOM workaround #57
• MP4 files failed to upload [WebM too, for large video files?] #61
• PR scripts/lb-wrapper: Try yt-dlp with "--format best --format-sort 'tbr~1000'" [testing shows need for URL validation, clearly communicated, for users who accidentally request the impossible after clicking "Download to IIAB" > "Start"] #63
  • PR Create scripts/lb-wrapper.greedy à la 'yt-dlp dl --format-sort size' [and refactor /usr/local/bin/lb-wrapper] #64
• Thumbnail of ["Downloaded to IIAB"] video is cut off & many files fail to be copied to /library/calibre-web [intermittent failures?] #77
  • "Download to IIAB" of Vimeo video ("Staff Pick" URL or regular URL) does not download [and 1 YouTube video too, then 2 YouTube videos worked] #78
  • "Download to IIAB" worked 5 out of 15 tries (trying to d/l individual YouTube videos) — "Tasks" view screenshots show the 10 failures [e.g. formerly live videos] #84

[holta]

Thanks everyone for the hard work (@deldesir @nzola @EMG70 and others!) as we work towards a legit, practical and complete solution here:

Hopefully very early in 2024 at the latest — putting this grassroots product on a rock-solid foundation for truly everyone 🏗️

[holta]

Tangentially-related diagnostics — to inform very purposefully tight codec/compression choices (PR #63 etc) alongside:

root@box:/library/calibre-web/MMTV NEWLOOK/URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT EN COURS, SOULÈVEMENT POPULAIRES DANS LA DIASPORA (1)# ls -l
total 1009544
-rw-r--r-- 1 root root 1033709017 Dec 24 17:36 "URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT E - MMTV NEWLOOK.mp4"
-rw-r--r-- 1 root root      56616 Dec 24 17:36  cover.jpg

root@box:~# mediainfo /library/calibre-web/MMTV\ NEWLOOK/URGENT\ PANIQUE\ CHEZ\ FATSHI\,COU.P\ D\'ÉTAT\ EN\ COURS\,\ SOULÈVEMENT\ POPULAIRES\ DANS\ LA\ DIASPORA\ \(1\)/URGENT\ PANIQUE\ CHEZ\ FATSHI\,COU.P\ D\'ÉTAT\ E\ -\ MMTV\ NEWLOOK.mp4
General
Complete name                            : URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT E - MMTV NEWLOOK.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 986 MiB
Duration                                 : 1 h 40 min
Overall bit rate                         : 1 365 kb/s
Frame rate                               : 25.000 FPS
Movie name                               : URGENT PANIQUE CHEZ FATSHI,COU.P D'ÉTAT EN COURS, SOULÈVEMENT POPULAIRES DANS LA DIASPORA
Performer                                : MMTV NEWLOOK
Recorded date                            : 20231224
Writing application                      : Lavf60.16.100
Comment                                  : https://www.youtube.com/watch?v=vbXu7WaifAo

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : [email protected]
Format settings                          : CABAC / 3 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 3 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 1 h 40 min
Bit rate                                 : 1 231 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.053
Stream size                              : 889 MiB (90%)
Title                                    : ISO Media file produced by Google Inc. Created on: 12/24/2023.
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : avcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 1 h 40 min
Bit rate mode                            : Constant
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 92.4 MiB (9%)
Title                                    : ISO Media file produced by Google Inc. Created on: 12/24/2023.
Language                                 : English
Default                                  : Yes
Alternate group                          : 1

[holta]

RECAP:

1. IIAB's version of Calibre-Web can be installed within 10 minutes (or less!) using the instructions at: https://github.com/iiab/calibre-web/wiki#wrench-installation
2. /usr/local/bin/lb-wrapper Line 12 should be UNcommented during testing/debugging (to force the download of overweight codecs from YouTube!)
3. /usr/local/bin/lb-wrapper Line 14 should be commented out during testing/debugging (to remove lightweight codecs!)

calibre-web/scripts/lb-wrapper

Lines 10 to 14 in 5758715

	# Or download largest possible HD-style / UltraHD videos, to try to force
	# out-of-memory "502 Bad Gateway" for testing of issues like #37 and #79
	# FORMAT_OPTIONS="--format-sort size"
	FORMAT_OPTIONS="--format best --format-sort 'tbr~1000'"

[holta]

@deldesir

Does the 9-hour video (8:54:03) below (https://youtu.be/9M4XKi25I2M) fail in the same way?

• Remove residual fragments from failed video #157 (comment)

[holta]

@deldesir is the failure to download @nzola's 2h video https://youtu.be/Zm-ROp6EMZo ("PARLONS EN LES SONDAGES EN RDC") related?

• Formerly live videos fail to download: "failed to download [download] ... does not pass filter (live_status=?not_live); skipping" #188 (comment)

[holta]

as we work towards a legit, practical and complete solution here

@mabuelhagag will take a look at this issue over coming days/weeks — ideally towards streaming 32 videos[*] simultaneously to 32 student devices, using IIAB on a 1GB RAM Raspberry Pi — as this used to work well in 2016 (FYI our testing 8+ years ago used KA Lite / Khan Academy Lite).

Thank you Mohamad!

[*] No matter the duration of each video, e.g. 9-minute videos or 9-hour videos. 🙏

NOTE: Some of those tests 8+ years ago weren't perfect, as we initially tested streaming of the very same video to all 32 (WiFi-connected) student devices. But more complete testing later confirmed that we could indeed stream DIFFERENT videos streamed (simultaneously) to the 32 student devices. ✅

[holta]

FYI these RAM / memory issues are more serious than ever, with LRN2.ORG's Calibre-Web crashing very regularly (daily or so, what is causing this?) Perhaps due to this very issue or very similar — with systemctl status calibre-web showing:

• oom-kill
• (code=killed, signal=KILL)
• calibre-web.service: Main process exited, code=killed, status=9/KILL

[holta]

Observation from @deldesir:

xklb offers playback capability using mpv. Unfortunately, there is no wrapper to embed mpv in a web player to leverage it. Finding a way to reuse playback code would [be] a step forward to fix the OOM issue

• Initial release: August 7, 2013
• https://github.com/mpv-player/mpv
• https://en.wikipedia.org/wiki/Mpv_(media_player)