Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggested improvements for scripts/lb-wrapper [refactor logging & subprocess substitution, etc] #87

Merged
merged 4 commits into from
Jan 10, 2024

Conversation

holta
Copy link
Member

@holta holta commented Jan 10, 2024

An executive decision was made to replace PR #86 with smaller PR's. Hopefully this (or similar) is one of those PR's.

This PR was tested on a Ubuntu 24.04 pre-release VM. And is already getting too large.

Thanks @deldesir and @EMG70 for reviewing. I think it's good enough until @deldesir and others improve on it later. But LMK your thoughts, with any questions/clarifications that are needed.

Example output
root@box:~# lb-wrapper https://youtu.be/2oMXIbOiASI
/usr/local/bin/lb
2024-01-10 01:12:03 - [Info] xklb version: 2.3.011
2024-01-10 01:12:03 - [Info] Old /library/downloads/calibre-web/survey.db moved aside.
2024-01-10 01:12:03 - [Info] Running xklb commands: lb tubeadd /library/downloads/calibre-web/survey.db https://youtu.be/2oMXIbOiASI -vv && lb dl /library/downloads/calibre-web/survey.db --prefix /library/downloads/calibre-web --video https://youtu.be/2oMXIbOiASI --format best --format-sort 'tbr~1000' --write-thumbnail -vv
2024-01-10 01:12:03 - [Error] library v2.3.011
2024-01-10 01:12:03 - [Error] ['/usr/local/bin/lb', 'tubeadd', '/library/downloads/calibre-web/survey.db', 'https://youtu.be/2oMXIbOiASI', '-vv']
2024-01-10 01:12:03 - [Error] {'extractor_config': {}, 'download_archive': '~/.local/share/yt_archive.txt', 'extra_media_data': {}, 'extra_playlist_data': {}, 'verbose': 2, 'db': <Database <sqlite3.Connection object at 0x7e99597d4400>>, 'database': '/library/downloads/calibre-web/survey.db', 'playlists': ['https://youtu.be/2oMXIbOiASI'], 'action': 'tubeadd'}
2024-01-10 01:12:03 - [Error] {'noprogress': True, 'skip_download': True, 'lazy_playlist': True, 'extract_flat': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000}
2024-01-10 01:12:04 - [Error] yt-dlp initialized 0.0635
2024-01-10 01:12:04 - [Info] [youtube] Extracting URL: https://youtu.be/2oMXIbOiASI
2024-01-10 01:12:04 - [Info] [youtube] 2oMXIbOiASI: Downloading webpage
2024-01-10 01:12:04 - [Info] [youtube] 2oMXIbOiASI: Downloading ios player API JSON
2024-01-10 01:12:04 - [Info] [youtube] 2oMXIbOiASI: Downloading android player API JSON
2024-01-10 01:12:04 - [Error] no such table: media
2024-01-10 01:12:04 - [Error] Importing playlist-less media https://youtu.be/2oMXIbOiASI
2024-01-10 01:12:05 - [Error] media.playlist_media_add 0.8982
2024-01-10 01:12:05 - [Error] ydl.extract_info done 0.0412
2024-01-10 01:12:05 - [Error]
2024-01-10 01:12:05 - [Error] Optimizing database
2024-01-10 01:12:05 - [Error] Processing table: media
2024-01-10 01:12:05 - [Error] Transforming column order: ['id', 'size', 'duration', 'time_uploaded', 'time_created', 'time_modified', 'time_deleted', 'time_downloaded', 'fps', 'view_count', 'path', 'webpath', 'extractor_id']
2024-01-10 01:12:05 - [Error] Creating index: id
2024-01-10 01:12:05 - [Error] Creating index: size
2024-01-10 01:12:05 - [Error] Creating index: duration
2024-01-10 01:12:05 - [Error] Creating index: time_uploaded
2024-01-10 01:12:05 - [Error] Creating index: time_created
2024-01-10 01:12:05 - [Error] Creating index: time_modified
2024-01-10 01:12:05 - [Error] Creating index: time_deleted
2024-01-10 01:12:05 - [Error] Creating index: time_downloaded
2024-01-10 01:12:05 - [Error] Creating index: fps
2024-01-10 01:12:05 - [Error] Creating index: view_count
2024-01-10 01:12:05 - [Error] Creating index: live_status
2024-01-10 01:12:05 - [Error] Creating index: path
2024-01-10 01:12:05 - [Error] Creating index: webpath
2024-01-10 01:12:05 - [Error] Creating index: uploader
2024-01-10 01:12:05 - [Error] Creating fts index: ['path', 'title']
2024-01-10 01:12:05 - [Error] Processing table: captions
2024-01-10 01:12:05 - [Error] Creating index: media_id
2024-01-10 01:12:05 - [Error] Creating index: time
2024-01-10 01:12:05 - [Error] Creating fts index: ['text']
2024-01-10 01:12:05 - [Error] Running VACUUM
2024-01-10 01:12:05 - [Error] Running ANALYZE
2024-01-10 01:12:05 - [Info]
2024-01-10 01:12:06 - [Error] library v2.3.011
2024-01-10 01:12:06 - [Error] ['/usr/local/bin/lb', 'dl', '/library/downloads/calibre-web/survey.db', '--prefix', '/library/downloads/calibre-web', '--video', 'https://youtu.be/2oMXIbOiASI', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '-vv']
2024-01-10 01:12:06 - [Error] {'profile': 'video', 'extractor_config': {}, 'extra_media_data': {}, 'extra_playlist_data': {}, 'prefix': '/library/downloads/calibre-web', 'sort': 'm.path like "http%"\n        , m.title IS NOT NULL desc\n        , m.path\n        , random()', 'where': [], 'include': [], 'exclude': [], 'limit': 7200, 'retry_delay': '14 days', 'db': <Database <sqlite3.Connection object at 0x7efc3e5fee30>>, 'verbose': 2, 'database': '/library/downloads/calibre-web/survey.db', 'playlists': ['https://youtu.be/2oMXIbOiASI'], 'defaults': ['sort', 'limit'], 'unk': ['--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail'], 'action': 'download'}
2024-01-10 01:12:06 - [Error] None
2024-01-10 01:12:06 - [Error] {'path': 'https://www.youtube.com/watch?v=2oMXIbOiASI', 'title': "Ice Cold Killers (Full Episode) | World's Deadliest", 'duration': 2664, 'time_created': 1704867124, 'size': 0, 'time_modified': 0, 'time_downloaded': 0, 'time_deleted': 0, 'extractor_key': 'Playlist-less media'}
2024-01-10 01:12:06 - [Error] {'noprogress': True, 'lazy_playlist': True, 'noplaylist': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'extractor_args': {'youtube': {'skip': ['authcheck']}}, 'logger': <xklb.tube_backend.download.<locals>.DictLogger object at 0x7efc3e4b6f90>, 'postprocessors': [{'key': 'FFmpegMetadata'}, {'key': 'FFmpegEmbedSubtitle'}], 'restrictfilenames': True, 'playlist_items': '1', 'extractor_retries': 3, 'retries': 12, 'retry_sleep_functions': {'extractor': <function download.<locals>.<lambda> at 0x7efc3d8e7d80>, 'http': <function download.<locals>.<lambda> at 0x7efc3d8e7e20>, 'fragment': <function download.<locals>.<lambda> at 0x7efc3d8e7ec0>}, 'outtmpl': {'default': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).200B_%(view_count)3.2D_[%(id).60B].%(ext)s', 'chapter': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).200B_%(section_number)03d_%(section_title)s_%(view_count)3.2D_[%(id).60B].%(ext)s'}, 'progress_hooks': [<function download.<locals>.<lambda> at 0x7efc3d8e7f60>], 'subtitlesformat': 'srt/best', 'format': 'best', 'format_sort': ['tbr~1000'], 'writethumbnail': True, '_warnings': ['"-f best" selects the best pre-merged format which is often not the best option.\n         To let yt-dlp download and merge the best available formats, simply do not pass any format selection.\n         If you know what you are doing and want only the best pre-merged format, use "-f b" instead to suppress this warning']}
downloading   0.0%  838.86KiB/s
downloading   0.0%  997.14KiB/s
downloading   0.0%    1.29MiB/s
downloading   0.0%    1.95MiB/s
downloading   0.0%    3.24MiB/s
downloading   0.0%    5.30MiB/s
downloading   0.0%    7.82MiB/s
downloading   0.1%    8.81MiB/s
downloading   0.2%   10.02MiB/s
downloading   0.4%   11.27MiB/s
downloading   0.7%   12.33MiB/s
downloading   1.5%   15.74MiB/s
downloading   2.9%   23.04MiB/s
downloading   3.6%   25.28MiB/s
downloading   3.6%  Unknown B/s
downloading   3.6%    1.70MiB/s
downloading   3.6%    2.17MiB/s
downloading   3.6%    3.11MiB/s
downloading   3.6%    5.06MiB/s
downloading   3.6%    8.40MiB/s
downloading   3.6%   11.45MiB/s
downloading   3.7%   13.29MiB/s
downloading   3.7%   17.95MiB/s
downloading   3.9%   22.60MiB/s
downloading   4.3%   30.11MiB/s
downloading   5.0%   39.73MiB/s
downloading   6.5%   45.44MiB/s
downloading   7.1%   50.00MiB/s
downloading   7.1%  Unknown B/s
downloading   7.1%  Unknown B/s
downloading   7.1%    4.12MiB/s
downloading   7.1%    5.33MiB/s
downloading   7.1%    7.92MiB/s
downloading   7.1%    7.22MiB/s
downloading   7.1%   10.58MiB/s
downloading   7.2%   16.52MiB/s
downloading   7.3%   21.07MiB/s
downloading   7.4%   32.76MiB/s
downloading   7.8%   44.49MiB/s
downloading   8.5%   30.78MiB/s
downloading  10.0%   51.58MiB/s
downloading  10.6%   53.64MiB/s
downloading  10.6%  Unknown B/s
downloading  10.6%  Unknown B/s
downloading  10.6%    4.31MiB/s
downloading  10.6%    5.77MiB/s
downloading  10.6%    8.26MiB/s
downloading  10.6%    7.83MiB/s
downloading  10.6%   12.31MiB/s
downloading  10.7%   17.04MiB/s
downloading  10.8%   26.55MiB/s
downloading  11.0%   41.19MiB/s
downloading  11.3%   43.31MiB/s
downloading  12.1%   32.20MiB/s
downloading  13.5%   42.62MiB/s
downloading  14.2%   48.76MiB/s
downloading  14.2%  Unknown B/s
downloading  14.2%    1.34MiB/s
downloading  14.2%    1.40MiB/s
downloading  14.2%    2.13MiB/s
downloading  14.2%    2.95MiB/s
downloading  14.2%    2.55MiB/s
downloading  14.3%    4.22MiB/s
downloading  14.3%    4.90MiB/s
downloading  14.4%    7.72MiB/s
downloading  14.6%   10.81MiB/s
downloading  14.9%   12.68MiB/s
downloading  15.7%   15.74MiB/s
downloading  17.1%   18.65MiB/s
downloading  17.8%   21.77MiB/s
downloading  17.8%  Unknown B/s
downloading  17.8%  Unknown B/s
downloading  17.8%  Unknown B/s
downloading  17.8%    9.74MiB/s
downloading  17.8%   13.17MiB/s
downloading  17.9%    9.27MiB/s
downloading  17.9%   13.15MiB/s
downloading  17.9%   18.85MiB/s
downloading  18.0%   28.09MiB/s
downloading  18.2%   22.22MiB/s
downloading  18.6%   36.47MiB/s
downloading  19.3%   38.90MiB/s
downloading  20.8%   46.70MiB/s
downloading  21.4%   52.24MiB/s
downloading  21.4%  Unknown B/s
downloading  21.4%  Unknown B/s
downloading  21.4%    6.45MiB/s
downloading  21.4%    9.87MiB/s
downloading  21.4%   13.43MiB/s
downloading  21.4%   11.30MiB/s
downloading  21.4%   15.09MiB/s
downloading  21.4%   21.79MiB/s
downloading  21.5%   17.49MiB/s
downloading  21.7%   28.28MiB/s
downloading  22.1%   34.32MiB/s
downloading  22.8%   39.22MiB/s
downloading  24.3%   40.57MiB/s
downloading  25.0%   48.63MiB/s
downloading  25.0%  Unknown B/s
downloading  25.0%    1.19MiB/s
downloading  25.0%    1.51MiB/s
downloading  25.0%    1.94MiB/s
downloading  25.0%    3.14MiB/s
downloading  25.0%    4.93MiB/s
downloading  25.0%    7.41MiB/s
downloading  25.1%   11.22MiB/s
downloading  25.2%   18.11MiB/s
downloading  25.3%   25.82MiB/s
downloading  25.7%   40.25MiB/s
downloading  26.4%   48.82MiB/s
downloading  27.9%   48.61MiB/s
downloading  28.5%   54.33MiB/s
downloading  28.5%  Unknown B/s
downloading  28.5%  Unknown B/s
downloading  28.5%    4.39MiB/s
downloading  28.5%    6.11MiB/s
downloading  28.5%    6.80MiB/s
downloading  28.5%    6.05MiB/s
downloading  28.5%    4.69MiB/s
downloading  28.6%    6.71MiB/s
downloading  28.7%    9.27MiB/s
downloading  28.9%   13.26MiB/s
downloading  29.2%   16.74MiB/s
downloading  30.0%   18.75MiB/s
downloading  31.4%   20.18MiB/s
downloading  32.1%   20.77MiB/s
downloading  32.1%  Unknown B/s
downloading  32.1%    2.61MiB/s
downloading  32.1%    3.33MiB/s
downloading  32.1%    4.23MiB/s
downloading  32.1%    6.69MiB/s
downloading  32.1%    9.59MiB/s
downloading  32.1%   12.09MiB/s
downloading  32.2%   16.08MiB/s
downloading  32.3%   24.16MiB/s
downloading  32.5%   15.46MiB/s
downloading  32.8%   25.45MiB/s
downloading  33.6%   25.12MiB/s
downloading  35.0%   39.88MiB/s
downloading  35.6%   45.70MiB/s
downloading  35.6%  Unknown B/s
downloading  35.6%    2.74MiB/s
downloading  35.6%    2.80MiB/s
downloading  35.6%    4.24MiB/s
downloading  35.6%    7.04MiB/s
downloading  35.7%    7.10MiB/s
downloading  35.7%    9.80MiB/s
downloading  35.7%   15.07MiB/s
downloading  35.8%    8.07MiB/s
downloading  36.0%   13.79MiB/s
downloading  36.4%   22.31MiB/s
downloading  37.1%   35.39MiB/s
downloading  38.6%   48.73MiB/s
downloading  39.2%   54.29MiB/s
downloading  39.2%  Unknown B/s
downloading  39.2%  Unknown B/s
downloading  39.2%    2.09MiB/s
downloading  39.2%    3.88MiB/s
downloading  39.2%    7.12MiB/s
downloading  39.3%    9.73MiB/s
downloading  39.3%   13.96MiB/s
downloading  39.3%   20.51MiB/s
downloading  39.4%   32.30MiB/s
downloading  39.6%   47.76MiB/s
downloading  40.0%   42.62MiB/s
downloading  40.7%   36.43MiB/s
downloading  42.2%   49.77MiB/s
downloading  42.8%   52.27MiB/s
downloading  42.8%  Unknown B/s
downloading  42.8%  Unknown B/s
downloading  42.8%  Unknown B/s
downloading  42.8%   14.57MiB/s
downloading  42.8%    6.77MiB/s
downloading  42.8%    8.18MiB/s
downloading  42.8%   12.18MiB/s
downloading  42.9%   18.34MiB/s
downloading  43.0%   27.65MiB/s
downloading  43.1%   39.60MiB/s
downloading  43.5%   47.67MiB/s
downloading  44.2%   47.25MiB/s
downloading  45.7%   47.14MiB/s
downloading  46.3%   53.20MiB/s
downloading  46.3%  Unknown B/s
downloading  46.3%    1.50MiB/s
downloading  46.3%    1.91MiB/s
downloading  46.3%    2.96MiB/s
downloading  46.3%    4.88MiB/s
downloading  46.3%    8.17MiB/s
downloading  46.4%   10.27MiB/s
downloading  46.4%   12.60MiB/s
downloading  46.5%   16.33MiB/s
downloading  46.7%   19.34MiB/s
downloading  47.0%   23.65MiB/s
downloading  47.8%   29.00MiB/s
downloading  49.2%   36.60MiB/s
downloading  49.9%   36.80MiB/s
downloading  49.9%  Unknown B/s
downloading  49.9%    1.64MiB/s
downloading  49.9%    2.00MiB/s
downloading  50.0%    2.88MiB/s
downloading  50.0%    4.43MiB/s
downloading  50.0%    5.76MiB/s
downloading  50.0%    8.00MiB/s
downloading  50.0%    9.09MiB/s
downloading  50.1%   11.03MiB/s
downloading  50.3%   12.99MiB/s
downloading  50.7%   15.05MiB/s
downloading  51.4%   18.86MiB/s
downloading  52.9%   25.14MiB/s
downloading  53.5%   27.75MiB/s
downloading  53.5%  Unknown B/s
downloading  53.5%  Unknown B/s
downloading  53.5%    6.66MiB/s
downloading  53.5%    9.01MiB/s
downloading  53.5%    7.71MiB/s
downloading  53.5%    9.31MiB/s
downloading  53.6%   13.28MiB/s
downloading  53.6%   20.44MiB/s
downloading  53.7%   31.28MiB/s
downloading  53.9%   42.67MiB/s
downloading  54.3%   40.62MiB/s
downloading  55.0%   58.02MiB/s
downloading  56.5%   61.87MiB/s
downloading  57.2%   59.94MiB/s
downloading  57.2%  Unknown B/s
downloading  57.2%    1.71MiB/s
downloading  57.2%    2.02MiB/s
downloading  57.2%    2.91MiB/s
downloading  57.2%    4.53MiB/s
downloading  57.2%    6.08MiB/s
downloading  57.2%    8.67MiB/s
downloading  57.3%   10.71MiB/s
downloading  57.4%   14.84MiB/s
downloading  57.6%   19.00MiB/s
downloading  57.9%   23.27MiB/s
downloading  58.7%   26.53MiB/s
downloading  60.1%   28.57MiB/s
downloading  60.7%   30.84MiB/s
downloading  60.7%  Unknown B/s
downloading  60.7%    2.52MiB/s
downloading  60.7%    3.14MiB/s
downloading  60.7%    4.53MiB/s
downloading  60.7%    6.78MiB/s
downloading  60.7%    8.93MiB/s
downloading  60.8%   11.52MiB/s
downloading  60.8%   17.23MiB/s
downloading  60.9%   26.85MiB/s
downloading  61.1%   27.32MiB/s
downloading  61.5%   34.23MiB/s
downloading  62.2%   41.89MiB/s
downloading  63.7%   43.90MiB/s
downloading  64.3%   49.17MiB/s
downloading  64.3%  Unknown B/s
downloading  64.3%  Unknown B/s
downloading  64.3%    4.62MiB/s
downloading  64.3%    5.30MiB/s
downloading  64.3%    7.86MiB/s
downloading  64.3%    7.39MiB/s
downloading  64.3%    4.94MiB/s
downloading  64.4%    8.45MiB/s
downloading  64.5%   12.28MiB/s
downloading  64.6%   19.36MiB/s
downloading  65.0%   34.49MiB/s
downloading  65.7%   50.86MiB/s
downloading  67.2%   46.59MiB/s
downloading  67.9%   55.02MiB/s
downloading  67.9%  Unknown B/s
downloading  67.9%    1.67MiB/s
downloading  67.9%    2.10MiB/s
downloading  67.9%    2.81MiB/s
downloading  67.9%    4.39MiB/s
downloading  67.9%    5.98MiB/s
downloading  67.9%    9.37MiB/s
downloading  68.0%    7.01MiB/s
downloading  68.0%    9.38MiB/s
downloading  68.2%   11.94MiB/s
downloading  68.6%   15.28MiB/s
downloading  69.3%   18.46MiB/s
downloading  70.8%   22.81MiB/s
downloading  71.5%   25.47MiB/s
downloading  71.5%  Unknown B/s
downloading  71.5%  Unknown B/s
downloading  71.5%  Unknown B/s
downloading  71.5%    4.43MiB/s
downloading  71.5%    7.37MiB/s
downloading  71.5%    4.16MiB/s
downloading  71.5%    8.06MiB/s
downloading  71.6%   15.73MiB/s
downloading  71.7%   23.69MiB/s
downloading  71.9%   36.14MiB/s
downloading  72.2%   53.46MiB/s
downloading  73.0%   62.49MiB/s
downloading  74.4%   66.63MiB/s
downloading  75.1%   43.61MiB/s
downloading  75.1%  Unknown B/s
downloading  75.1%  Unknown B/s
downloading  75.1%  Unknown B/s
downloading  75.1%   11.90MiB/s
downloading  75.1%   14.52MiB/s
downloading  75.2%   13.55MiB/s
downloading  75.2%   14.85MiB/s
downloading  75.2%   19.62MiB/s
downloading  75.3%   32.04MiB/s
downloading  75.5%   44.67MiB/s
downloading  75.9%   29.43MiB/s
downloading  76.6%   45.79MiB/s
downloading  78.1%   54.07MiB/s
downloading  78.7%   52.17MiB/s
downloading  78.7%  Unknown B/s
downloading  78.7%  893.67KiB/s
downloading  78.7%  859.72KiB/s
downloading  78.7%    1.48MiB/s
downloading  78.7%    2.55MiB/s
downloading  78.7%    4.63MiB/s
downloading  78.7%    6.30MiB/s
downloading  78.8%   10.76MiB/s
downloading  78.9%   17.71MiB/s
downloading  79.0%   28.96MiB/s
downloading  79.4%   45.02MiB/s
downloading  80.1%   57.62MiB/s
downloading  81.6%   38.52MiB/s
downloading  82.3%   43.05MiB/s
downloading  82.3%  Unknown B/s
downloading  82.3%    1.26MiB/s
downloading  82.3%    1.64MiB/s
downloading  82.3%    2.54MiB/s
downloading  82.4%    3.95MiB/s
downloading  82.4%    6.79MiB/s
downloading  82.4%    7.55MiB/s
downloading  82.4%   12.67MiB/s
downloading  82.5%   17.13MiB/s
downloading  82.7%   19.54MiB/s
downloading  83.1%   33.44MiB/s
downloading  83.8%   50.41MiB/s
downloading  85.3%   43.97MiB/s
downloading  86.0%   52.94MiB/s
downloading  86.0%  Unknown B/s
downloading  86.0%  Unknown B/s
downloading  86.0%    4.58MiB/s
downloading  86.0%    5.26MiB/s
downloading  86.0%    8.84MiB/s
downloading  86.0%    9.62MiB/s
downloading  86.0%   13.96MiB/s
downloading  86.1%   18.09MiB/s
downloading  86.2%   25.31MiB/s
downloading  86.4%   37.29MiB/s
downloading  86.7%   54.73MiB/s
downloading  87.5%   57.63MiB/s
downloading  88.9%   38.35MiB/s
downloading  89.5%   41.13MiB/s
downloading  89.5%  Unknown B/s
downloading  89.5%  Unknown B/s
downloading  89.5%    4.89MiB/s
downloading  89.5%    5.49MiB/s
downloading  89.5%    8.58MiB/s
downloading  89.5%    7.35MiB/s
downloading  89.5%    4.22MiB/s
downloading  89.6%    7.51MiB/s
downloading  89.7%   12.24MiB/s
downloading  89.9%   18.11MiB/s
downloading  90.2%   30.22MiB/s
downloading  91.0%   47.47MiB/s
downloading  92.4%   58.23MiB/s
downloading  93.0%   64.36MiB/s
downloading  93.0%  Unknown B/s
downloading  93.0%  Unknown B/s
downloading  93.0%    4.68MiB/s
downloading  93.0%    4.95MiB/s
downloading  93.0%    7.88MiB/s
downloading  93.0%    6.94MiB/s
downloading  93.0%   11.13MiB/s
downloading  93.1%    3.66MiB/s
downloading  93.2%    6.72MiB/s
downloading  93.4%    9.83MiB/s
downloading  93.7%   17.31MiB/s
downloading  94.5%   29.13MiB/s
downloading  95.9%   25.31MiB/s
downloading  96.6%   30.74MiB/s
downloading  96.6%  Unknown B/s
downloading  96.6%    1.21MiB/s
downloading  96.6%    1.49MiB/s
downloading  96.6%    2.43MiB/s
downloading  96.6%    3.86MiB/s
downloading  96.7%    6.56MiB/s
downloading  96.7%    7.85MiB/s
downloading  96.7%    9.50MiB/s
downloading  96.8%   11.02MiB/s
downloading  97.0%   12.61MiB/s
downloading  97.4%   13.63MiB/s
downloading  98.1%   14.56MiB/s
downloading  99.6%   17.97MiB/s
downloading 100.0%   18.58MiB/s
downloading 100.0% 34.65MiB/s
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '0.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '804.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '536.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '268.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '1340.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '1608.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:18 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '1072.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:19 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '2144.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:19 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '1876.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:19 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '2663.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:19 - [Error] ('ffmpeg', '-nostdin', '-nostats', '-xerror', '-v', '16', '-err_detect', 'explode', '-ss', '2412.00', '-i', '/library/downloads/calibre-web/Youtube/Nat_Geo_WILD/Ice_Cold_Killers_Full_Episode_World_s_Deadliest_182.03k_[2oMXIbOiASI].mp4', '-t', '1', '-f', 'null', '/dev/null')
2024-01-10 01:12:19 - [Error] [https://www.youtube.com/watch?v=2oMXIbOiASI]: No news is good news
2024-01-10 01:12:19 - [Error] Extra media data {'corruption': 0}
2024-01-10 01:12:19 - [Info] lb-wrapper's xklb commands (download) completed successfully.
root@box:~#

Related:

@holta holta added the enhancement New feature or request label Jan 10, 2024
@holta
Copy link
Member Author

holta commented Jan 10, 2024

Example test results — after running the same/above test using the Download to IIAB button:

root@box:~# tree -h /library/calibre-web/
[4.0K]  /library/calibre-web/
├── [4.0K]  Nat Geo WILD
│   └── [4.0K]  Ice Cold Killers (Full Episode) , World's Deadliest (1)
│       ├── [274M]  Ice Cold Killers (Full Episode) , World's - Nat Geo WILD.mp4
│       └── [ 66K]  cover.jpg
├── [4.0K]  config
│   └── [124K]  app.db
├── [340K]  metadata.db
└── [ 13K]  metadata_db_prefs_backup.json

4 directories, 5 files

@holta
Copy link
Member Author

holta commented Jan 10, 2024

Just FYI this PR also:

  • Changes scripts/lb-wrapper chmod permissions from 100644 → 100755 (might as well make it executable out-of-the-box, for easier CLI rapid-testing etc).
  • Deletes scripts/lb-wrapper.greedy in favor of Lines 10-12 near the top of scripts/lb-wrapper here:
# Or download largest possible HD-style / UltraHD videos, to try to force
# out-of-memory "502 Bad Gateway" for testing of issues like #37 and #79
# FORMAT_OPTIONS="--format-sort size"

@holta holta requested a review from deldesir January 10, 2024 06:39
Copy link
Collaborator

@deldesir deldesir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

else
log "An error occurred while running the command. Output: ${OUTPUT}"
log "Error" "Error $rc occurred while running lb-wrapper's xklb commands."
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good you log the return code here.

scripts/lb-wrapper Outdated Show resolved Hide resolved
@holta
Copy link
Member Author

holta commented Jan 10, 2024

@deldesir can you confirm the ~50+ "Errors" (pasted in "Example output" above) are ugly-but-acceptable for now?

i.e. if giving more visibility to these minor errors from xklb and yt-dlp and FFmpeg etc is a central purpose of this PR:

➡️ But should we perhaps relabel Line 64 with log "Warning" $line ⚠️ instead of log "Error" $line ? So that words like "error" and "Error" are reserved for more seriously urgent problems within our logs / logging?

# 1>&2 redirect back-to-STDERR explained at https://stackoverflow.com/a/15936384
eval "${XKLB_FULL_CMD}" \
> >(while read -r line; do log "Info" "$line"; done) \
2> >(while read -r line; do log "Error" "$line" 1>&2; done) &
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2> >(while read -r line; do log "Error" "$line" 1>&2; done) &
2> >(while read -r line; do log "Warning" "$line" 1>&2; done) &

scripts/lb-wrapper Outdated Show resolved Hide resolved
@deldesir
Copy link
Collaborator

Thanks @deldesir and @EMG70 for reviewing. I think it's good enough until @deldesir and others improve on it later. But LMK your thoughts, with any questions/clarifications that are needed.
Example output

You should have noticed a couple of messages in the output that are tagged as errors, but they are related to the normal xklb execution. These messages shouldn't impact the functionality of lb-wrapper; they are just part of the informative output of xklb printed in STDERR instead of STDOUT.

Copy link
Collaborator

@deldesir deldesir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far.

@deldesir deldesir assigned deldesir and holta and unassigned holta and deldesir Jan 10, 2024
@holta
Copy link
Member Author

holta commented Jan 10, 2024

Good Enough: PR #86 introduced most of the ideas here (let's give credit!) but grew somewhat too large to understand.

@deldesir and @EMG70 please put additional recommendations in new tickets or PR's to further improve on lb-wrapper ✅

So we keep TDD momentum with regular, small, clearly explained PR's, Thank You!

ASIDE: This PR #87 can also serve to gently push us all to do the Earnest Hard Work of breaking apart larger ideas (large PR's) into smaller ideas (small PR's) — in order to build a Stronger Collaboration Culture upholding IIAB communities in all countries:

"Programs must be written for people to read and only incidentally for machines to execute." —Hal Abelson, circa 1984

@holta
Copy link
Member Author

holta commented Jan 11, 2024

➡️ But should we perhaps relabel Line 64 with log "Warning" $line ⚠️ instead of log "Error" $line ? So that words like "error" and "Error" are reserved for more seriously urgent problems within our logs / logging?

Some prefer the word "Debug" as it's shorter than "Warning" and somewhat more precise!

Either is fine for me, but brevity's better possibly?

@deldesir and @avni: do you have a preference?

@deldesir
Copy link
Collaborator

deldesir commented Jan 11, 2024

"Debug" aligns more with them IMHO. They are informative messages.

@holta
Copy link
Member Author

holta commented Jan 12, 2024

"Debug" aligns more with them IMHO. They are informative messages.

Done! As part this separate PR:

@holta holta changed the title Suggested improvements for scripts/lb-wrapper Suggested improvements for scripts/lb-wrapper [refactor logging & subprocess substitution, etc] Jan 12, 2024
@avni
Copy link
Member

avni commented Feb 26, 2024

Sorry for missing this earlier,

➡️ But should we perhaps relabel Line 64 with log "Warning" $line ⚠️ instead of log "Error" $line ? So that words like "error" and "Error" are reserved for more seriously urgent problems within our logs / logging?

Some prefer the word "Debug" as it's shorter than "Warning" and somewhat more precise!

Either is fine for me, but brevity's better possibly?

@deldesir and @avni: do you have a preference?

Agree that error seems to be a misnomer here and should be reserved for more urgent problems. Since this is only for informative purposes, I agree with @deldesir that "Debug" makes sense. "Warning" would be used for something wasn't working as expected, imo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants