An attempt to download [a YouTube channel] resulted in an accumulation (long list) of "unavailable videos" in Tasks, from all current and previously attempted failed downloads [why is the very same video individually downloadable?]

[EMG70]

Describe the bug/problem
An attempt to download playlists results in an accumulation of long list in Tasks " unavailable videos" from all current and previously attempts.

A clear and concise description of what the bug is. If you are asking for support, please check our Wiki if your question is already answered there.

Unavailable videos for any reason are expected to show as those that have failed for the playlist not cumulative list from previous downloads.

To Reproduce

Download
Steps to reproduce the behavior:

1. Go to 'https://www.youtube.com/@agribusinesswithwadzanaima275' name of playlist
2. Click on 'https://www.youtube.com/watch?v=EiMFnDquvdM' failing video being added to the list of unavailable videos
3. https://www.youtube.com/watch?v=b4RLo3tgpoA failing video being added to the list of unavailable videos
4. Scroll down to '....'
5. See error

Logfile
SUDO IIAB-DIAGNOSTICS - https://paste.centos.org/view/10fc9a97

calibre-web.log

Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.

Expected behavior

Unavailable videos for any reason are expected to show as those that have failed for the playlist not cumulative list from previous downloads.

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Windows 10/Raspberry Pi OS]
• Python version: [e.g. python2.7]
• Calibre-Web version: [e.g. 0.6.8 or 087c4c5 (git rev-parse --short HEAD)]:
• Docker container: [None/LinuxServer]:
• Special Hardware: [e.g. Rasperry Pi Zero]
• Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR]
  

Additional context
Add any other context about the problem here. [e.g. access via reverse proxy, database background sync, special database location]
This is a members only video https://www.youtube.com/watch?v=MfRdmsbs3R8 but also gets added to list of unavailable accumulative list.

This video https://www.youtube.com/watch?v=EiMFnDquvdM fails to download as part of playlist https://www.youtube.com/@agribusinesswithwadzanaima275 but successfully downloaded as an individual url.(last line of screenshot)

The results are however not consistent ,some individual public urls are still failing. eg https://www.youtube.com/watch?v=AT4khVKBBkg which plays well in youtube.

[deldesir]

Here is why:

calibre-web/scripts/lb-wrapper

Line 82 in 486b7bb

xklb_full_cmd="lb tubeadd '${XKLB_DB_FILE}' '${URL_OR_SEARCH_TERM}' --force ${VERBOSITY}"

This line of code from lb-wrapper ensures, with the use of --force, to update already existed metadata for a given path as explained in https://github.com/chapmanjacobd/library/blob/main/xklb/usage.py#L570-L572

• When a video url is submitted, the path is the video url, the metadata for this url is updated. The download goes through
• When a Youtube Playlist is submitted, the path is the playlist url, the metadate for the playlist url is updated, but the video urls path are not because they are secondary. Previous failures are seen and handled then.

[holta]

@deldesir which below are most relevant? (In helping us explain — and decide what to do next!)

• PR UPDATE paths instead of REPLACE + FOOTNOTE re: Archivists-vs-YouTube having conflicting world views, on how to define "Duplicate Videos" #125 (added --force to tubeadd)
• Subsequent playlists redownloads fail #150
  • PR Enable subsequent [re]downloads of playlists #151

[holta]

@EMG70 just FYI @deldesir is looking into this, but it might not be easy!

Something equivalent to xklb tubeadd's --force might be needed for entire playlists — not just single videos ?

(Sorting out how xklb helps us with download errors & metadata-extract.py for entire playlists — not just single videos — likely needs work!)

[EMG70]

@EMG70 just FYI @deldesir is looking into this, but it might not be easy!

Something equivalent to xklb tubeadd's --force might be needed for entire playlists — not just single videos ?

(Sorting how xklb helps us with download errors & metadata-extract.py for entire playlists — not just single videos — likely needs work!)

Thank you

[holta]

1. Go to https://www.youtube.com/@agribusinesswithwadzanaima275 name of playlist

@EMG70: Were you aware that you are trying to download an entire YouTube Channel here — rather than a standard playlist?

@deldesir is investigating if this might possibly be exacerbating the problem.

1. FWIW this particular channel contains 220 videos and 0 [zero] playlists...)

   

2. Whereas many YouTube channels contain playlists — e.g. https://youtube.com/@bbc has 6 tabs including "Playlists" ...

   

Tangentially, @deldesir is being affected by IP addresses (cloud VPS IP addresses especially?) being banned / blocked by YouTube, as mentioned below...

• YouTube video(s) not downloadable (or viewable?) from certain IP address(es): "Sign in to confirm you’re not a bot. This helps protect our community" #243

[EMG70]

Yes I was aware of her 220 agricultural videos under her channel ,unfortunately she doesn't group them by Playlist.I have downloaded same videos successfully before my calibre-web got corrupted after upgrading to 24.04 in last two weeks .
It could well be the new Youtube restrictions which came up recently.

[holta]

@deldesir suspects the large number of videos (220 in this case) might be causing this problem?

Regardless whether or not contained in a standard playlist.

[deldesir]

Yes, the number of videos might trigger a restriction. 220 videos were processed to get their metadata extracted so 100 of them could be downloaded. Tests with single videos downloaded in another VM in another location went through without a problem.

[EMG70]

Thank you ,I think that explains it .This said calibre-web /server was built circa March 2024, probably before the 100 video limit was implemented ,so explains why i initially managed. I guess things changed when i ran sudo iiab-update-f.

[holta]

@EMG70 can you please retest, on a completely clean/fresh VM? @deldesir too should investigate & test wherever he can help shed light, as I just tried "Download to IIAB" with https://www.youtube.com/@agribusinesswithwadzanaima275 and the 220 videos are downloading perfectly so far... (100% success with the first 10 videos anyway!)

[EMG70]

Will do

[holta]

Will do

Thanks! FYI @EMG70 33 videos (of 220) downloaded successfully over about 30min, so I cut it off by running systemctl restart calibre-web

PS I could have run rm /library/calibre-web/xklb-metadata.db to force the remaining ~187 videos not to try again — which will likely to happen the next time "Download to IIAB" is attempted! ASIDE: In future, this kind of canceling of stale downloading jobs (might!) be done automatically by something like... ?

• PR Prevent [upcoming and truly live YouTube] videos from carrying over [to] subsequent download tasks [& discussion of UTC for code safety — avoid TZ code complexity wherever possible!?] #212

[EMG70]

Tested with a new VM ,100 videos successfully downloaded from the channel https://www.youtube.com/@agribusinesswithwadzanaima275.

calibre-web.log

[holta]

@deldesir should (speculate!) as to the true root cause:

What really changed since a week ago, to cause this to start working again?

[holta]

@EMG70 when exactly did you try downloading "up to 300 videos" instead of "up to 100 videos" ?

(Thanks for clarifying, just in case this happens to be related!)

[holta]

@EMG70 when exactly did you try downloading "up to 300 videos" instead of "up to 100 videos" ?

@EMG70 is the timeline/summary below accurate, to help us all understand?

• Prior to 2024-08-06, @EMG70 tried downloading "up to 300 videos" on an Ubuntu 23.10 machine, that on 2024-08-06 had problems (likely due to Moodle dev branch renamed from 'master' to 'main' iiab#3791) after upgrading to Ubuntu 24.04.

  • (While very unfortunate indeed... that would appear to be... completely separate from understanding this issue An attempt to download [a YouTube channel] resulted in an accumulation (long list) of "unavailable videos" in Tasks, from all current and previously attempted failed downloads [why is the very same video individually downloadable?] #238 here ??)

• On 2024-08-19 and 2024-08-20, @EMG70 tried and failed to download of youtube.com/@agribusinesswithwadzanaima275 to his newly installed IIAB on Ubuntu Desktop 24.04 (i3-4130T, 8G RAM, 469G disk): https://paste.centos.org/view/10fc9a97

[holta]

@deldesir please help us understand what could have caused most of 100 videos to fail to download (Unavailable Video(s):) on 2024-08-20 (this issue #238 !)

• Speculative ideas need to be circulated, to promote deeper understanding among all, in case this-or-similar happens again!

• ASIDE: @EMG70 suspects that if he had not updated calibre-web on his old 23.10... he would still be able to download above 100 videos? (Which certainly interesting, and somewhat tragic that he lost a few "bookshelves" of downloaded videos, that appears to be 100% off-topic to this specific issue right here!)

[holta]

@deldesir please help @EMG70 & all of us deepen our intuition as to what might-or-might-not have been happening to cause this last week!

@EMG70 asks:

This accumulation of failed videos [ Unavailable Video(s): ] was also happening on a VM last week when i tried same channel,probably meaning something changed on latest script ?

[deldesir]

@deldesir please help us understand what could have caused most of 100 videos to fail to download (Unavailable Video(s):) on 2024-08-20 (this issue #238 !)

Since reproducibility of the issue is not possible anymore and the log doesn't contain an error relevant to the failures, I am afraid we'll have to rely on untested assumptions.

What I know is at the time of the bug, yt-dlp 2024.08.06 and xklb 2.9.026 were in use.

[holta]

at the time of the bug, yt-dlp 2024.08.06 and xklb 2.9.026 were in use.

• yt-dlp 2024.08.06 ✅
• xklb 2.9.026 ⚠️

Ok! xklb 2.9.026 could easily be tested to check if that might be the cause?

As outlined in https://github.com/iiab/calibre-web/wiki#xklb-and-yt-dlp-backend-integration — everyone should do this regularly to keep assumptions grounded in reality:

To revert to a prior version, e.g. 2.9.026:

pipx install --force xklb==2.9.026

[holta]

To revert to a prior version, e.g. 2.9.026:

pipx install --force xklb==2.9.026

Much safer method below! (as wiki doc explains)

pipx uninstall xklb
pipx install xklb==2.9.026

[deldesir]

Tested with xklb 2.9.026 without a problem. Might be related to xklb-metadata.db being corrupted at the time.

[EMG70]

Thanks all .Probably off topic,Am i right in saying one can edit code in /usr/local/calibre-web-py3/cps/constants.py if they really need to download more than 100 videos ?
as on screenshot below:

[holta]

/usr/local/calibre-web-py3/cps/constants.py

Yes this is off-topic 😄

But of course feel free to try! And if YouTube eventually blocks your IP address (e.g. #243 or similar), or you run into any other problems, definitely let us know!

RECAP: @EMG70 modified MAX_VIDEOS_PER_DOWNLOAD = 100 to MAX_VIDEOS_PER_DOWNLOAD = 300 in the past... and everyone is free to try doing similar...

calibre-web/cps/constants.py

Lines 51 to 52 in 486b7bb

	# Maximum number of videos to download, from a playlist or channel
	MAX_VIDEOS_PER_DOWNLOAD = 100