Overall "Download to IIAB" workflow

[deldesir]

(Attempt to explain the logic behind success and failure of a video download)

I. Metadata Fetch Task

1. lb tubeadd is run:

   • The lb tubeadd command is executed to fetch metadata for the media URL.
     • media.webpath is set to the original URL.
     • media.path is initially set to the original URL.
     • media.error is set to NULL.

   Notes:

   • lb tubeadd will fail due to an unsupported URL or if yt-dlp returns no data, the metadata fetch task will not proceed further, and no download task will be created for the URL.
   • lb tubeadd will silently exit when the URL is already present and nothing is recorded in media.error (i.e unavailable fragments case)

2. Sending URL to "Download Task":

   • After successfully running lb tubeadd, the metadata fetch task checks the media table in the database:
     • If media.duration is not NULL and media.duration > 0, the URL is considered valid and is sent to the download task.
     • If media.duration is NULL or media.duration = 0, the URL is marked as unavailable, and the task will log an error and terminate.

   Notes:

   • Only URLs with valid durations are forwarded to the download task, ensuring that only downloadable videos are processed further.

II. Download Task

1. Running lb dl:

   • The lb dl command is executed to download the media file.
     • media.webpath remains the original URL.
     • media.path should be updated to the actual file path on the disk upon successful download.
     • media.error remains NULL if no errors occur during the download process.

   Notes:

   • If lb dl fails to download the video, media.path will not be updated correctly (i.e., it might remain as an URL or contain an invalid path).
   • The task will log any errors and update the media.error field accordingly, except for videos failed due to unavailable fragments (to be fixed)

2. Upon Successful Download:

   • If the download is successful:
     • A timestamp is appended to media.webpath to indicate the time of download.
     • The downloaded file's path is stored in media.path.

Detailed Steps for Success and Failure

Success:

1. Metadata Fetch Task:

   • lb tubeadd runs successfully.
   • Metadata (including duration) is fetched and stored in the database.
   • URL is forwarded to the download task if media.duration is valid.

2. Download Task:

   • lb dl runs successfully.
   • Video is downloaded, and the file path is updated in the database.
   • media.webpath is updated with a timestamp.
   • Task status is set to STAT_FINISH_SUCCESS.

Failure:

1. Metadata Fetch Task Failure:

   • lb tubeadd fails due to an unsupported URL or no data returned.
   • Task logs an error and terminates without creating a download task.

2. Download Task Failure:

   • lb dl fails to download the video (e.g., due to network issues, video removal, or other errors).
   • media.path remains incorrect or NULL, leading to potential NoneType errors when accessed.
   • media.error is updated with the error message.
   • Task status is set to STAT_FAIL.

TODO:

• Calibre-Web/XKLB Tuning:

  • Understand delays on downloads (likely due to xklb retrying...) Why ~100 minute ETA to complete "Metadata Fetch" stage (seems very slow) for a playlist of 35 short videos? [Metadata Fetch: ... failed: unsupported operand type(s) for /: 'NoneType' and 'int'] [hundreds of videos stuck in xklb-metadata.db, painstakingly processed every time, but never downloading?] #189

• Error Handling:

  • Set an error message for videos failed due to unavailable fragments Remove residual fragments from failed video #157, @EMG70's YouTube playlist downloading problems 2024-06-10: (1) (This is taking longer than expected) (2) Metadata Fetch: [YouTube video URL] failed: 'NoneType' object cannot be interpreted as an integer [@avni results: (3) failed to download: None (4) it keeps trying to redownload failed videos] #178
  • Prevent playlist from failing due to a failed video

• Database Operations:

  • Ensure media.error exists, if not create it. Ensure error and live_status columns are created from the start #191
  • The tasks interact with xklb-metadata.db to fetch and update media metadata and paths. Ensure failures doesn't prevent its reuse
  • Proper database transactions and error handling ensure that the database state remains consistent even in case of failures, but views-per-day calculation may fail if one of requested value is missing. Check for likeliness.

• Redownloading / Retrying

  • Currently, redownload of failed videos is not supported. Remove media.error or apply timestamp to their media.webpath