"Download to IIAB" worked 5 out of 15 tries (trying to d/l individual YouTube videos) — "Tasks" view screenshots show the 10 failures [e.g. formerly live videos]

[holta]

Screenshots from @nzola show 5 successes + 10 failures = 15 total attempts.

This is on his Raspberry Pi 4 where he installed IIAB's version of Calibre-Web a week ago:

HIS OBSERVATIONS:

• He noticed that gigantic/long videos sometimes work and small/short videos sometimes fail.

• He noticed that failed attempts seem to (always or almost always) display "100%" in the Tasks view almost immediately.

QUESTION FOR @deldesir:

• Can you use the above list of 10 failed YouTube video downloads to try to reproduce & diagnose these failures?

Possibly related:

• Thumbnail of ["Downloaded to IIAB"] video is cut off & many files fail to be copied to /library/calibre-web [intermittent failures?] #77
• "Download to IIAB" of Vimeo video ("Staff Pick" URL or regular URL) does not download [and 1 YouTube video too, then 2 YouTube videos worked] #78
• Clicking "Watch in Browser - MP4" works w/ 8GB RAM but crashes calibre-web.service w/ 2GB RAM / memory (MP4 is 1.0GB; users see "502 Bad Gateway" due to OOM crash) #79

[nzola]

admin@box:~ $ sudo su
root@box:/home/admin# iiab-diagnostics

Gathers IIAB diagnostics into 1 file, to accelerate troubleshooting. USAGE:

iiab-diagnostics
sudo iiab-diagnostics # USE 'sudo' FOR MORE
sudo iiab-diagnostics PATH/FILE1 PATH/FILE2 ... # COMPLETE RESULTS !!

Can you provide a short public nickname: (no spaces!) nzola

Compiling diagnostics...

0. HW + SW Quick Summary
   ls -ltr /etc/iiab/install-flags # IIAB install flags

1. Files Specially Requested: (from "iiab-diagnostics PATH/FILE1 PATH/FILE2")

2. Regular Files:

   /.iiab-image
   /etc/iiab/iiab.env
   /etc/iiab/iiab.ini
   /etc/iiab/local_vars.yml
   /etc/iiab/iiab_state.yml
   /etc/resolv.conf
   /etc/network/interfaces
   /etc/hostapd/hostapd.conf
   /etc/wpa_supplicant/wpa_supplicant.conf
   /library/www/html/home/menu.json

3. Content of Directories: (1-level deep)

   /etc/network/interfaces.d
   /etc/systemd/network
   /etc/NetworkManager/system-connections
   /etc/netplan

4. Output of Commands:

   uname -a # Linux kernel
   free # RAM memory
   lscpu # CPU details
   df -h # Disk usage
   df -ah # Disk usage detail
   lsblk # Partition mount points
   blkid # Mount point details
   /etc/fstab
   lshw -C network # Network hardware/interfaces
   ip addr # Network interfaces
   ifconfig # Network interfaces (old view)
   ip route # Routing table
   netstat -rn # Routing table (old view)
   bridge -d link # Bridge for LAN side
   sudo netstat -natp # Ports/Services in use
   systemctl status dnsmasq # Is dnsmasq running?
   sudo journalctl -b 0 -u dnsmasq # dnsmasq log
   networkctl # systemd-networkd status
   nmcli d # NetworkManager status
   sudo journalctl -b 0 -u networkd-dispatcher # networkd-dispatcher log
   rfkill list # Are WiFi and Bluetooth interfaces blocked?
   iw reg get # Detected WiFi country code / legal frequencies
   iw dev # List wireless interfaces
   iw list # List capabilities of all wireless devices
   systemctl status hostapd # Downstream Wi-Fi: Is hostapd running?
   ls -l /etc/wpa_supplicant # Upstream Wi-Fi
   ps -AH # Process hierarchy: staging of hostapd & wpa_supplicant?
   dmesg | grep -i -e 80211 -e 802.11 -e wireless -e wifi -e wlan -e broadcom -e brcm -e bcm -e realtek | head -100 # Wi-Fi firmware/driver msgs
   lspci -nn # Devices on PCI buses
   ls -l /lib/firmware/cypress/43430 # RPi Zero W & 3 WiFi firmware
   ls -l /lib/firmware/cypress/43455 # RPi 3 B+ & 4 WiFi firmware
   env # Environment variables
   node -v # Node.js version
   npm -v # npm version
   /opt/iiab/kiwix/bin/kiwix-serve --version # kiwix-tools
   journalctl -t IIAB-CMDSRV # Admin Console CMDSRV log

5. Firewall Rules:

   sudo iptables-save # Firewall rules
   sudo ufw status verbose # Firewall status & rules
   sudo: ufw: command not found

6. Log Files: (e.g. last 100 lines of each)

   grep -B2 "SEE ERROR ABOVE" /opt/iiab/iiab/*.log # for skip_role_on_error
   /opt/iiab/iiab/iiab-install.log
   /opt/iiab/iiab/iiab-configure.log
   /opt/iiab/iiab/iiab-debug.log
   /opt/iiab/iiab/iiab-network.log
   /opt/iiab/iiab-admin-console/admin-install.log
   /var/log/messages
   /var/log/syslog

COMPLETE! Your diagnostics file (140K, 2563 lines) is:

/etc/iiab/diag/8.1_raspbian-12_2024-01-03_00:52:16_PST_nzola

Publish it to a web pastebin? [Y/n] y

PUBLISHING TO URL... http://sprunge.us/NHWkDx

[nzola]

root@box:/home/admin# pastebinit -b sprunge.us /var/log/calibre-web.log
http://sprunge.us/ff5WL4
root@box:/home/admin# iiab-diagnostics

Gathers IIAB diagnostics into 1 file, to accelerate troubleshooting. USAGE:

iiab-diagnostics
sudo iiab-diagnostics # USE 'sudo' FOR MORE
sudo iiab-diagnostics PATH/FILE1 PATH/FILE2 ... # COMPLETE RESULTS !!

Can you provide a short public nickname: (no spaces!) nzola

Compiling diagnostics...

0. HW + SW Quick Summary
   ls -ltr /etc/iiab/install-flags # IIAB install flags

1. Files Specially Requested: (from "iiab-diagnostics PATH/FILE1 PATH/FILE2")

2. Regular Files:

   /.iiab-image
   /etc/iiab/iiab.env
   /etc/iiab/iiab.ini
   /etc/iiab/local_vars.yml
   /etc/iiab/iiab_state.yml
   /etc/resolv.conf
   /etc/network/interfaces
   /etc/hostapd/hostapd.conf
   /etc/wpa_supplicant/wpa_supplicant.conf
   /library/www/html/home/menu.json

3. Content of Directories: (1-level deep)

   /etc/network/interfaces.d
   /etc/systemd/network
   /etc/NetworkManager/system-connections
   /etc/netplan

4. Output of Commands:

   uname -a # Linux kernel
   free # RAM memory
   lscpu # CPU details
   df -h # Disk usage
   df -ah # Disk usage detail
   lsblk # Partition mount points
   blkid # Mount point details
   /etc/fstab
   lshw -C network # Network hardware/interfaces
   ip addr # Network interfaces
   ifconfig # Network interfaces (old view)
   ip route # Routing table
   netstat -rn # Routing table (old view)
   bridge -d link # Bridge for LAN side
   sudo netstat -natp # Ports/Services in use
   systemctl status dnsmasq # Is dnsmasq running?
   sudo journalctl -b 0 -u dnsmasq # dnsmasq log
   networkctl # systemd-networkd status
   nmcli d # NetworkManager status
   sudo journalctl -b 0 -u networkd-dispatcher # networkd-dispatcher log
   rfkill list # Are WiFi and Bluetooth interfaces blocked?
   iw reg get # Detected WiFi country code / legal frequencies
   iw dev # List wireless interfaces
   iw list # List capabilities of all wireless devices
   systemctl status hostapd # Downstream Wi-Fi: Is hostapd running?
   ls -l /etc/wpa_supplicant # Upstream Wi-Fi
   ps -AH # Process hierarchy: staging of hostapd & wpa_supplicant?
   dmesg | grep -i -e 80211 -e 802.11 -e wireless -e wifi -e wlan -e broadcom -e brcm -e bcm -e realtek | head -100 # Wi-Fi firmware/driver msgs
   lspci -nn # Devices on PCI buses
   ls -l /lib/firmware/cypress/43430 # RPi Zero W & 3 WiFi firmware
   ls -l /lib/firmware/cypress/43455 # RPi 3 B+ & 4 WiFi firmware
   env # Environment variables
   node -v # Node.js version
   npm -v # npm version
   /opt/iiab/kiwix/bin/kiwix-serve --version # kiwix-tools
   journalctl -t IIAB-CMDSRV # Admin Console CMDSRV log

5. Firewall Rules:

   sudo iptables-save # Firewall rules
   sudo ufw status verbose # Firewall status & rules
   sudo: ufw: command not found

6. Log Files: (e.g. last 100 lines of each)

   grep -B2 "SEE ERROR ABOVE" /opt/iiab/iiab/*.log # for skip_role_on_error
   /opt/iiab/iiab/iiab-install.log
   /opt/iiab/iiab/iiab-configure.log
   /opt/iiab/iiab/iiab-debug.log
   /opt/iiab/iiab/iiab-network.log
   /opt/iiab/iiab-admin-console/admin-install.log
   /var/log/messages
   /var/log/syslog

COMPLETE! Your diagnostics file (140K, 2563 lines) is:

/etc/iiab/diag/8.1_raspbian-12_2024-01-03_01:00:05_PST_nzola

Publish it to a web pastebin? [Y/n] y

PUBLISHING TO URL... http://sprunge.us/h1Hrnh

[nzola]

root@box:/home/admin# pastebinit -b sprunge.us /var/log/xklb.log
http://sprunge.us/KrvSRx
root@box:/home/admin# iiab-diagnostics

Gathers IIAB diagnostics into 1 file, to accelerate troubleshooting. USAGE:

iiab-diagnostics
sudo iiab-diagnostics # USE 'sudo' FOR MORE
sudo iiab-diagnostics PATH/FILE1 PATH/FILE2 ... # COMPLETE RESULTS !!

Can you provide a short public nickname: (no spaces!) nzola

Compiling diagnostics...

0. HW + SW Quick Summary
   ls -ltr /etc/iiab/install-flags # IIAB install flags

1. Files Specially Requested: (from "iiab-diagnostics PATH/FILE1 PATH/FILE2")

2. Regular Files:

   /.iiab-image
   /etc/iiab/iiab.env
   /etc/iiab/iiab.ini
   /etc/iiab/local_vars.yml
   /etc/iiab/iiab_state.yml
   /etc/resolv.conf
   /etc/network/interfaces
   /etc/hostapd/hostapd.conf
   /etc/wpa_supplicant/wpa_supplicant.conf
   /library/www/html/home/menu.json

3. Content of Directories: (1-level deep)

   /etc/network/interfaces.d
   /etc/systemd/network
   /etc/NetworkManager/system-connections
   /etc/netplan

4. Output of Commands:

   uname -a # Linux kernel
   free # RAM memory
   lscpu # CPU details
   df -h # Disk usage
   df -ah # Disk usage detail
   lsblk # Partition mount points
   blkid # Mount point details
   /etc/fstab
   lshw -C network # Network hardware/interfaces
   ip addr # Network interfaces
   ifconfig # Network interfaces (old view)
   ip route # Routing table
   netstat -rn # Routing table (old view)
   bridge -d link # Bridge for LAN side
   sudo netstat -natp # Ports/Services in use
   systemctl status dnsmasq # Is dnsmasq running?
   sudo journalctl -b 0 -u dnsmasq # dnsmasq log
   networkctl # systemd-networkd status
   nmcli d # NetworkManager status
   sudo journalctl -b 0 -u networkd-dispatcher # networkd-dispatcher log
   rfkill list # Are WiFi and Bluetooth interfaces blocked?
   iw reg get # Detected WiFi country code / legal frequencies
   iw dev # List wireless interfaces
   iw list # List capabilities of all wireless devices
   systemctl status hostapd # Downstream Wi-Fi: Is hostapd running?
   ls -l /etc/wpa_supplicant # Upstream Wi-Fi
   ps -AH # Process hierarchy: staging of hostapd & wpa_supplicant?
   dmesg | grep -i -e 80211 -e 802.11 -e wireless -e wifi -e wlan -e broadcom -e brcm -e bcm -e realtek | head -100 # Wi-Fi firmware/driver msgs
   lspci -nn # Devices on PCI buses
   ls -l /lib/firmware/cypress/43430 # RPi Zero W & 3 WiFi firmware
   ls -l /lib/firmware/cypress/43455 # RPi 3 B+ & 4 WiFi firmware
   env # Environment variables
   node -v # Node.js version
   npm -v # npm version
   /opt/iiab/kiwix/bin/kiwix-serve --version # kiwix-tools
   journalctl -t IIAB-CMDSRV # Admin Console CMDSRV log

5. Firewall Rules:

   sudo iptables-save # Firewall rules
   sudo ufw status verbose # Firewall status & rules
   sudo: ufw: command not found

6. Log Files: (e.g. last 100 lines of each)

   grep -B2 "SEE ERROR ABOVE" /opt/iiab/iiab/*.log # for skip_role_on_error
   /opt/iiab/iiab/iiab-install.log
   /opt/iiab/iiab/iiab-configure.log
   /opt/iiab/iiab/iiab-debug.log
   /opt/iiab/iiab/iiab-network.log
   /opt/iiab/iiab-admin-console/admin-install.log
   /var/log/messages
   /var/log/syslog

COMPLETE! Your diagnostics file (144K, 2566 lines) is:

/etc/iiab/diag/8.1_raspbian-12_2024-01-03_01:03:28_PST_nzola

Publish it to a web pastebin? [Y/n] y

PUBLISHING TO URL... http://sprunge.us/a4HMy6

[holta]

Thanks @nzola — in summary:

• /var/log/calibre-web.log
  http://sprunge.us/ff5WL4?en

• /var/log/xklb.log
  http://sprunge.us/KrvSRx?en

• iiab-diagnostics
  http://sprunge.us/NHWkDx?en

[deldesir]

Improvements have been made since. Feel free to close this issue as completed.

[holta]

@nzola can you confirm that the success rate for downloading individual videos is now much better than 5 successes out of 15 tries?

Thanks much!

[nzola]

@nzola can you confirm that the success rate for downloading individual videos is now much better than 5 successes out of 15 tries?

Thanks much!

@deldesir @holta I have tried downloading all the failing 7 videos and here are the results:
https://www.youtube.com/watch?v=BK0XGf20l84
https://www.youtube.com/watch?v=VCM8tg_mGSw
https://www.youtube.com/watch?v=w8snrdaoTUs&t=2s
https://www.youtube.com/watch?v=5BO9nhtF0Cc
https://www.youtube.com/watch?v=rbEsoe8F-l4&t=7788s
https://www.youtube.com/watch?v=Drec4XAMJzI&t=6737s
https://www.youtube.com/watch?v=w8snrdaoTUs&t=7s

PUBLISHING TO URL... https://dpaste.com/8SH7KTVFT

[holta]

Huge thanks @nzola for posting your results!

⏩ @deldesir why does the bottommost of the 7 videos (and no others, i.e. just https://www.youtube.com/live/w8snrdaoTUs) show "To force a retry, submit the URL again." ?

Related:

• Formerly live videos fail to download: "failed to download [download] ... does not pass filter (live_status=?not_live); skipping" #188

[deldesir]

Thanks @nzola for your results. All of these videos have something in common. They are "live" videos. IIAB Calibre-Web does not support downloading live videos yet. IIAB Calibre-Web uses xklb as its underlying downloader. xklb doesn't support downloading live videos yet. To make such thing possible, a fix should be shipped to xklb. I'll take care of this.

@holta, the retry message is one of the additions made this week. IIAB Calibre-Web knows the video failed in a previous attempt and lets the user know. It also clears the record for the failed video so the user can force redownload it.

[holta]

⬆️ Great explanation from @deldesir above. ⬆️

@nzola we hope the situation will improve dramatically in coming days 🙏 as testing validates the ~3 main scenarios:

• Add --live flag to support downloading of "live" videos #199 (comment)

[nzola]

Screenshots from @nzola show 5 successes + 10 failures = 15 total attempts.

This is on his Raspberry Pi 4 where he installed IIAB's version of Calibre-Web a week ago:

HIS OBSERVATIONS:

• He noticed that gigantic/long videos sometimes work and small/short videos sometimes fail.
• He noticed that failed attempts seem to (always or almost always) display "100%" in the Tasks view almost immediately.

QUESTION FOR @deldesir:

• Can you use the above list of 10 failed YouTube video downloads to try to reproduce & diagnose these failures?

Possibly related:

• Thumbnail of ["Downloaded to IIAB"] video is cut off & many files fail to be copied to /library/calibre-web [intermittent failures?] #77
• "Download to IIAB" of Vimeo video ("Staff Pick" URL or regular URL) does not download [and 1 YouTube video too, then 2 YouTube videos worked] #78
• Clicking "Watch in Browser - MP4" works w/ 8GB RAM but crashes calibre-web.service w/ 2GB RAM / memory (MP4 is 1.0GB; users see "502 Bad Gateway" due to OOM crash) #79

All the 5 videos have passed now. Thank you @deldesir for your hard work!
PUBLISHING TO URL... https://dpaste.com/58L7DH4WD