Retry failed downloads before giving up

[heydojo]

Builds randomly fail for me if there is network traffic congestion or some other connection issue.

For example, I see:

Starting package 20/31: jasper:x64-windows
Building package jasper[core]:x64-windows...
-- Downloading https://github.com/mdadams/jasper/archive/version-2.0.16.tar.gz...
-- Downloading https://github.com/mdadams/jasper/archive/version-2.0.16.tar.gz... Failed. Status: 22;"HTTP response code said error"
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:175 (message):

      Failed to download file.
      If you use a proxy, please set the HTTPS_PROXY and HTTP_PROXY environment
      variables to "https://user:password@your-proxy-ip-address:port/".
      Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

Call Stack (most recent call first):
  scripts/cmake/vcpkg_from_github.cmake:132 (vcpkg_download_distfile)
  ports/jasper/portfile.cmake:3 (vcpkg_from_github)
  scripts/ports.cmake:90 (include)


Error: Building package jasper:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `.\vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: jasper:x64-windows
  Vcpkg version: 2020.02.04-nohash-external

Additionally, attach any relevant sections from the log files above.

But I then re-run the vcpkg install command and the installation transaction succeeds without the same error.

Proposed solution

I think the download client (aria2 iirc) needs to try three times with approximately a three second pause in between each retry before giving up and allowing the build to fail.

Other package managers elsewhere do this and retry at least once in the event of a download error.
Some also provide cached mirror copies of source archives as a fall back in the event of a download error, which may also be an area of enhancement for another time.

Additional context
The reason the current method is bad is that if you currently use vcpkg with CI/CD then a single vcpkg download failure may provide a build failure for your build or test environment.

Three download retry attempts for source archives would be optimal to try to minimise such failures.

[heydojo]

https://github.com/microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_download_distfile.cmake#L127

Please consider adding some or all of the following:

• -m 3 https://aria2.github.io/manual/en/html/aria2c.html#cmdoption-m
• --retry-wait=3 https://aria2.github.io/manual/en/html/aria2c.html#cmdoption-retry-wait
• --max-file-not-found=3 https://aria2.github.io/manual/en/html/aria2c.html#cmdoption-max-file-not-found

Thanks!

[JackBoosY]

I think this issue was resolved.

[albertvaka]

I don't think it was... or if there's an option to retry, I can't seem to find it. What is the option?

[JackBoosY]

@albertvaka After double confirm, it isn't.