Curl errors from the binarycaching http backend

[moritz-h]

---

Is your feature request related to a problem? Please describe.
When using binarycaching with the http backend in manifest mode, any curl error (i.e. server not reachable) immediately fails the whole configure step.

Proposed solution
Handle any curl errors as cache miss, maybe print a warning, but do not error and quit. That a server is temporarily not reachable is not that uncommon, and the build process should not depend of the availability of a cache. If the cache is not reachable just build that package locally as it would happen with a cache miss.
Additionally, vcpkg supports multiple caching backends, so I could theoretically configure multiple cache backends as possible fallback option, but this does not make sense if any single cache backend error stops the build.

[LilyWangLL]

Thanks for posting this issue. Could you please provide the simple repro project? And did this error has any error message?

[moritz-h]

I have a test repo here, checkout the very first commit for the most minimal version: https://github.com/moritz-h/vcpkg-binarycaching-test/tree/f39c728310cd3617614078baa1b8c2437196c49c
and then add this line to CMakeLists.txt Line 24:
set(ENV{VCPKG_BINARY_SOURCES} "clear;http,https://some-non-existing-url.example.com/{triplet}-{name}-{sha},readwrite")

The relevant output is

 Bootstrapping vcpkg before install
 Bootstrapping vcpkg before install - done
 Running vcpkg install
 A suitable version of cmake was not found (required v3.24.0). Downloading portable cmake v3.24.0...
 Downloading cmake...
   https://github.com/Kitware/CMake/releases/download/v3.24.0/cmake-3.24.0-linux-x86_64.tar.gz ->
 [BUILD_DIR]/_deps/vcpkg-download-src/downloads/cmake-3.24.0-linux-x86_64.tar.gz
 Extracting cmake...
 Detecting compiler hash for triplet x64-linux...
 The following packages will be built and installed:
   * fmt[core]:x64-linux -> 9.0.0
     spdlog[core]:x64-linux -> 1.10.0#1
   * vcpkg-cmake[core]:x64-linux -> 2022-07-18
   * vcpkg-cmake-config[core]:x64-linux -> 2022-02-06#1
 Additional packages (*) will be modified to complete this operation.
 Attempting to fetch 4 package(s) from HTTP servers
 Error: curl failed to execute with exit code: 6
 Running vcpkg install - failed
 CMake Error at [BUILD_DIR]/_deps/vcpkg-download-src/scripts/buildsystems/vcpkg.cmake:860 (message):
   vcpkg install failed.  See logs for more information:
   [BUILD_DIR]/vcpkg-manifest-install.log
 Call Stack (most recent call first):
   [SYSTEM_CMAKE_INSTALL_DIR]/share/cmake-3.23/Modules/CMakeDetermineSystem.cmake:124 (include)
   CMakeLists.txt:30 (project)

That curl itself returns an error exit code 6 is fine as the host is actually not resolvable in this example, but it should not bubble to the top to become an CMake error stopping the build. The mentioned vcpkg-manifest-install.log file has the exact same content as above without further details.

[moritz-h]

@LilyWangLL Do you need any further information about this bug or can you reproduce it?
(Not wanting to introduce any pressure on fixing status, just want to make sure there is no waiting for information deadlock).

[LilyWangLL]

@LilyWangLL Do you need any further information about this bug or can you reproduce it? (Not wanting to introduce any pressure on fixing status, just want to make sure there is no waiting for information deadlock).

Thanks for your reply. I will test and confirm this issue soon.

[moritz-h]

@LilyWangLL I think the category port-bug is wrong. This is a vcpkg-tool problem using curl, not using the curl port.

Also here is an even simpler reproduction example, without manifest mode, just run:
./vcpkg install fmt --binarysource="clear;http,https://invalid-url.example.com/{sha},readwrite"

[LilyWangLL]

@moritz-h I use your project https://github.com/moritz-h/vcpkg-binarycaching-test/tree/f39c728310cd3617614078baa1b8c2437196c49c locally, but I only got curl error message: Error: curl failed to execute with exit code: 6. Use the command ./vcpkg install fmt --binarysource="clear;http,https://invalid-url.example.com/{sha},readwrite", I got the same curl error.

Is this expected behavior? I can't repro the following error:

 Running vcpkg install - failed
 CMake Error at [BUILD_DIR]/_deps/vcpkg-download-src/scripts/buildsystems/vcpkg.cmake:860 (message):
   vcpkg install failed.  See logs for more information:
   [BUILD_DIR]/vcpkg-manifest-install.log
 Call Stack (most recent call first):
   [SYSTEM_CMAKE_INSTALL_DIR]/share/cmake-3.23/Modules/CMakeDetermineSystem.cmake:124 (include)
   CMakeLists.txt:30 (project)

My configure command: cmake -G "Visual Studio 17 2022" -A x64 -DVCPKG_TARGET_TRIPLET=x64-windows

[moritz-h]

@LilyWangLL The curl error is what this issue is about.
The other error text is the generic "an error occured, look into the log" message I get always for all kind of vcpkg errors in manifest mode. I can reproduce this output on Windows using your configure command and also on Ubuntu. But in the mentioned vcpkg-manifest-install.log also only the same curl error line appears. So I think only the curl error is relevant for this issue. With the second method /vcpkg install fmt --binarysource="clear;http,https://invalid-url.example.com/{sha},readwrite", I also only see the line error: curl failed to execute with exit code 6.

[LilyWangLL]

Thanks for your reply, the error message error: curl failed to execute with exit code 6. is caused by invalid url, this is by design. Because when using install option --binarysource=http,<url_template>[,<rw>[,<header>]], vcpkg will read caching from this url.

[moritz-h]

Yes, that is the point what this issue is about. I would argue that it may be better to handle connection errors as cache miss instead of a hard error for the whole build.
Temporal errors on network connections should not be that uncommon, like cache server is in maintenance or whatever else could happen.
With the current design I am making the cache availability a hard dependency of the build process. That is not what I think a cache is about. I would expect if the cache is not reachable this may could be a warning but otherwise should be handled the same way as if the cache is missing the specific binary I'm asking for.
Further, vcpkg allows to add multiple caches, so I may use that feature to add several caches as backup if one is not available. But with the current design every single connection error to one of the caches leads to a hard build error.

At the very least an option to allow cache connections to fail would be nice.

I see the example you are using running manually ./vcpkg install it kind of makes sense to error because I just can rerun the command without the cache option. But with manifest mode I probably have the caching server setup fixed somewhere in the project config.

[LilyWangLL]

I will turn this issue into a discussion for other user's opinion.

[recsater]

I fully agree with and support the author's opinion.

[framillien]

Hello, We hit the exact same issue. The cache in our achitecture is made to be an improvement on build speed. Having a build failure when our cache server is not available is not an option. Also, as said in another comment, if we have different levels of caches, we should not fail when a cache server is missing.