Consolidate port and tool downloads to a single point of truth

[GordonSmith]

---

Is your feature request related to a problem? Please describe.
vcpkg currently manages third party downloads in variety of ways, with different rules / meta locations and local caching. This can be problematic when trying to archive and reconstruct a specific development environment (build machines with limited egress rights for example).

Meta information:

• Port source code - URLs inside files in each port.
• vcpkg.exe - URLs inside shell scripts: scripts/bootstrap.*
• tooling - URLs inside xml file: scripts/vcpkgTools.xml
• others - msys, perl, etc. inside various cmake files: scripts/cmake/*

Local caching:

• Port source code - vcpkg/downloads (configurable).
• vcpkg.exe - vcpkg (non configurable?)
• tooling - vcpkg/downloads (non configurable???)
• others - ???

Proposed solution

1. Add optional new section(s) to the manifest file called "tools" and "tools-overlays"
2. Add a default manifest into the vcpkg repo (the root of the ports folder ???) to replace scripts/vcpkgTools.xml.
3. Move the logic for downloading and "installing" tools into the ports folder (ports/cmake, ports/7zip or ports/tool-7Zip???)

New rules:

1. Manifest will automatically "extend" the default manifest (bonus marks for supporting a specific "extends" label in the vcpkg.json to override this).
2. The user can then "override" the default tools, by specifying their own inside their vcpkg.json manifest adhering to same rules as regular ports
3. The abi_info will reflect the above (bonus marks to be able to "flag" each "tool" with an "abi": false to ignore it)

Benefits:

1. Tools will get the same multilayered cache support as regular ports.
2. Tools can be fetched from local NuGet servers for internal build machines. (bonus marks to have the option to download the tools for ALL os's)
3. Keeping a local copy of the "downloads" folder will now also include the build tools
4. More control for the user, while maintaining the current rules by default
5. Simpler control over (some of) the ABI info
6. Extendable - It would be a logical place to add NSIS, CodeSign, mono etc.

Notes:

• I am sure not all tooling will fit into the above paradigm (vcpkg.exe for example) - if v1 only migrated the contents of vcpkgTools.xml it would still be a huge win IMO

Sample "default" vcpkg.json

{
   "$schema":"https://raw.githubusercontent.com/microsoft/vcpkg/master/scripts/vcpkg.schema.json",
   "name":"vcpkg",
   "version":"2023.01.09",
   "tools":[
      {
         "name":"cmake",
         "os":"linux",
         "version":"3.24.0",
         "exeRelativePath":"cmake-3.24.0-linux-x86_64/bin/cmake",
         "url":"https://github.com/Kitware/CMake/releases/download/v3.24.0/cmake-3.24.0-linux-x86_64.tar.gz",
         "sha512":"8eb604b80ee6c771d43cc3e430b2a1065319167cb4913258147482d6ef8d6b83ff4451e3173b0ce6e0546a35d6731f334a72479855abe1958b25678cafda21d0",
         "archiveName":"cmake-3.24.0-linux-x86_64.tar.gz"
      },
      {
         "name":"cmake",
         "os":"freebsd",
         "version":"3.20.4",
         "exeRelativePath":"usr/local/bin/cmake",
         "url":"https://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/All/cmake-3.20.4.txz",
         "sha512":"3e5b675d7ff924f92996d912e2365582e687375109ef99c9073fb8196bb329243a406b218cf1358d7cc518988b311ce9e5bf87de4d64f2e6377b7c2bc8894475",
         "archiveName":"cmake-3.20.4.txz"
      },
      {
         "name":"git",
         "os":"windows",
         "abi": false,
         "version":"2.7.4",
         "exeRelativePath":"mingw32\bin\\git.exe",
         "url":"https://github.com/git-for-windows/git/releases/download/v2.35.1.windows.2/PortableGit-2.35.1.2-32-bit.7z.exe",
         "sha512":"f98df16641a615bbc36c5c319c78abb780d824ff35ed5f2a095e2d5fce2acacfc7f6532c96f1c81d3d3cd8a642858cbd9de0e0e5fcde9ca11e6ad9f6598bb82f",
         "archiveName":"PortableGit-2.35.1.2-32-bit.7z.exe"
      },
      {
         "name":"git",
         "os":"linux",
         "abi": false,
         "version":"2.7.4",
         "exeRelativePath":"",
         "url":"",
         "sha512":""
      }
   ]
}

[dg0yt]

Start with this:
https://github.com/microsoft/vcpkg/blob/master/docs/users/assetcaching.md

[dg0yt]

Which tools are actually missing from your downloads folder?

[GordonSmith]

Start with this: https://github.com/microsoft/vcpkg/blob/master/docs/users/assetcaching.md

Thanks - am I correct and saying there is still no NuGet support?

Which tools are actually missing from your downloads folder?

My bad - I hadn't been using the --downloads-root=xxx with my vcpkg fetch nuget | tail -n 1 command - you are correct re the downloads file. To be pedantic I would say the "vcpkg.exe" and "msys"?

I added benefit 6...

[dg0yt]

To be pedantic I would say the "vcpkg.exe" and "msys"?

The msys downloads all properly go to downloads. That's why msys packages which where removed upstream always bite a) new user and b) users which didn't care for asset or downloads caching in there CI setups.

[Neumann-A]

You can also set environment variable VCPKG_DOWNLOADS instead of passing the option.

[LilyWangLL]

cc @BillyONeal