Provide a staging registry for testing of updates before PRs are merged

[IanButterworth]

Problem

It's currently not possible/hard to test the product of a ygg PR in the wild (outside of build tests) before merge. You can build locally but testing the ygg PR output specifically would be best.

Suggestion

A staging registry for yggdrasil (say https://github.com/JuliaPackaging/YggdrasilStagingRegistry) where:

• versions are registered for successful PR commits, with the artifacts pointing to the correct pre-release location
• versions are overwritten for new commits when the version doesn't change (LocalRegistry.jl doesn't currently support this AFAIK)
• versions are yanked (or deleted?) once the PR is merged & registered in General or closed

The user can then add the registry during testing and remove once done. (Probably good to suggest people don't leave it on to protect against pre-release breakage)

This may make it easier for ygg CI to be setup in github actions, to run a test payload with the new version before PR merge and JLL registration on General.

[imciner2]

It's currently not possible/hard to test the product of a ygg PR in the wild (outside of build tests) before merge. You can build locally but testing the ygg PR output specifically would be best.

Have you seen differences between the local and ygg build outputs? They should generally be the same, since it should get the same dependency versions.

You can always deploy the built JLL to a GitHub and then use that, but the main annoyance is actually pointing the dependencies to a git URL is broken and the only way I got it to work was to explicitly dev my JLL build into the environment I run BB from. That let me test how changes in libblastrampoline would propagate to consumers like SuiteSparse for instance.

[IanButterworth]

Have you seen differences between the local and ygg build outputs?

I can't say I have, but I think a big part of this is being able to run standard downstream package CI with the pre-release version on the various platforms, which is obviously not very compatible with the local build approach.

There are ways to do all this, just not easy ways, so downstream testing seems rare.

[imciner2]

I can't say I have, but I think a big part of this is being able to run standard downstream package CI with the pre-release version on the various platforms, which is obviously not very compatible with the local build approach.

Putting my Ygg merger hat on, having those results visible in the CI here is more useful, so I would say this usecase should be setup by having another set of pipelines in the AZP flow here that pulls the artifact from the build steps, devs that, and then runs the tests of a defined set of packages. Those packages could be specified in the build_tarballs somehow to indicate the "main consumers" of a package (usually it is probably only one main consumer, i.e. the one that provides the wrapper).