packages_path not searched during rez-release

[Anti-Distinctlyminty]

tl;dr

When performing a rez-release only release_packages_path is being searched, resulting in the following error

Resolving build environment: platform-windows arch-AMD64 python-3 sesi_download_api
10:23:13 ERROR    PackageFamilyNotFoundError: package family not found: platform (searched: Z:\packages)

...because platform is defined locally, per machine.

Details

We have multi-user machines so I install to C:\ProgramData\Rez\packages rather than the default .rez directory in the user profile of whatever user happens to be logged in. After installing Rez, a bind quickstart is performed:

SET "REZ_LOCAL_PACKAGES_PATH=C:\ProgramData\Rez\packages"
C:\opt\rez\Scripts\rez\rez.exe bind --quickstart

So the platform package is located in C:\ProgramData\Rez\packages.

My config file looks like this:

{
    "cache_packages_path": "C:\\ProgramData\\Rez\\cache",
    "local_packages_path": "C:\\ProgramData\\Rez\\packages",
    "default_cachable_per_package": {
        "houdini": True,
        "nuke": True,
        "redshift": True,
        "sidefx_labs": True,
    },
    "packages_path": [
        "C:\\ProgramData\\Rez\\packages",
        "Z:\\packages",
    ],
    "release_packages_path": "Z:\\packages",
}

But if I perform a rez-release it appears that the packages_path is not being used to locate packages, but only the release_packages_path, which doesn't contain package families that only make sense per-machine ('platform' in this case).

So I'm either under a misunderstanding of how rez works when it comes to implicit packages, or I misunderstand how to configure rez to releases packages. Some insight would be much appreciated.

[predat]

Hi @Anti-Distinctlyminty,

"rez-release" tool ignore packages installed in the local folder "local_packages_path" to avoid that a package installed in packages_path (usually a shared folder on the network) requires a package that is only local.

What's stopping you from having the 'platform' package on volume Z: ?

If you have different OS or version of OS, you can use "platform_map" to manage that (cf:https://github.com/AcademySoftwareFoundation/rez/blob/master/src/rez/rezconfig.py#L303)

[Anti-Distinctlyminty]

Hi @predat ,

What's stopping you from having the 'platform' package on volume Z: ?

Potential word salad

After some writing, I think it comes down to me being confused about the role of implicit packages. For example, the platform package that is installed during a rez-bind --quickstart creates C:\ProgramData\rez\packages\platform\windows\package.py. My thinking was that the platform-specific package would need to only be available locally, so that rez can ensure that each machine looks up a variable that is suitable for that workstation.

However, my thinking must be wrong because I see that the if I rename C:\ProgramData\rez\packages\platform then call rez-env I still get

requested packages:
~platform==windows           (implicit)
...

I guess I'm having trouble with getting my head around the role of platform packages in a central location. But it's just a mechanism to be able to select the correct variant (please correct me if I'm wrong).

So essentially I just need to do a rez-bind --quickstart --release then releasing with rez-release in the future should all work fine?

---

Edit: I tried that and it's still not working. Calling rez-bind --quickstart --release fails at python with

OSError: [WinError 50] The request is not supported: 'c:\\program files\\python39\\python.exe' -> '\\\\truenas.nymus.local\\mnt\\bigdisk\\software\\packages\\python\\3.9.13\\platform-windows\\arch-AMD64\\os-windows-10.0.19045.SP0\\bin\\python'

Again, this python package works if installed locally, but fails with the network install (mounted drive with NFS). because it's trying to create a hardlink. So I'm back to square one.

[instinct-vfx]

A bit to unpack here.

1. As @predat already said rez-release does never include local packages paths as that could easily lead to a released package with missing dependency. rez-release goes to great length to ensure released packages are working and follow proper workflows (e.g. no local git changes etc.).
2. platform, os, architecture and their cousins are indeed a mechanism to select the proper configuration.
3. If a package is added as an implicit package it will be added as a weak reference only! You can tell in your last example by the ~ in front of the request. This basically means "if platform is part of the final request, it has to be windows" but it does not add platform to the request itself.
4. In your original post i think what happens is that you actually DO have platform as part of the request (either by the package you release or by a package that is a dependency) so it becomes part of the request and rez-release fails as there is no matching variant in your packages_path.
5. You should release your platform and other implicits centrally. You do not necessarily need them locally

With all that said and out of the way a word of caution: You need to be very careful with rez-bind on windows. It's fine for the implicits but do not use it for python as it is broken (and also meant to be deprecated. The reason it failed for you is one of the issues as it tries to create a symlink.

My suggestion for now would be to use rez-bind to create platform, os and arch packages using release mode but for no other packages. As said you do not necessarily need these locally unless your platform is not released centrally (in which case you can of course just release the new platform centrally).

[Anti-Distinctlyminty]

Thanks for the detailed reply. I've removed all packages previously released centrally by rez-bind except os, platform and arch. A rez-release now gets further, but now gets stopped as my build requires python-3. Is there a workaround or should I just create my own python-3 package that installs centrally?

[instinct-vfx]

I have a small CLI tool here for creating python packages on windows automatically: https://github.com/techartorg/rez_utils/tree/main/rezify_python

[Anti-Distinctlyminty]

And with that, I have finally been able to perform a release. Thank you @predat and @instinct-vfx, it's appreciated :)