Feature request: "proxy" packages that support direct editing of files without rebuild

[chadrik]

Hi all,
I think this topic has probably come up at every studio that has ever used Rez. I think adoption would be faster, and developer workflows would be simpler and safer if Rez supported this concept natively. Let me know what you think.

Goal

As a Rez developer I want a way for local rez packages to reference live files, so that I can rapidly iterate without needing to rebuild after every change.   Files that I might want to edit in-place include interpreted source code (python, shell scripts), configuration files, and the package.py files themselves. 

Motivation

In a typical Rez studio environment, there are many rez packages which do not require compilation as part of the build process.  The build commands for these packages may simply copy some folders and files to the install location.  Some Rez packages are used to bundle together particular packages and versions, and so the package.py file is the sole file that requires editing and installing.

In these situations, it is a drain on developer time to rebuild packages each time that a file is edited. 

Proposal

• Add a new --proxy flag to rez build: when a package is built using rez build --proxy --install Rez will install a special proxy package.py that redirects to the source package.py

• This proxy package will be installed to the local packages path, so that it will be found by the normal Rez discovery mechanism

• It will have the bare minimum of attributes (such as name and version), along with two new attributes

  • proxied_package: path to the package.py that produced it

  • proxied_root:  (optional) path which should override the root variable within commands.  This can be provided via  rez-build --proxy --proxied-root=..., otherwise it is taken from the source package’s default_proxied_root if present.  

• When Rez resolves a proxy package.py, it will load the source (aka "proxied") package.py indicated by the proxied_package attribute. If present, it will inject the value of proxied_root as the root variable within the commands context. This allows packages which have a similar source and release folder layouts to "just work". 

• There will also be a new context variable called is_proxied which will be set to True when a package is loaded via a proxy stub; this can be used to add conditional logic to the commands context.

• For safety, proxy installs should not be supported by rez release

• Various commands which list packages in a resolve should indicate when a local package is proxied

Example files

Here's an example package.py that uses this feature:

name = "foo"
version = "1.1.0"

# path is relative to the location of this package.py file
default_proxied_root='.'

def commands():
    # for this env var, the source and release folders layouts are the same, so replacing root "just works"
    env. OCIO.prepend("{root}/ocio")

    # for this env var, there's a difference between the source and release layout, so we have to add some logic
    if is_proxied:
        env.PYTHONPATH.prepend("{root}/python")
    else:
        env.PYTHONPATH.prepend("{root}/site-packages")

Here's the "proxy" package that would be installed into the local packages path:

name = "foo"
version = "1.1.0"

proxied_root='/path/to/foo/'
proxied_package='/path/to/foo/package.py'

Rejected ideas

Add machinery inside package.py files

We could use the very flexible package.py files to emulate this behavior. One idea was to modify our build command to install a .redirect.json file alongside the package.py, then when the commands runs we would check for the existence of this file, and change the behavior.

Cons:

• Checking for the presence of the .redirect.json file adds overhead even when a package is not in editable mode. 

• The package.py itself must be re-installed after editing for changes to take place (unless we also use a symlink to the package.py which adds further complexity and room for failure).

• Since Rez itself is not aware of this concept, when Rez prints a resolve it will not indicate whether the package is a "proxy" pacakge or not.  This can be confusing and dangerous for developers

Use pip install --editable

Cons:

• The package.py itself must be re-installed after editing for changes to take place (unless we also use a symlink to the package.py which adds further complexity and room for failure).

• Since Rez itself is not aware of this concept, when Rez prints a resolve it will not indicate whether the package is a "proxy" pacakge or not.  This can be confusing and dangerous for developers

• Only works for Python projects

[chadrik]

Also, just to be clear, if we all agree this is a good feature, my team can implement it and submit a PR for it.

[maxnbk]

I suppose I'd have a couple questions.

Is the intent that the entire package is treated as rooted-at the source? It's very common for packages to only sort of... Actually become their structured whole when built into their install folder. Basically, I see this as only useful for python packages, where the source structure often/usually (but not always) is identical to the target structure (even then, it can be.. Not that uncommon for there to be differences, or extra things introduced or renamed or whatever in the build process).

I ask because.. Maybe it makes sense to hang this feature on a mechanism that, perhaps, matches up with a "very clean way to express a packages whose source is identical to its target". For instance, I'm wary of developers using this, and then somehow expecting that their "not identical source" doesn't act like their "the way it usually builds and installs" target. This is poorly named, but I'm wondering if an attribute like "easy_copy" could express a type of package whose build command is an automatic "equivalent to rsync -larv" style copy, and that automatically enables or disables such a feature, to basically marry the idea that "only if your source and target would be identical, can you reasonably expect to use this in a non-side-effect-ensured-way".

Of course, I can see reasons why power users may not like that, but I've seen a lot of developer misunderstandings around this kind of idea, and while the feature is interesting, I think it may need some kind of guardrail, even if it's just a warning that "You're using a proxied package, results may vary depending on your setup", or something to that effect.

[chadrik]

Is the intent that the entire package is treated as rooted-at the source?

It's most useful for packages that follow this paradigm, but as I've shown in my example package above, you could use the is_proxied variable to handle differences in source vs install layout.

This is poorly named, but I'm wondering if an attribute like "easy_copy" could express a type of package whose build command is an automatic "equivalent to rsync -larv" style copy, and that automatically enables or disables such a feature.

I considered something like this as well, but I don't want to hamstring power users. Here's a compromise:

1. introduce a new built-in build_system called copytree, that does a simple copytree from root to install directory.
2. if a package is built in proxy mode and build_system != "copytree" then rez prints a warning such as "When using proxy mode, if source and install layouts are not the same it could result in unexpected behavior. Consider setting build_system to copytree."

I'm not sure if a build_system is the right way to handle this but it seems like an easy way for a package to opt-in to this behavior. I also considered build_command=True, but this seems too opaque.