Community plugins versus official plugins & plugin stabilization

[tgolsson]

Hey!

This started out as a discussion on Slack, and @sureshjoshi rightfully suggested this belongs more here.

I'd like to start a discussion around where Pants plugins/backends should live and be developed. To contextualize this, I'm developing quite a few plugins that I use to manage my cluster (k8s + kustomize), build docs (mdbook), manage secrets (bitwarden + secrets-plumbing), and building container images (oci). All of these are on PyPi. I also want to contribute a bit to making Rust a viable language for Pants. And since @lassemand's Rust fmt PR, I have a bit of a workflow dilemma.

My issue is that I'm a dogfooding person. I write code, and then I use it until I find an issue. Then I use the fixed version until I need another feature. I add that, and continue. I don't think the way the Pants repo workflows and its release process is very well-suited for that kind of work, and definitely not for quick iteration. It takes a long time for PRs to go through reviews, and even longer to be released. I've gone through four versions of my OCI builder in the same timeframe that Pants has released a single version. If that was upstream and I added a feature now, I fear it'd not be available for downstream users until well after summer.

Thus, when I think about contributing to Rust-support and dogfooding that work, every feature needing to be part of a future release makes it very hard to test and adopt in production incrementally. New users cannot be gained on a timescale shorter than a release cycle. New contributors won't see results until months later. Bugfixes can be backported, but there still needs to be a full Pants release. On contrast, a standalone plugin could potentially automate releases of their single tool to run on every commit, and new contributors will have benefit from their work very quickly.

I've used Bazel professionally for quite a while and I think the rules_x workflow as a plugin flow is much more sustainable as a contribution model. Bazel core is quite lean, and most things are developed independently. This does lead to breakages and incompatibilities, but for the most part it's easy to add new rulesets, and new features in a plugin can be shipped at any point during the lifetime of a bazel version. This means that new contributors (or plugin developers) can form niche communities, with their own interests and their own timelines. There's no need to become a "Bazel contributor" to work on plugins.

In contrast, for Pants, it seems like the assumption is that all backends trend towards going upstream. I think that this makes the barrier-to-entry much higher than it needs to be, as potential contributors have to jump from "Pants user" to "Pants contributor" - there's no "Pants plugin contributor" role in the ecosystem to speak of. I think this is a missed opportunity for Pants--especially in light of the commercial side scaling down--and I think it hurts the longterm ability to build community.

Therefore; my thought is that for most new languages, tools, etc, the default aim should be for it to be developed outside of Pants, at its own timescale - with proper support and guidance from the community at large. This will put more pressure on stabilization and definitely add some initial pains, but I think it'd be more sustainable long term.

Would love to hear what others think about this, especially if you see yourself in this "inbetween" plugin-but-not-Pants contributor role.

---

I'll let others reiterate their points as they feel necessary! @sureshjoshi @thejcannon

[cognifloyd]

Some of the core APIs are going to be reworked. Dependency graph traversal, for example, has had a variety of proposals that would change API there.

I have been working on a backend that produces system packages. But, I really needed to inject additional logic into the dep traversal rules. Had I been working on this in an external repo, I would have had to copy a bunch of code to make my changes, basically duplicating transitive dep tree traversal (and therefore losing out on any of the cached reuse of role results). I hate copying code in most cases (though go lang has taught me some of the benefits of doing so).

Because I'm working on this packaging backend as a backend for pantsbuild/pants, it is natural for me to just make my changes in core pants. I ended up breaking my PRs into several smaller PRs, so the PR trail suggests isolated improvements to core pants. But, I'm only working on core because of my plugin. So, including these plugins with core pants pushes the core to improve to fix these issues.

Also, several proposals have been floating around about multiple dependency fields or some kind of "tagged" dependencies data structure. Changes like this require extensive changes to the plugins. Having those plugins in pantsbuild/pants facilitates taking all those plugin use cases into account when building these significant architectural changes in the plugin API. Surveying external plugins or somehow quantifying which APIs are most used would be difficult to do and difficult to grok how usage would affect these changes.

All of that work to change the dependency traversal logic overlaps/conflicts with other efforts to speed it up by moving more of the core rules into rust.

But, having a good ecosystem of external plugin development would be an excellent thing. I feel we need to get through more of these core refactorings before we have a core plugin API that we can mark stable to better facilitate that. Once parts of the API are documented as stable, then those parts should be subject to a longer deprecation policy similar to the user-facing interface.

[tgolsson]

Good points! Do you think this is the general case though? My experience with six plugins is that most things where I copy/diverge upstream code is so niche it doesn't make sense either way. Though to be fair, my plugins are fairly shallow apart from the OCI one. The one exception is that I have an (IMO) much better publish_python_distribution that can take secrets from any source, just not the host environment. I use that to publish using secrets from my Bitwarden vault.

My worry with this chain of arguments in general is two-fold though: the first one is I wonder how many people turn around in the door instead of contributing to Pants core. I've only made a few PRs, and it's... slow. The second is that "we need to do a few refactorings first..." is a never ending story for many projects. I've said that about so many things before, and people still manage to use them despite them being steaming piles in my eyes. Breaking changes are bad, sure, but it's easier to fix what's broken than what isn't there in the first place.

[sureshjoshi]

I never thought about this discussion point directly - but you're right. I'm on around plugin 20-ish, and about half have required some small tweaks (or alternatively, mid-sized workarounds) to Pants.

Tom's right though, "just a bit more cleanup" is the story of my life.

[cognifloyd]

There are current (awesome) efforts to further automate the release process for pants.

Pants is good at building and publishing multiple distributions. I can imagine a scenario where the plugins are still in the pantsbuild/pants repo, but get released as separate pypi wheels, and could be released more frequently. So, separating the plugin release process from the core pants release process might be interesting, and would facilitate showing off pants' capabilities around monorepos.

That does require automating almost everything around releasing things from pantsbuild/pants. So those automation efforts become even more vital.

[tgolsson]

Yeah; this is a fair point. I guess the question is what's the pros/cons of having everyone push into one place versus a dozen separate repos. I can see that this would work, but what's the gain (from a community/new contributor/...) perspective in your mind? Apart from showcasing the technical part of it.

[sureshjoshi]

I had intended to copy/paste my commentary from Slack, but in lieu of that:

I think there is a happy middle ground between first-party backends + plugins, and then some community-managed plugins. I don't have the slightest clue what that middle ground is just yet.

I'm of two minds with the current Pants monorepo, which is that I like that everything is in once place and updates to Pants proliferate through all backends and plugins. That also means that per-Pants-version plugin maintenance doesn't need to happen, which is not necessarily the case for community plugins which could need versioned releases for each Pants release (at least, until we have a stable API).

What I don't like, though, is that everything slows down when it comes to contributing, testing, CI, etc. As a perfect example, I've had a shell completions task done for weeks, but can't even fmt my local Pants repo anymore, because somehow scala-fmt is throwing exceptions on my machine, even though I've never touched any of the JVM backends before 🤷🏽

This type of problem appears to hit me a lot, so what I've moved to is that I do most of my plugin development in an isolated repo, and then work on potential PRs later on.

Some of these types of issues were recently discussed and there are some potential next actions into making the development/release process more seamless.

Another reason I like the idea of community plugins is from the aspect of, not "everything" needs to be in the monorepo - and each new plugin/backend/thing contributes to future maintenance, CI time, and potential cases of "this would be cool, but we don't want to break XYZ".

What would be an interesting thought experiment to see would be to enumerate all of the current backends and plugins, and to do an honest evaluation of "could this be a community plugin" (however we define that).

Lots of languages/frameworks do this kinda thing already, but I think ansible did it in relatively recent history (like, past few years) - where they split off their core plugins from community plugins.

[huonw]

As a perfect example, I've had a shell completions task done for weeks, but can't even fmt my local Pants repo anymore, because somehow scala-fmt is throwing exceptions on my machine, even though I've never touched any of the JVM backends before 🤷🏽

This is a total distraction to the overall discussion here, but, to work around these problems/unblock this work, could you either disable scalafmt or run on only a subset of the repo? E.g. instead of pants fmt ::, filter manually, like pants fmt --scalafmt-skip :: or pants fmt --only=black ::, or run pants fmt path/to/changed/files:: (and similarly for pants test :: and pants check ::).

Alternatively, just submit what you have as a draft PR and let the pants repo CI catch any mistakes. 😅

[benjyw]

So I think in general we do want to move towards a looser coupling between the Pants core and various backends.

The obvious way to facilitate this is to stabilize the Plugin API and have proper deprecation cycles for it. Currently we allow ourselves to modify the Plugin API at will, which has allowed us to iterate on it rapidly. But it is slowly converging towards stability, and so maybe now is a good time to consider it.

Alternatively, a compromise might be to have a category of plugins that are not part of pants core, but are "blessed" or known by the Pants maintainers, and are subject to the same versioning as Pants, and are tested alongside it (at least during Pants releases).

So, e.g., we know that myplugin 2.18 must be compatible with pants 2.18, etc. And we can know relatively early if pants 2.19 breaks myplugin 2.19.

[thejcannon]

Thoughtdump of what I think we need to provide to make external plugins "first-class-citizens" in Pants: (form https://pantsbuild.slack.com/archives/C0D7TNJHL/p1693498471829329)

• How should they be tested?
• How should they easily support (and test) multiple Pants versions?
• How should they be catalogued for people to find them?
• Can we provide infra/tech for docs hosting of them? So their docs look like ours?
• How can we help plugin resolution (e.g. deps of plugins)
  • I'm thinking the set of plugins for a repo should be backed by a Pex lockfile. e.g. plugins is the requrements specifier, and plugins_lockfile determines how they're loaded/versions/etc...
• How can we help ensure plugins have 0 sdist dependencies
  • The Pex lockfile could easily do this with only-binary set to :all: