Upgrade the default version of black from 23.3.0 to 24.4.2

[huonw]

This upgrades the default black version from 23.3.0 to 24.4.2. This has two "user-visible" changes:

• formatting changes: we're ticking over into a new year, and thus pick up the "2024" stable style in 24.1.0: https://black.readthedocs.io/en/stable/change_log.html#id18 . Depending on the codebase, this potentially results in significant reformatting (and does so for Pants).
• requires 3.8+: as of 23.7.0, running under Python 3.7 is no longer supported (https://black.readthedocs.io/en/stable/change_log.html#id60), and thus the interpreter constraints need to be bumped to 3.8+. It still supports formatting 3.7 code, while running on a newer interpreter.

There's various other changes in the release notes, like support for new syntax, bug fixes and performance improvements.

Because Pants uses its own black backend this PR also reformats the world, according to the new style.

The commits are independently sensible:

1. Upgrade black itself
2. Adjust the flake8 rules due to a new conflict (https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#e701-e704)
3. Reformat the whole world

[huonw]

(see [release notes for 24.1.0](https://black.readthedocs.io/en/stable/change_log.html) for details)

NB. this doesn't link to the release notes heading directly because they're just numbered links from the top, not permalinks, e.g. https://black.readthedocs.io/en/stable/change_log.html#id18

This means a direct link will be wrong/misleading by the time we release Pants 2.23, if black does any releases in that time, since the 18th heading won't be 24.1.0 any more.

psf/black#4344

[benjyw]

This is very disruptive to users - they upgrade Pants and suddenly all their code reformats...

I guess that is just how Black is, and we should go along with it?

[huonw]

Yes, agreed. I was trying to alleviate some of the issue with the release notes call out, but we might want to do more leg work.

As you say, this is how black is, and I don't think pants should be imposing its own ideas on tools, thus getting stuck on really old versions...

Just brainstorming some ideas:

1. build infrastructure for Python tools similar to "known versions" for external tools, so Pants could have several built-in lockfiles that can be chosen between more simply than creating a whole custom resolve
2. turn Black into an external tool somehow, e.g. have a parallel repo that publishes pexes for it
3. have separate backends for each version of black, e.g. a pants.backend.python.lint.black23, pants.backend.python.lint.black24 then black25, with the current pants.backend.python.lint.black => pants.backend.python.lint.black23
4. more visible warnings/callouts about the change than just release notes
5. maybe you can think of other things?

I'm inclined to either 1 or 4, because as you say, that's just how black is... Thoughts?

[lilatomic]

I think that if we shipped both lockfiles, we could make rolling back be as simple as creating a resolve that points to the lockfile using the internal syntax (resource://), something like

[python.resolve]
black = "resource://pants/backend/python/lint/black/black23.lock"
[black]
install_from_resolve = "black"

[kaos]

It would be nice if it was less involved to override which version of black to use. i.e. provide the version and pants takes care of the locking and what-not behind the scenes. But I think that is for another issue, and here, the disruption comes from black, and I'd say we don't want to get stuck on an increasingly old version for the sake of shielding users from that. Also, there is a way out of it--so we're not imposing anything impossible.

I like @lilatomic's idea as middle ground to offer an alternative built-in lockfile. But I don't know how well that plays with the change in ICs.. or if it's enough to set those along with the lockfile in the config?

[benjyw]

Yeah, agreed. Live by the tool, die by the tool...

[huonw]

I think that if we shipped both lockfiles, we could make rolling back be as simple as creating a resolve that points to the lockfile using the internal syntax (resource://), something like

Nice idea, I'll have a play with that and see if it seems like a viable "ease the transition" path.

[huonw]

Nice, that seems to work well, running the following config (with PANTS_SOURCE=... on this branch) uses the old 23.3 version and the thus the old formatting style.

[GLOBAL]
pants_version = "2.21.0"
backend_packages = ["pants.backend.python", "pants.backend.python.lint.black"]

[python]
interpreter_constraints = ["==3.10.*"]
enable_resolves = true

[python.resolves]
black = "resource://pants.backend.python.lint.black/black-23.3.lock"

[black]
install_from_resolve = "black"

I've pushed this and rewritten the release notes to explain the upgrade path.

I'm thinking we could then deprecate this lockfile in Pants 2.24, and remove in Pants 2.26 (or something like that).

What do you think? Is it worth it?

[benjyw]

I think we can even deprecate in 2.23, and remove in 2.24. That would be more in the spirit of Black's apparent upgrade strategy of "just disrupt the world". But we don't need to deprecate the lockfile's existence, just the relying on it by default?

[huonw]

Hm, I'm not 100% sure I understand your idea. Are you suggesting this?

• In pants 2.23:
  • still use black 23 by default, but emits deprecation warnings
  • we encourage users to (explicitly) switch to black 24 somehow, potentially with a resources://.../black24.lock lockfile packaged
• In pants 2.24:
  • switch to black 24 by default
  • retain the black 23 lockfile for a user to use explicitly, if required

Or maybe something else?

[benjyw]

Hm, I'm not 100% sure I understand your idea. Are you suggesting this?

• In pants 2.23:

  • still use black 23 by default, but emits deprecation warnings
  • we encourage users to (explicitly) switch to black 24 somehow, potentially with a resources://.../black24.lock lockfile packaged

• In pants 2.24:

  • switch to black 24 by default
  • retain the black 23 lockfile for a user to use explicitly, if required

Or maybe something else?

Yes, this. Or we can get rid of the black 23 lockfile if we prefer. I'm just saying that we don't have to.

[huonw]

We've just branched for 2.23, so merging this pull request now will come out in 2.24, please move the release notes updates to docs/notes/2.24.x.md. Thank you!