Eventually implement support for the PEP 735 "dependency groups"

[webknjaz]

What's the problem this feature will solve?

It's just a new source of deps and an extension to the first round of the pyproject.toml runtime deps resolution.

Describe the solution you'd like

$sbj. If somebody's interested in making a PoC, it's probably fine to start drafting something already. Need to wait for the PEP acceptance to actually merge it in, though.

Alternative Solutions

... not doing this?

Additional context

Refs:

• https://peps.python.org/pep-0735/
• https://discuss.python.org/t/pep-735-dependency-groups-in-pyproject-toml/39233/237
• POC for pip: pypa/pip@main...sirosen:pip:pep-735-support

[chrysle]

What are you planning to have for dependency group support? An option to collect only such dependencies, or alongside with normal and build deps? And the possibility to collect only specified dependency groups?

[webknjaz]

Same as #1960 attempted, just in the proper place. Semantically, dependency groups don't imply including runtime dependencies so they should definitely be collected and output separately. However, we shouldn't actively prevent combining them with other deps (runtime, or runtime+extra, or other dependency groups; though the latter seems to be covered by the PEP itself).

[chrysle]

However, we shouldn't actively prevent combining them with other deps

IIUC that would mean actually creating an option like --include-dependency-groups/--all-dependency-groups, next to --only-dependency-groups?

[webknjaz]

Aren't the past two the same?

[chrysle]

Aren't the past two the same?

Yes, this was unintended. Is the rest of my assumption valid?

[webknjaz]

I think so. It of course could be iterated on during a PoC review..

[webknjaz]

UPD: said PEP is accepted as of two weeks ago. So we should implement support for it.

[davidism]

Is there an idea for how this will work? I'd expect to be able to move each .in template into a separate dependency group instead. Then pip-compile pyproject.toml --group tests would read the tests group from pyproject.toml and output to tests.txt (or to -o {name}). Does that sound reasonable?

Currently I have different environments as separate templates: requirements/test.in, requirements/typing.in, etc. Now I'd have tests and typing dependency groups instead but otherwise get the same behavior.

[webknjaz]

Is there an idea for how this will work? I'd expect to be able to move each .in template into a separate dependency group instead.

Yeah, that's kinda the use-case this PEP solves.

Then pip-compile pyproject.toml --group tests would read the tests group from pyproject.toml and output to tests.txt (or to -o {name}). Does that sound reasonable?

This needs to be defined, I think. While working on support for the distribution package build dependencies (#1681 / https://pip-tools.readthedocs.io/en/stable/#maximizing-reproducibility), we realized that there would be cases where one would want to extract those dependencies in isolation and cases of getting combined deps (--build-deps-for=wheel / --all-build-deps / --only-build-deps ).

There's a precedent that doing pip-compile pyproject.toml requirements.in works and sources the deps from several places. That can also be combined with --extras, IIRC.

So to proceed meaningfully, we need to define the semantics and UX for all of these combinations. Do we need --[no-]runtime or some other ways to distinguish what the users would like to encode in their constraint file generation requests? Additionally, I would like to keep #2124 in mind as well, while answering these questions.

Currently I have different environments as separate templates: requirements/test.in, requirements/typing.in, etc. Now I'd have tests and typing dependency groups instead but otherwise get the same behavior.

Simlarly, I have even more envs defined, mostly one per tox target with multiple constraint files (aka DIY lock files) for each, since combinations of different platforms/runtimes can count as separate environments (fun stuff: https://github.com/ansible/awx-plugins/tree/c7fc0a1/dependencies/lock-files#readme).