pip-compile --extra doesn't take into account already pinned versions

[ianliu]

I'm trying to move away from *.in files and manage everything from pyproject.toml, but I'm facing a problem where pip-compile --extra doesn't take into account the already pinned versions from pip-compile (without --extra).

Environment Versions

1. OS Type: Arch Linux
2. Python version: 3.11.3
3. pip version: pip 23.1.2 from /usr/lib/python3.11/site-packages/pip (python 3.11)
4. pip-tools version: 6.13.0

Steps to replicate

1. Create the following pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "foobar"
version = "1"
dependencies = ["django==4.1.7"]

[project.optional-dependencies]
dev = ["pytest"]

3. Execute pip-compile --no-header --no-annotate --resolver backtracking to obtain:

asgiref==3.7.2
django==4.1.7
sqlparse==0.4.4

5. Remove the restriction from the django dependency from the pyproject.toml, and verify that pip-compile doesn't update any package.
6. Execute pip-compile --no-header --no-annotate --resolver backtracking --extra dev -o dev-requirements.txt, and see that the django dependency has been updated in the dev-requirements.txt:

asgiref==3.7.2
django==4.2.2
iniconfig==2.0.0
packaging==23.1
pluggy==1.2.0
pytest==7.4.0
sqlparse==0.4.4

I also tried to pass --pip-args "-c requirements.txt", but nothing changed.

Expected result

Actually, I would expect a way to pin only the dev dependencies, since the main dependencies were already pinned. Some workflow similar to the Workflow for layered requirements but with pyproject.toml instead of *.in files.

[AndydeCleyre]

Is it correct that you're pinning versions in requirements.txt, then compiling dev-requirements.txt, and want the pinned items from the former to inform the latter?

Currently pip-compile only reads from the same output file being generated for the "preferred constraints" behavior.

If I understand right, then at least a workaround would be to cp requirements.txt dev-requirements.txt before compiling dev-requirements.txt.

If you were using dev-requirements.in you could include -c requirements.txt, but using only pyproject.toml as input I really don't know how constraints support is (either in standards or individual backends).

[AndydeCleyre]

Related: #1364

[ianliu]

Is it correct that you're pinning versions in requirements.txt, then compiling dev-requirements.txt, and want the pinned items from the former to inform the latter?

Yes. Essentially, I was trying to achieve the same "Layered workflow" available with *.in files, but using the pyproject.toml. I also don't understand why --pip-args "-c requirements.txt" doesn't work.

[atugushev]

I also don't understand why --pip-args "-c requirements.txt" doesn't work.

Digging into the issue I found that pip parses requirements from -c requirements using get_requirements() method. pip-tools doesn't use this method. It loads requirements in its own way which is why --pip-args "-c requirements.txt" doesn't work properly.

I suppose this can be fixed by calling get_requirements() and extending the existing pins after parsing source and lock files.

[atugushev]

@ianliu could you please try this PR (#1936)? I've added support for pip-compile -c requirements.txt, which I believe should resolve the issue.