Remove support for already-installed requirements

[fcrick]

I have a scenario where if I make the following requirements.txt file:

cryptography<=3.2.1,>2.9.2
azure-storage-blob==12.8.1

and pass it with -r, everything is fine.

If I add this to the script i'm running, and remove the -r flag:

__requires__ = [
    'cryptography<=3.2.1,>2.9.2',
    'azure-storage-blob>12',
]

I get an error because my site-packages has jwt 1.1.0 and cryptography 3.2.1 installed:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 3.4.7 which is incompatible.

What i'd like:

• a flag that will give me the -r behavior without having to make another separate file

What I'd love:

• a flag that will make it not matter what I have installed in site-packages at all. I just want a one line command that will run my script regardless of what might have been installed with pip on the machine in the past.

[jaraco]

My instinct is that these behaviors should be implemented by pip itself. pip-run basically just uses pip install --target $TEMP {reqs}. If pip could implement the behaviors you describe above, then pip-run would inherit those behaviors directly. I'd start by reaching out to the pip team to see if there could be an option for pip install --target that would enable the requirements.txt like behavior for requirements supplied on the command-line.

It's conceivable that pip-run could create a temporary requirements.txt file for __requires__ in a script or even for requirements supplied on a command-line, but that would be messy (extra file creation/cleanup, more difficult error reporting, clumsy invocation). I'd rather if there were a straightforward way to direct pip to enact the desired behavior.

[jaraco]

I was able to replicate the issue:

draft $ python -m venv env
draft $ env/bin/pip install -q cryptography==3.2.1 jwt==1.1.0 pip-run
WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/jaraco/draft/env/bin/python -m pip install --upgrade pip' command.
draft $ cat > script.py
__requires__ = [
    'cryptography<=3.2.1,>2.9.2',
    'azure-storage-blob>12',
]
draft $ env/bin/pip-run -- script.py
Collecting azure-storage-blob>12
  Using cached azure_storage_blob-12.9.0-py2.py3-none-any.whl (356 kB)
Collecting msrest>=0.6.21
  Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting azure-core<2.0.0,>=1.10.0
  Using cached azure_core-1.19.0-py2.py3-none-any.whl (176 kB)
Collecting cryptography>=2.1.4
  Downloading cryptography-35.0.0-cp36-abi3-macosx_10_10_x86_64.whl (2.5 MB)
     |████████████████████████████████| 2.5 MB 2.7 MB/s 
Collecting requests>=2.18.4
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting six>=1.11.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting cffi>=1.12
  Using cached cffi-1.14.6-cp310-cp310-macosx_10_9_universal2.whl
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting requests-oauthlib>=0.5.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, six, requests, pycparser, oauthlib, requests-oauthlib, isodate, cffi, msrest, cryptography, azure-core, azure-storage-blob
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 35.0.0 which is incompatible.
Successfully installed azure-core-1.19.0 azure-storage-blob-12.9.0 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 cryptography-35.0.0 idna-3.2 isodate-0.6.0 msrest-0.6.21 oauthlib-3.1.1 pycparser-2.20 requests-2.26.0 requests-oauthlib-1.3.0 six-1.16.0 urllib3-1.26.7
WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/jaraco/draft/env/bin/python -m pip install --upgrade pip' command.

I can replicate the issue without involving the script:

draft $ env/bin/pip-run 'cryptography<=3.2.1,>2.9.2' 'azure-storage-blob>12' -- -c pass
Collecting azure-storage-blob>12
  Using cached azure_storage_blob-12.9.0-py2.py3-none-any.whl (356 kB)
Collecting cryptography>=2.1.4
  Using cached cryptography-35.0.0-cp36-abi3-macosx_10_10_x86_64.whl (2.5 MB)
Collecting msrest>=0.6.21
  Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting azure-core<2.0.0,>=1.10.0
  Using cached azure_core-1.19.0-py2.py3-none-any.whl (176 kB)
Collecting six>=1.11.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting requests>=2.18.4
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting cffi>=1.12
  Using cached cffi-1.14.6-cp310-cp310-macosx_10_9_universal2.whl
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, six, requests, pycparser, oauthlib, requests-oauthlib, isodate, cffi, msrest, cryptography, azure-core, azure-storage-blob
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 35.0.0 which is incompatible.
Successfully installed azure-core-1.19.0 azure-storage-blob-12.9.0 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 cryptography-35.0.0 idna-3.2 isodate-0.6.0 msrest-0.6.21 oauthlib-3.1.1 pycparser-2.20 requests-2.26.0 requests-oauthlib-1.3.0 six-1.16.0 urllib3-1.26.7
WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/jaraco/draft/env/bin/python -m pip install --upgrade pip' command.

But if I try to replicate the behavior with pip only, the error doesn't occur:

draft $ env/bin/pip install --target out 'cryptography<=3.2.1,>2.9.2' 'azure-storage-blob>12'
Collecting cryptography<=3.2.1,>2.9.2
  Using cached cryptography-3.2.1-cp35-abi3-macosx_10_10_x86_64.whl (1.8 MB)
Collecting azure-storage-blob>12
  Using cached azure_storage_blob-12.9.0-py2.py3-none-any.whl (356 kB)
Collecting cffi!=1.11.3,>=1.8
  Using cached cffi-1.14.6-cp310-cp310-macosx_10_9_universal2.whl
Collecting six>=1.4.1
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting msrest>=0.6.21
  Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting azure-core<2.0.0,>=1.10.0
  Using cached azure_core-1.19.0-py2.py3-none-any.whl (176 kB)
Collecting requests>=2.18.4
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting requests-oauthlib>=0.5.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, six, requests, pycparser, oauthlib, requests-oauthlib, isodate, cffi, msrest, cryptography, azure-core, azure-storage-blob
Successfully installed azure-core-1.19.0 azure-storage-blob-12.9.0 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 cryptography-3.2.1 idna-3.2 isodate-0.6.0 msrest-0.6.21 oauthlib-3.1.1 pycparser-2.20 requests-2.26.0 requests-oauthlib-1.3.0 six-1.16.0 urllib3-1.26.7
WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/jaraco/draft/env/bin/python -m pip install --upgrade pip' command.

It seems there's something about the way that pip-run is invoking pip is leading to the wrong version of cryptography being installed.

[jaraco]

Aha! I figured out why it's happening, and here's how to replicate the behavior with pip alone:

draft $ rm -rf out && env/bin/pip install --target out 'azure-storage-blob>12'
Collecting azure-storage-blob>12
  Using cached azure_storage_blob-12.9.0-py2.py3-none-any.whl (356 kB)
Collecting msrest>=0.6.21
  Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting cryptography>=2.1.4
  Using cached cryptography-35.0.0-cp36-abi3-macosx_10_10_x86_64.whl (2.5 MB)
Collecting azure-core<2.0.0,>=1.10.0
  Using cached azure_core-1.19.0-py2.py3-none-any.whl (176 kB)
Collecting six>=1.11.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting requests>=2.18.4
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting cffi>=1.12
  Using cached cffi-1.14.6-cp310-cp310-macosx_10_9_universal2.whl
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, six, requests, pycparser, oauthlib, requests-oauthlib, isodate, cffi, msrest, cryptography, azure-core, azure-storage-blob
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 35.0.0 which is incompatible.
Successfully installed azure-core-1.19.0 azure-storage-blob-12.9.0 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 cryptography-35.0.0 idna-3.2 isodate-0.6.0 msrest-0.6.21 oauthlib-3.1.1 pycparser-2.20 requests-2.26.0 requests-oauthlib-1.3.0 six-1.16.0 urllib3-1.26.7

Notice that the cryptography requirement is omitted from the invocation to pip install. That's because pip-run checks for packages already satisfied as a performance optimization:

draft $ env/bin/python -q
>>> from pip_run import deps
>>> list(deps.not_installed(['cryptography<=3.2.1,>2.9.2', 'azure-storage-blob>12']))
['azure-storage-blob>12']

[jaraco]

I wish pip had the "already installed" behavior built in natively. Looks like there is some prior efforts:

• -t flag seems to cause "--upgrade-strategy only-if-needed" to be ignored pypa/pip#5178
• install --target should offer option to supplement existing packages pypa/pip#4116
• Native namespace package does not install properly when specifying a target directory (-t option) pypa/pip#10110

If pip install -t had a way to enable/disable "already installed", then pip-run could remove deps.not_installed and pip-run would work as intended.

Would you consider working with the pip team to provide a solution?

[jaraco]

In #104, we're considering completely rethinking the way pip-run is installed and invoked, putting it in its own environment and isolating its own dependencies from the target environment. This approach would eliminate the purpose of honoring "already-installed" dependencies.

Maybe the best thing to do is deprecate support for already-installed dependencies and just always install the whole dependency tree.

The original motivation for supporting already-installed requirements was inspired by how easy_install (setuptools) would only download and build eggs if they weren't installed and weren't already satisfied by packages in ./.eggs. That use case is no longer in the remit of pip-run.

A secondary motivation for not installing already installed packages was for better performance in an environment with a lot of packages that are already satisfied. It does seem, however, as if that use-case is rare and the behavior can even be undesirable (causing surprise when pip-run packaging) resolves very quickly and possibly to a stale version because packaging is already a dependency of pip-run, so is always satisfied. The more of #85 that can be implemented will alleviate the performance costs of always computing from scratch (and can be avoided today with use of PIP_RUN_RETENTION_STRATEGY=persist for some cases).

Since pip is uninterested in supporting the needed feature, let's just deprecate the feature in pip-run.

[jaraco]

As I think about it more, I'm not sure a deprecation period is useful. My guess is that not a lot of users are reliant on that behavior, and even if they are, the difference will primarily be a lower startup time. I would simply emit a deprecation warning (or comparable user warning) when the code filters out existing deps, but there would be nothing a user could reasonably do to avoid those messages without being incompatible with the future behavior.

I'm considering instead just to remove the behavior in a breaking release. Users can pin to the old version if they need that old behavior.

We can yank and add a deprecation notice if it turns out to cause more than a modicum of disruption.

[jaraco]

Released as pip-run 13.