-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PEP 440 version matching of local version identifier clause (e.g. 1.2.3+local) not implemented correctly in solver #2543
Comments
Same problem here - is there any way to disable URL filename version checking? |
Same problem. I tried to install pytorch and torchvision with local .whl files via these command:
When I install pytorch, it was successful.
How can I solve this? |
You can't, the resolver or the package's dependencies need to change and so far both parties aren't acting on this. I have fallen back to managing all my deps except torch and vision with poetry and then installing torch and vision with pip in the virtualenv afterwards. It sucks but it's the only reliable way right now. |
Thanks for your reply. Hope a perfect solution will come soon. |
Is there any update on this? It is a real problem for my projects where I rely on torch. |
I feel you! No news yet. |
I wrote an ugly workaround. In pyproject.toml, I specify torch and torchvision using torch = "1.5.1+cpu"
torchvision = "0.6.1+cpu" I added a
Here is the monster: @duty
def fix_torch_lock(ctx):
def _fix_torch_lock():
lock_file = Path("poetry.lock")
lock_lines = lock_file.read_text().splitlines(keepends=False)
in_torch_package = False
for index, line in enumerate(lock_lines):
if line == 'name = "torch"':
in_torch_package = True
elif line == 'version = "1.5.1"' and in_torch_package:
lock_lines[index] = 'version = "1.5.1+cpu"'
in_torch_package = False
elif line == 'torch = "1.5.1"':
lock_lines[index] = 'torch = "1.5.1+cpu"'
elif line.startswith(' {file = "torch-1.5.1-'):
lock_lines[index] = "\n".join([
' {file = "torch-1.5.1+cpu-cp36-cp36m-linux_x86_64.whl", hash = "md5:UPDATE-ME"},',
' {file = "torch-1.5.1+cpu-cp36-cp36m-win_amd64.whl", hash = "md5:UPDATE-ME"},',
' {file = "torch-1.5.1+cpu-cp38-cp38-linux_x86_64.whl", hash = "md5:UPDATE-ME"},',
' {file = "torch-1.5.1+cpu-cp38-cp38-win_amd64.whl", hash = "md5:UPDATE-ME"},',
])
elif line == "[[package]]" and in_torch_package:
in_torch_package = False
lock_file.write_text("\n".join(lock_lines))
ctx.run(_fix_torch_lock, title="Fixing poetry.lock for torch")
@contextlib.contextmanager
def use_torch_no_cpu():
pyproject_file = Path("pyproject.toml")
original_contents = pyproject_file.read_text()
temp_contents = re.sub(r'torch = "1\.5\.1\+cpu"', 'torch = "1.5.1"', original_contents)
try:
pyproject_file.write_text(temp_contents)
yield
finally:
pyproject_file.write_text(original_contents)
@duty(post=[fix_torch_lock], capture=False)
def lock(ctx):
with use_torch_no_cpu():
ctx.run("poetry lock") I wasn't sure about the sensitivity of the MD5 sums, so I redacted them. You can easily find them by running Note that this is very, very specific to my use case. The only version supported by this code is 1.5.1, for Python 3.6 or 3.8. I can now run |
Same problem. I have not found any legal way to solve this problem and remove poetry from project =( |
After spending a couple of hours on this issue, I found a "solution" by combining Poetry and pip just for PyTorch. You don't need to specify the wheel URLs directly and thus remain cross-platform. I'm using Poe The Poet, a nice task runner for Poetry that allows to run any arbitrary command.
You can run:
and then:
Hope it helps. |
Adding a datapoint here, I tried just manually adding the URLs for torch and torchvision in my pyproject.toml file as follows:
But then when I run
|
any news on this? still a problem with pytorch... |
@Korijn |
Take the following two statements:
The specified version identifier for the dependency is
This if statement is not matched, because the specified version identifier is |
What if a local version changes what torchvision depends on ? |
I don't see how that is relevant. |
@niteshthakur:
|
@stadlerb IMO, the version matching rules should apply to all types of dependencies and not just direct. At the same time :
|
@niteshthakur So are we on the same page that poetry might be doing something wrong here? After all, there seems to be no reason to require |
@stadlerb this definitely is a special case for poetry to handle where it should include the local specified versions also as candidate versions. |
Any updates? This is currently blocking us from rolling out Poetry to any of our projects that use torch. |
This is still not solved :/ |
I hit this hard again today. I seem unable to install torchvision 0.5.0+cu100 on Linux after Poetry installs dependencies fine on macOS, choosing 0.5.0 package specific to macOS, effectively. I'm using Poetry 1.1.10 installed with The error when running
I validated the hash for the $ wget https://download.pytorch.org/whl/cu100/torchvision-0.5.0%2Bcu100-cp37-cp37m-linux_x86_64.whl
$ sha256 torchvision-0.5.0+cu100-cp37-cp37m-linux_x86_64.whl
4943355e6e3ea33c8e9b9c475d6b9a4b9bb8e69b1991c93e8a6b80207841bf34 torchvision-0.5.0+cu100-cp37-cp37m-linux_x86_64.whl This hash matches the one that Poetry reports. I don't know where Poetry got these hash candidates, but I'll bet that it's related to this in my [tool.poetry.dependencies]
torchvision = [
{ version = "0.5.0+cu100", markers = "sys_platform == 'linux'"},
{ version = "0.5.0", markers = "sys_platform == 'darwin'"}
] My lockfile was generated on macOS… Mirror the repository with: curl https://eternalphane.github.io/pytorch-pypi/torchvision/ \
| rg -o "https(.*)whl\"" \
| sed -e 's/"//g' \
| aria2c -d . -i - -x $(nproc) (See the aside below if you're confused about this repo URL for PyTorch.) Then hash everything: sha256 * > hashes And look for the hashes in the for i in sha256:0ca9cae9ddf1784737493e201aa9411abe62a4479b2e67d1d51b4b7acf16f6eb, sha256:1a68d3d98e074d995f3d42a492cca716b0d94605a6fadddf0ce9665425968669, sha256:1af6d7b0a515d2a83fe9b6e7969b57ba94ba87a3333e7ed707324a5be1ef5f60, sha256:2bf1dc1e16c73c5810d96e4ea463e61129e890100740cd57724413a84d301e41, sha256:323500d349d8d91ce2662de41212e8eb1845c68dbf5d4f215ca1e94c7f20723b, sha256:358967343eaba74fd748a87f40ea75ca23757e947dbef9a11cd53414d707f793, sha256:35e9483858cf8a38debc647c74741605c5c12448d314aa96961082380aadf7e5, sha256:4dd05cbc497210928ae3d4d6194561985263c879c3554e9f1823a0fa43d35746, sha256:517425af7d41b64caae0f5d9e6b14eeb48d6e62d45f302b73a11a9ec5ee3b6c8, sha256:78d455a1da7d10bd38f2e2a0d2ac285e4845c9e7e28aafdf068472cc96bd156b, sha256:9e85ba17ff93a0cf6afd39b9a0ad56ca7321db4f1eb90d2034d3b0ecd79be47b, sha256:a696ec5009eb52356508eb9b23ddb977043fb82ff7b204459e4c81aca1e5affe, sha256:aa4354d339de2c5ea2633a6c94294c68bae3e42a4b099624299e2a50c9e97a85, sha256:ec7e4cd54f5ff3a889b90f24b33da1fa9fe3f78d17348965678d9503de1e4a49, sha256:fea3d431bf639c0719afff5972eb568ebe143eba447c1c8bb491c7dfb0025ed6; do echo $i; done | cut -d ':' -f 2 | sed 's/,//g' | xargs -t -I % rg % hashes Three hits:
Then I go look at torchvision = [
{file = "torchvision-0.5.0-cp27-cp27m-macosx_10_7_x86_64.whl", hash = "sha256:35e9483858cf8a38debc647c74741605c5c12448d314aa96961082380aadf7e5"},
{file = "torchvision-0.5.0-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:323500d349d8d91ce2662de41212e8eb1845c68dbf5d4f215ca1e94c7f20723b"},
{file = "torchvision-0.5.0-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:ec7e4cd54f5ff3a889b90f24b33da1fa9fe3f78d17348965678d9503de1e4a49"},
{file = "torchvision-0.5.0-cp35-cp35m-macosx_10_6_x86_64.whl", hash = "sha256:4dd05cbc497210928ae3d4d6194561985263c879c3554e9f1823a0fa43d35746"},
{file = "torchvision-0.5.0-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:9e85ba17ff93a0cf6afd39b9a0ad56ca7321db4f1eb90d2034d3b0ecd79be47b"},
{file = "torchvision-0.5.0-cp35-cp35m-win_amd64.whl", hash = "sha256:2bf1dc1e16c73c5810d96e4ea463e61129e890100740cd57724413a84d301e41"},
{file = "torchvision-0.5.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:358967343eaba74fd748a87f40ea75ca23757e947dbef9a11cd53414d707f793"},
{file = "torchvision-0.5.0-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:fea3d431bf639c0719afff5972eb568ebe143eba447c1c8bb491c7dfb0025ed6"},
{file = "torchvision-0.5.0-cp36-cp36m-win_amd64.whl", hash = "sha256:1a68d3d98e074d995f3d42a492cca716b0d94605a6fadddf0ce9665425968669"},
{file = "torchvision-0.5.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1af6d7b0a515d2a83fe9b6e7969b57ba94ba87a3333e7ed707324a5be1ef5f60"},
{file = "torchvision-0.5.0-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:a696ec5009eb52356508eb9b23ddb977043fb82ff7b204459e4c81aca1e5affe"},
{file = "torchvision-0.5.0-cp37-cp37m-win_amd64.whl", hash = "sha256:0ca9cae9ddf1784737493e201aa9411abe62a4479b2e67d1d51b4b7acf16f6eb"},
{file = "torchvision-0.5.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:517425af7d41b64caae0f5d9e6b14eeb48d6e62d45f302b73a11a9ec5ee3b6c8"},
{file = "torchvision-0.5.0-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:aa4354d339de2c5ea2633a6c94294c68bae3e42a4b099624299e2a50c9e97a85"},
{file = "torchvision-0.5.0-cp38-cp38-win_amd64.whl", hash = "sha256:78d455a1da7d10bd38f2e2a0d2ac285e4845c9e7e28aafdf068472cc96bd156b"},
] Sure enough, the lockfile only has the macOS branch of the conditional Or do I need to generate this lockfile on Linux? If I generate it on Linux, Aside on PyTorch repoThis repo https://eternalphane.github.io/pytorch-pypi/torchvision/ is a PyPI-compatible relisting of PyTorch's original downloads page, which was only usable with
|
Can you check if this issue is still present in poetry 1.2.0b1? I was able to successfully run
|
Closing this as the original seems to be resolved as per @radoering 's comment. Poetry |
I'll try to confirm as @radoering's example configuration locks on path dependencies and therefore doesn't seem representative of the original provided reproduction example. |
There is also a long post here stating this issue still exists: #4231 (comment) Not sure this is solved and can be closed. |
I have responded to that comment. Reopening this, however I suspect this might be fixed with #4729. |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (
-vvv
option).OS version and name: Windows 10, but also in a Linux docker container running Ubuntu 18.04
Poetry version: 1.0.9
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/Korijn/b88b1605221574f51ee72b75efc03dbe
Issue
In PEP 440, it says the following in the section "Version Matching":
I believe this clause is being interpreted incorrectly by Poetry. When running
poetry lock
on apyproject.toml
file (see the gist linked above) with the folowing lines, poetry incorrectly reports a conflict:As said in the first line of the debug output, the package torchvision (0.5.0+cpu) depends on torch (1.4.0). However, I've specified that I need torch (1.4.0+cpu). The PEP says that the local identifier MUST be ignored when performing the version match for the constraint coming from the torchvision (0.5.0+cpu) package. So I would say that this conflict is actually an acceptable solution, given the constraints.
Note that Pip has no problem installing this combination of packages.
This is an issue with Poetry's solver.
The text was updated successfully, but these errors were encountered: