Recover "python-libtorrent" package on pypi.org

[AllSeeingEyeTolledEweSew]

It would be nice to write python projects that depend on libtorrent. Today that's difficult, partly because libtorrent doesn't exist on pypi.org in any meaningful form. #5325 and #4218 are moving to help with this.

Historically, libtorrent's setup.py has historically used the name python-libtorrent. This name (not libtorrent) would be used to declare a dependency on libtorrent; and python tools (pip, etc) would search pypi for this name.

Someone published unofficial builds of libtorrent on pypi under this name, at https://pypi.org/project/python-libtorrent .

Also, linux distros have packaged libtorrent with python packaging metadata that declares the python-libtorrent name. See alpine and ubuntu.

This means that there may be python projects that depend on python-libtorrent and do work in some environments. So changing this name is not ideal.

(for what it's worth, I don't believe deluge has ever depended on python-libtorrent. It just includes a vendored copy of libtorrent and a custom build system)

It would be nice if arvid could take over the project from pypi, so we can publish wheels there. There's a process for doing this: https://www.python.org/dev/peps/pep-0541/#how-to-request-a-name-transfer

[filips123]

In which cases is python-libtorrent as a package metadata actually used? I think this is only used for installing from PyPI (which doesn't really matter because there are no official releases on PyPI) and checking which packages are installed (using pip list/pip freeze or maybe some runtime checks but I doubt anyone is using this because they are using custom build system in first place).

Also, it seems that Ubuntu, Debian and Alpine are currently still using 1.x version of libtorrent, so this change might fall into "big 2.0 update" (although it is few months late).

I personally think just libtorrent would be better because it makes sense that library published to PyPI is probably a Python library.

[AllSeeingEyeTolledEweSew]

In which cases is python-libtorrent as a package metadata actually used?

I don't have evidence anything depends on python-libtorrent, but it's also hard to disprove. pypi.org doesn't seem to have a way to browse "packages that depend on this package", and there could always be a PPA that depends on the debian/ubuntu packages.

In any case, hopefully it's a small clerical task to recover ownership of the pypi project. If it's easy, we can just use it; if it's too difficult, we'll know we have to switch project names.

[AllSeeingEyeTolledEweSew]

@arvidn any update on this? I was hoping it's not too much work on your end.

We've made a lot of progress on the python build system. We could consider manually building and uploading some wheels now, ahead of having them auto-built in CI.

[arvidn]

wouldn't it be easier to pick a new name? like "libtorrent"? I can add a pypi API token to github, to make it available to actions

[AllSeeingEyeTolledEweSew]

You should probably register "libtorrent" right now anyway, since it's available.

Even if we want to switch to "libtorrent", I think it's wise to try to recover "python-libtorrent", to prevent confusion and/or malicious dependency attacks. (Similarly if we recover "python-libtorrent" and want to stick with that, it's good to also own "libtorrent")

[arvidn]

as far as I can tell, the only way to claim a name is to upload a package.. is there an easier way?

[AllSeeingEyeTolledEweSew]

Interesting, I can't see a way either. I guess they've "streamlined" the process

You could just change the name to "libtorrent" in setup.py in a local checkout, then

$ python setup.py bdist_wheel
$ twine upload dist/*

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[AllSeeingEyeTolledEweSew]

unstale

Any word on this? Per the PEP link, it should just be creating a new github issue.

If there are blockers or you just don't want to do it, we should probably just close this issue and say we're going to change the name to just "libtorrent".

[arvidn]

sorry, I spent a little bit of time looking into this, but. Is libtorrent-rasterbar a more established name for the python package?

I'm also quite unfamiliar with how to make CI actually upload builds, and how to make some builds be considered stable releases.

[AllSeeingEyeTolledEweSew]

sorry, I spent a little bit of time looking into this, but. Is libtorrent-rasterbar a more established name for the python package?

It's most common for the distribution name (the pypi name) to match the module name, hence libtorrent. This does help avoid collisions between installed files.

This isn't a strict rule, it could be libtorrent-rasterbar if you want. I think the only upside is branding.

I'm also quite unfamiliar with how to make CI actually upload builds, and how to make some builds be considered stable releases.

I'll put some thoughts in #6188

[cas--]

I'm also quite unfamiliar with how to make CI actually upload builds, and how to make some builds be considered stable releases.

It would consider any tagged releases as stable?

I would pick libtorrent over libtorrent-rasterbar since it's cornered the Python market ;)

Always seems redundant to me to have python- in a package name on pypi. And that prefix likely came from packaging on debian. I have actively avoided registering libtorrent until we could get a reliable build system so used deluge-libtorrent for my windows builds.

It would be very helpful for all if we can get manylinux and windows wheels on pypi, thanks @AllSeeingEyeTolledEweSew for pushing this forward!

[AllSeeingEyeTolledEweSew]

pypi/support#1299

[cas--]

Just to note here that after testing the new wheels built by @AllSeeingEyeTolledEweSew in #6188 I have changed my mind about using the libtorrent pypi repo. I failed to remember that the python package is named in setup.py as python-libtorrent which will be the package name that is version controlled by package managers e.g. pip.

To prevent problems with a libtorrent package and python-libtorrent package both providing a libtorrent module and Python import manager guessing wrong, we should maintain the python-libtorrent naming convention for the package.

[AllSeeingEyeTolledEweSew]

the python package is named in setup.py as python-libtorrent which will be the package name that is version controlled by package managers e.g. pip.

I thought about this too, but I can't think of any existing scenarios that will break if we change.

Any downstream package with a dependency on python-libtorrent is definitely broken today, since https://pypi.org/project/python-libtorrent/ actually contains no files right now.

As far as I know, pip/pypi is the only package system that respects the name in setup.py.

There are existing packages for libtorrent's python bindings in distro package managers like debian and alpine, but they aren't affected by the setup.py name. I can't find any packages in conda, chocolatey or vcpkg for libtorrent python bindings. Not sure if there are other package managers to check.

To prevent problems with a libtorrent package and python-libtorrent package both providing a libtorrent module and Python import manager guessing wrong, we should maintain the python-libtorrent naming convention for the package.

We can put a !python-libtorrent dependency in our releases on libtorrent, which should prevent this.

All that said, the only upside to changing the name to "libtorrent" is aesthetics, assuming we're able to recover https://pypi.org/project/python-libtorrent/. But I actually think this is a perfectly good reason to change. I expect it'll lead to more users due to better clarity, and more users is better.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[arvidn]

@AllSeeingEyeTolledEweSew what's left now to make a pypi package?

[AllSeeingEyeTolledEweSew]

Should be:

• go to project > actions > cibuildwheel
• click "Run workflow"
• Pick branch (probably one run with RC_1_2 and one with RC_2_0)
• Upload to pypi, either
  • ...from within the workflow:
    • create API tokens on both pypi and test-pypi, with write access to the libtorrent project
    • add the tokens as github secrets PYPI_API_TOKEN and TEST_PYPI_API_TOKEN
    • run the workflow with PUBLISH_TEST and/or PUBLISH inputs, to enable the publishing step
    • (note the workflow should cache build artifacts, so you can re-run this step quickly if upload fails)
  • ... or manually:
    • download the workflow artifacts
    • upload the artifacts using twine upload

[arvidn]

I have ownership over the python-libtorrent project. When searching, I can't find a project called just libtorrent, and I can't find a way to add a project with that name.

[cas--]

My understanding is that python-libtorrent is to be used to retain continuity, did the PR rename the package in setup.py?

[arvidn]

I would have expected to find the pypi project name that the build will be uploaded to could be found in either .github/workflows/cibuildwheel.yml or in pyproject.toml.. But I don't see "libtorrent" in either of those.

[cas--]

The metadata in setup.py sets the package name and the pypi project

libtorrent/bindings/python/setup.py

Line 450 in 53f90cf

name="libtorrent",

I honestly don't think changing the package name to just libtorrent is a good idea and this should be reverted

I know I previously argued about changing to just libtorrent but that was before I realised how the package name is used in linux distros thus could have a packaging problem since renaming would make a separate package unaware of the original

[arvidn]

I see. I also don't see the python-libtorrent project on test.pypi.org.. so I don't see how I can create an access token for TEST_PYPI_API_TOKEN

[cas--]

Ah will need to create a separate testpypi account and push the package to take ownership

[arvidn]

oh, you push the package first and then claim ownership?

[arvidn]

I start the PUBLISH_TEST workflow (after adding an API token for my test.pypi.org account. At least two jobs completed green, but I still don't see any packages on test.pypi.org

[cas--]

Did you change the package name in setup.py from libtorrent back to python-libtorrent?

[AllSeeingEyeTolledEweSew]

I start the PUBLISH_TEST workflow (after adding an API token for my test.pypi.org account. At least two jobs completed green, but I still don't see any packages on test.pypi.org

publishing happens at the end, once all builds are done. My reasoning was that if one build fails, maybe we want to fix it before publishing anything.

The emulated architectures (aarch64, ppc64le and s390x) take a few hours to build, and they already benefit from ccache, such that libtorrent is only built once per arch. This is why I left out ppc64le and s390x.

[AllSeeingEyeTolledEweSew]

@arvidn I had previously uploaded artifacts to https://test.pypi.org/project/libtorrent/ for testing, so I'm the owner of the project. I had invited you as owner (presumably it sent you an email, the invites are per-username) but it looks like the invitation expired. I re-sent it.

If you create an updated API token and update the github secret and re-run the workflow, it should pick up the cached builds and hopefully work

[AllSeeingEyeTolledEweSew]

Success! https://test.pypi.org/project/libtorrent/1.2.15/

[AllSeeingEyeTolledEweSew]

@cas--

I know I previously argued about changing to just libtorrent but that was before I realised how the package name is used in linux distros thus could have a packaging problem since renaming would make a separate package unaware of the original

I racked my brain but couldn't come up with a scenario where this was necessarily the case. Presumably e.g. the alpine package py3-libtorrent-rasterbar will continue existing. The distribution name in libtorrent-*.dist-info/ will change between 1.2.14 and 1.2.15, but I can't think of anything that could meaningfully depend on that.

The good news is that if I'm wrong, the workflow makes it pretty easy to rewrite history and fudge releases of python-libtorrent. We just create a branch from v1.2.15, change the name back to python-libtorrent in the branch, and release to pypi from there.

[arvidn]

@AllSeeingEyeTolledEweSew could invite me to the libtorrent project on pypi as well? then I'll trigger a real build

[AllSeeingEyeTolledEweSew]

done

[AllSeeingEyeTolledEweSew]

🎉 https://pypi.org/project/libtorrent/1.2.15/

[cas--]

It is important because once the package is installed in the system it is named python-libtorrent but now there is a potential conflicting package called libtorrent and Python will find both when trying import. Instead of different versions of the python-libtorrent package.

[AllSeeingEyeTolledEweSew]

Can you elaborate? The extension is/always was named libtorrent

[cas--]

The other problem is that source code package name has just changed under linux packaging from python-libtorrent to libtorrent and may break automated scripts

[cas--]

Can you elaborate? The extension is/always was named libtorrent

No it was named python-libtorrent until you changed in a few months ago: 8ea0e60

[cas--]

I feel that there are too many unintended consequences that we may or may not have considered to change the name of the bindings package for aesthetic reasons

Remember this is not just about PyPi this is about the python binding package name that is defined in the source code and how that is installed into the system and how the Python package manager handles it. Having two different package names for this library is not a good idea...

An example scenario is that once python-libtorrent is installed pip can remove or warn that an existing package is installed if you install python-libtorrent from pypi. With pip installing pypi libtorrent package that is it not possible to detect so now there are multiple packages installed providing the same library

[AllSeeingEyeTolledEweSew]

I have counterarguments, but I can't think of a way to prove either of us right, and I'm much more interested in closing this ticket than debating.

@arvidn would it be very annoying to change the name in setup.py back to python-libtorrent, and re-run the workflow?

One caveat is we never recovered https://test.pypi.org/project/python-libtorrent/ . I opened pypi/support#1513 for that just now. Arvid won't be able to upload to test-pypi until that's resolved. I don't think that's a reason to delay uploading to production pypi though.

[arvidn]

is there no way to make python-libtorrent an alias to libtorrent?

Does this name also affect what you say after import in client code?

[AllSeeingEyeTolledEweSew]

is there no way to make python-libtorrent an alias to libtorrent?

There is, but support is poor, and I think @cas-- 's concerns are about distro packagers and other non-pypi environments.

Does this name also affect what you say after import in client code?

No. 8ea0e60 changed the distribution name (the pip install name) but not the extension name (the import name).

The biggest advantage of libtorrent over python-libtorrent I can think of is just making it easier to find and reducing user friction.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.