Consider using a package manager

[danieleades]

I propose migrating Sphinx from its current build system (Flit) to a more comprehensive package manager, with a preference for Rye.

Advantages

• Simplified dependency management: Rye provides a unified interface for managing both production and development dependencies.
• Reproducible builds: Rye ensures consistent environments across different development machines and CI/CD pipelines.
• Built-in virtual environment management: Rye automatically creates and manages virtual environments, reducing setup complexity for contributors.
• Improved security: Rye includes built-in security features like checksums for dependencies.
• Modern Python packaging standards: Rye fully supports PEP 517/518 and uses pyproject.toml as the primary configuration file.
• Lockfile support: Rye generates lockfiles, ensuring exact dependency versions are used across all environments.
• Easy publishing: Rye simplifies the process of building and publishing packages to PyPI.
• Cross-platform compatibility: Rye works consistently across Windows, macOS, and Linux.
• Active development: Rye is actively maintained and developed, ensuring long-term support and improvements.
• Extremely fast

Potential Disadvantages

• Learning curve: Contributors will need to learn a new tool and workflow.
• Migration effort: Significant changes to our build configuration and potentially our CI/CD pipelines will be required.
• Community adoption: Rye is relatively new compared to some alternatives, which might affect community support.
• Compatibility concerns: We'll need to ensure all our current build processes and scripts are compatible with Rye.

Alternatives

Poetry: A popular, mature package manager with similar features to Rye. Very high adoption, but slow, and uses non-standard dependency specification
PDM: Another modern package manager with a growing community. Adoption on the low side, few (but very active) maintainers
Flit (current system): Lightweight and simple, good for single-module projects, but lacks some modern features such as multiple packages, comprehensive dependency management, lockfile support, venv management, toolchain management, etc.

Implementation Plan

1. Create a separate branch for the migration.
2. Update pyproject.toml to be compatible with Rye.
3. Generate requirements.txt and requirements-dev.txt files.
4. Adjust CI/CD pipelines to use Rye.
5. Update contribution guidelines and documentation.

[electric-coder]

Extremly fast

I'm skeptical of Rust tools for one simple reason, checking wikipedia:

On February 8, 2021, the formation of the Rust Foundation was announced by its five founding companies (AWS, Huawei, Google, Microsoft, and Mozilla).

None of those companies strike me as being open-source oriented, so if there's a pure Python solution like Poetry I'll rather stay inside the ecosystem.

When Ruff showed up I thought: "ok, so the future is Rust becoming the language of choice because being compiled it can't be beat in terms of performance, but... Will that imply a future tooling dependency of Python on Rust? Because of performance?!" If that'll be the case, we're seeing the early signs here in this post. Because if we follow this proposal, it'll be the second Sphinx dependency on Rust tooling.

[danieleades]

this is a bit of a weird take. I'll try to address your concerns in turn

Extremly fast

I'm skeptical of Rust tools for one simple reason, checking wikipedia:

On February 8, 2021, the formation of the Rust Foundation was announced by its five founding companies (AWS, Huawei, Google, Microsoft, and Mozilla).

None of those companies strike me as being open-source oriented, so if there's a pure Python solution like Poetry I'll rather stay inside the ecosystem.

I'm not sure what your specific concern is here- that one of these companies will supplant the others and do evil? that they will form a cabal which has some negative impact on Sphinx later?

Rust is community maintained, and receives support from some big tech companies. This is precisely the same model that Python uses. In fact most of the companies you've listed are also members of the Python Software Foundation. Guido van Rossum is literally employed by microsoft.

When Ruff showed up I thought: "ok, so the future is Rust becoming the language of choice because being compiled it can't be beat in terms of performance, but... Will that imply a future tooling dependency of Python on Rust? Because of performance?!" If that'll be the case, we're seeing the early signs here in this post. Because if we follow this proposal, it'll be the second Sphinx dependency on Rust tooling.

I think you're drawing a very long bow here. Introducing Rust-based tooling is absolutely orthogonal to the question of whether Python itself will have a dependency on Rust. This is not about introducing Rust to the sphinx codebase, this is about using the best tool for the job. The fact that two suggested tools have been based on Rust says more about Rust being a good language for creating such tools than it does about there being some kind of a rust conspiracy. I'm really struggling to understand your specific concern here, could you explain?

Some more points that might help address your concerns:

Sphinx is highly unlikely to be python all the way down anyway. A great many python libraries using bindings to C, often for performance reasons. Increasingly, python libraries use Rust bindings for the same reason, but also because Rust provides both security and ergonomics benefits when compares to C. Rust bindings in Python is a natural evolution of the existing status quo. Some python libraries you may know that leverage Rust bindings:

• cryptography
• PyArrow
• Numpy
• Pydantic

Regardless, this proposal is not about introducing Rust bindings. It's about leveraging the best available tooling to manage the dev environment.

[electric-coder]

It's about leveraging the best available tooling to manage the dev environment.

And I'm not the slightest bit sold on that idea. Poetry has 5k closed issues wheres Rye has 334 that's an order of magnitude multiplied by two. Poetry has 1.3k Qs on SO but I wasn't able to find a Rye tag (I suppose that says something about adoption)...

Which takes me to my second concern:

Potential Disadvantages

Learning curve:
Migration effort:
Community adoption:

This falls just short of becoming "locked into" Rye. An investment of time, attention and work. If a change is warranted wouldn't Poetry be a better choice because it's mature (or more widely used, thus aquired expertise being more likely to be reusable)? What if Rye doesn't take off nor mature well (some nice libraries out there were discontinued after much more effort had been put into them). And if we were to compare the pros of Rye vs Poetry, what would be the difference of one vs the other (that's a more relevant question)?

I'm really struggling to understand your specific concern here, could you explain?

I'm just skeptical, and skepticism is a healthy virtue when shopping for SW solutions. So for the sake of discussion, here are some more of my reservations. (As usual, it's a proponents onus to produce the necessary apologetic to any questions arising.)

---

BTW, Python seemed all nice and dandy when I first started out with it. That was until I started running into frequent mypy bugs and the like. So that's a more serious concern: Rye has a open/closed issue ratio of 243/334 so that would roughly equate to a 72% chance of running into a bug every time we're happily trying to get something done. That prospect alone looks gloomy, to not say terrifying, to me. (Hey I'm done developing for the day! Oh no!! Another inexplicable mypy bug, so that's another 4-5 hours extra on a single line...)

[danieleades]

@electric-coder if you re-read the body of this issue you'll see that i'm advocating for using a package manager, with a preference for Rye. I also list Poetry as one of the alternatives in my proposal. I use Poetry extensively and am a regular contributor to it, so that option is absolutely on the table also. Poetry has some distinct disadvantages, some of which i've listed above.

[chrisjsewell]

Yeh the take on Rust above is indeed, let's say, misguided; Rust is only the second language to be accepted into the Linux kernel (https://docs.kernel.org/rust/) it's pretty damn good.

But anyway, @danieleades as to your suggestion or Rye "ensuring long-term support ", well no that's not technically true 😅 Rye will eventually be deprecated for uv: astral-sh/rye#1164 (comment)
Rye is pretty good, I've been using it for a mono-repo for which at present it is really the only game in town, but still it does have some limitations,
things like lock-files are not cross-python version (they just use pip-tools).

I think in a year or so uv will be the thing everyone starts to migrate to for package management, and maybe sphinx can.
But in the meantime, I don't think there is any strong case to change the current setup.

Note, you can already use https://github.com/tox-dev/tox-uv for sphinx development (this is what I do)

(As for poetry, the horrible non-PEP compliant metadata is a deal-breaker for me)

[danieleades]

lock-files are not cross-python version (they just use pip-tools).

Lockfiles are newly cross-platform (opt in for now)!

See https://rye.astral.sh/guide/pyproject/#toolryeuniversal

Totally agree on rye having planned obsolescence, but I heartily believe there'll be a solid off-ramp from there onto whatever UV ends up delivering later

[chrisjsewell]

But still it's too early in the rye/uv life-cycle to move such a large, established project onto it, lets wait for them to solidify a bit, which I'm sure they will do

[chrisjsewell]

as predicted @danieleades 😄 https://astral.sh/blog/uv-unified-python-packaging, astral-sh/rye#1342
(but still needs a little time to bake)

[electric-coder]

@chrisjsewell and I'm still underwhelmed :| if they rewrote mypy and fixed the 2.5k bugs while at it I would be, indeed, impressed :)

[danieleades]

@chrisjsewell and I'm still underwhelmed :| if they rewrote mypy and fixed the 2.5k bugs while at it I would be, indeed, impressed :)

Ha that's actually planned, kind of.

"Multi-file analysis" is in the roadmap, which would in principle allow mypy-like analysis

[danieleades]

i created #12817 if anyone wants to clone that and have a hack around.

Don't think migrating would be very difficult (this branch was about 10mins effort).

The speed up in CI is immediately obvious- package resolution is dramatically faster. Locally it's a much better experience too.

Being able to test different python version just by adding a --python {VERSION} argument to commands is very nice.

@electric-coder i dare you to try and not like it once you've used it ;)

[chrisjsewell]

Being able to test different python version just by adding a --python {VERSION} argument to commands is very nice

I would note, I already use tox-uv for developing this repo, so this is already easily possible, and you can run against multiple python version in one go, which uv does not yet have

[electric-coder]

@electric-coder i dare you to try and not like it once you've used it ;)

I'm sure I will like it but right now I'm way too busy with my own projects... The description looks appetizing :) wish I could spare some time :(

Since I've been complaining, I'll continue by saying that all things about the Python Packaging Authority used to be horrible :P up until recently.