Add [tools] table to Pipfile format

[ncoghlan]

The briefcase project is currently evaluating their options for moving away from setup.py as their input format for specifying their input dependencies.

The two most obvious potential candidates for that task would be Pipfile (pipenv-style) and pyproject.toml (poetry-style).

I'd like to recommend Pipfile for the purpose, but there's a technical problem with that: there's no option to include a "[tools.briefcase]" section with the additional settings that briefcase needs.

Now, my reading of https://github.com/pypa/pipenv/blob/master/pipenv/patched/pipfile/api.py is that pipenv is currently just ignoring all unknown sections, so allowing this would just be a documentation change that doesn't require a pipenv update.

(@freakboy3742 @techalchemy @uranusjr)

[pradyunsg]

What are the reasons to not use pyproject.toml here?

[freakboy3742]

@pradyunsg So, there are two possible modes of usage here:

The first is an application that is purely for "app-style" distribution. That is, the project won't ever be uploaded to PyPI - it will only ever be turned into an app bundle and distributed through an app store or download link. An example would be a game - there's no point being in PyPI, because end users just want to get and play the game.

The second is an application that is a hybrid - something that is uploaded to PyPI, and is also available as an app download. An example would a developer tool - something like a graphical debugger or an IDE. There may be value in having the tool on PyPI, because a developer may want to have a command line interface to the tool, localized to their virtual environment; but there may also be a use for developers to download the tool from an app store and install it as an OS-level resource.

My understanding from my conversation with @ncoghlan at PyCon AU 2019 was that these two modes would lend themselves to 2 different project configurations. AIUI, Nick's position was that because the first mode didn't involve any PyPI consequences, it made sense that it wouldn't have a pyproject.toml - in the same way that a website project doesn't have a pyproject.toml. The second mode involves a PyPI interaction, and thus a pyproject.toml makes sense.

This was all driven by my personal confusion over what we (i.e., BeeWare and Briefcase in particular) should be doing. Based on the current state of documentation, it's not 100% clear (to me, anyway) what the "ideal end-state" of packaging in the pyproject.toml/pipenv era actually is.

[uranusjr]

I think I am about +0 to this; pyproject.toml should be a reasonable solution, but I am still in favour of Pipfile gaining this field.

I agree with the app/library/hybrid style distinction and how an app style does not need a pyproject.toml. But there are already tools that should accommodate this non-library need (e.g. Black) that uses only pyproject.toml. From past discussions PyPA people seem to be OK with that, which leads me to think that PyPA’s position is app style projects can use pyproject.toml, even if they don’t need/want it otherwise.

That said, I don’t think this precludes Pipfile gaining the tool(s) section. It is not strictly needed, but the distinction could still be useful in situations. Hence the +0.

[pradyunsg]

I'm -1 on this (to the extent that it matters anyway).

Having a pyproject.toml isn't an indication of being an index-upload-able Python project. It's already possible to use pyproject.toml for a project that won't end up on PyPI.

Not having a setup.py or setup.cfg means that setuptools can't build the project into distributions, so you can't go around installing or uploading it with pip/twine, unless you change your build system.

---

My main issue here, is that we already have a lot of folks feeling "there's too many places to configure tools so we won't support this new thing" (I can immediately think of mypy, but that's the most visible name too here) and our current "soft" answer/recommendation is to support configuration via your tool specific file and pyproject.toml.

Doing this adds another location which makes things more complicated and I don't think the benefit we gain from doing this worth it -- end users can get away with needing an extra file in the root of the project. But they're putting configuration in a tool specific file format, instead of the standards backed one. That it's one file until they start using a tool that only supports the standards backed format. 🙃

[ncoghlan]

I currently see having pyproject.toml as absolutely being a marker that a project is intended for distribution via PyPI, and is expected to be installable via a Python installer like pip. Hence the build system section, which only makes sense if you're working on a pip-installable component.

That's part of why pipenv doesn't use pyproject.toml as its input file: pipenv is a PyPI consumption tool, not a PyPI publishing tool.

Now, the counter-proposal to that would be to say that the conceptual split should instead be that Pipfile is just a way to declare Python build and runtime dependencies when using a non-Python build system, so it's perfectly fine to have both pyproject.toml and Pipfile in the one project - that would bring the app distribution case closer to the case where pipenv is used to manage test environments for a regular PyPI component.

It also has the benefit of minimising the changes required to documentation - we'd just find somewhere to say "Just as Pipfile may be useful for managing development dependencies in a mostly pyproject.toml based project, pyproject.toml may be used to hold tool settings in a mostly Pipfile based application deployment project".

The associated specific UX recommendation to @freakboy3742 would be to support two styles of use:

• briefcase settings in pyproject.toml, dependencies from Python project in same repo
• briefcase settings in pyproject.toml, dependencies from Pipfile in same repo (probably opt-in as an explicit setting, to avoid confusion with the CI environment management use case)

[pradyunsg]

I currently see having pyproject.toml as absolutely being a marker that a project is intended for distribution via PyPI, and is expected to be installable via a Python installer like pip. Hence the build system section, which only makes sense if you're working on a pip-installable component.

Ah, so that's our core disagreement here.

FWIW, I had the same position when we had started rolling out PEP 517. However, then we explicitly decided that it's okay to not require a [build-system] in the pyproject.toml file (something I was not in favor of then and am OK with now). Thus, today, pyproject.toml can be (and is being) used as just a project-related-tooling configuration file -- without explicitly mentioning anything about how-to-build in it. As an example, Warehouse uses it like that.

---

The associated specific UX recommendation to @freakboy3742 would be to support two styles of use:

I'm perfectly cool with this recommendation. :)