installing distgen from PyPi failed with six traceback #149

phracek · 2024-09-06T08:35:27Z

On Fedora host machine, I would like to install distgen by 'pip3 install distgen --upgrade' but it failed
with

pip3 install distgen --upgrade
Collecting distgen
  Downloading distgen-1.18.tar.gz (71 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.4/71.4 kB 2.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: distro in /usr/lib/python3.12/site-packages (from distgen) (1.9.0)
Requirement already satisfied: jinja2 in /usr/lib/python3.12/site-packages (from distgen) (3.1.4)
Requirement already satisfied: six in /usr/lib/python3.12/site-packages (from distgen) (1.16.0)
Requirement already satisfied: pyyaml in /usr/lib64/python3.12/site-packages (from distgen) (6.0.1)
Requirement already satisfied: setuptools in /usr/lib/python3.12/site-packages (from distgen) (69.0.3)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/lib64/python3.12/site-packages (from jinja2->distgen) (2.1.3)
Building wheels for collected packages: distgen
  Building wheel for distgen (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for distgen (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [77 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_manpages
      generating man/dg.1
      Traceback (most recent call last):
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 421, in build_wheel
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 503, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 318, in run_setup
          exec(code, locals())
        File "<string>", line 35, in <module>
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/command/bdist_wheel.py", line 384, in run
          self.run_command("build")
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-djd_m6ov/distgen_1291d9546f0d47d5bda803cdb945501d/build_manpages/build_manpages/build_manpages.py", line 85, in run
          self.run_command(DEFAULT_CMD_NAME)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-fbnk3gqa/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-djd_m6ov/distgen_1291d9546f0d47d5bda803cdb945501d/build_manpages/build_manpages/build_manpages.py", line 74, in run
          parser = get_parser(data['import_type'], data['import_from'], data['objname'], data['objtype'])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-install-djd_m6ov/distgen_1291d9546f0d47d5bda803cdb945501d/build_manpages/build_manpages/build_manpage.py", line 43, in get_parser
          return get_parser_from_file(import_from, objname, objtype)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-install-djd_m6ov/distgen_1291d9546f0d47d5bda803cdb945501d/build_manpages/build_manpages/build_manpage.py", line 37, in get_parser_from_file
          filedict = run_path(filename)
                     ^^^^^^^^^^^^^^^^^^
        File "<frozen runpy>", line 286, in run_path
        File "<frozen runpy>", line 98, in _run_module_code
        File "<frozen runpy>", line 88, in _run_code
        File "bin/dg", line 12, in <module>
          import six
      ModuleNotFoundError: No module named 'six'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for distgen
Failed to build distgen
ERROR: Could not build wheels for distgen, which is required to install pyproject.toml-based projects```

The text was updated successfully, but these errors were encountered:

phracek · 2024-09-06T09:07:46Z

@frenzymadness Do you have any idea, what's going on? Even the system is clear from pypi package, then it failed with traceback.

Or should we replace six with something different?

frenzymadness · 2024-09-06T10:10:37Z

The problem is during building manual pages which I think @praiskup knows more about.

We should release universal wheels on PyPI so when users install distgen, they don't have to build their own.

duzda · 2024-09-06T10:30:28Z

Hello!

I believe this is due to using a legacy build system and not having a pyproject.toml specified, which in turn means, that dependencies are never getting resolved properly. This is just a wild guess, however see pypa/pip#8559

Worth giving it a shot, just add wheel as a requirement, and add pyproject.toml, sample pyproject provided:

[tool.poetry]
name = "distgen"
version = "1.18"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"
distro = "^1.9.0"
Jinja2 = "^3.1.4"
six = "^1.16.0"
PyYAML = "^6.0.2"
setuptools = "^74.1.2"
wheel = "^0.44.0"

[tool.poetry.group.dev.dependencies]
pytest = "^8.3.2"
pytest-catchlog = "^1.2.2"
pytest-cov = "^5.0.0"
coverage = "^7.6.1"
flake8 = "^7.1.1"

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

More here this is just a quick poetry version.

UPDATE: The error message is probably OR instead of AND, therefore either add wheel as a requirement, OR add pyproject.toml

SlouchyButton · 2024-09-06T13:27:02Z

After further analysis with @duzda it seems that this happens in newer versions of pip, when required packages are not installed before calling pip install (including the wheel which is currently not a part of requirements.txt).

This is very possibly due to new sandboxing of pip install. Due to how distgen handles man page generation, it needs all the dependencies available at build time. This can't happen with new pip, as the build itself is running sandboxed. So even calling pip install -r requirements.txt before running pip install doesn't resolve this. The wheel is required to allow the build to run without being sandboxed.

It tries to install this man page, regardless of whether it is installed via pip or RPM, but in case of pip it doesn't make sense. At least not anymore, since pip shouldn't be used to install system-wide packages, and mostly should be used inside virtual environments.

The only possible purpose for man page would be in case of the RPM package, but that can be dealt with when building the RPM itself, and it shouldn't be done for PyPi.

The ideal fix here would be to add pyproject.toml which afaik replaces setup.py, that means the man page creation will have to be reworked anyway. A simple quick fix would be to just stop generating man pages altogether and possibly do so only in RPM build, but separately to the distgen build itself.

Since this project is supposed to be onboarded to packit, there will be changes to RPM building process anyway and man page generation can be included.

After discussing this issue with @phracek it is not an immediate concern as it still works on GitHub workflows and mac, supposedly because of an outdated version of pip, but that is only temporary.

I will be making PR with a fix that will disable man page generation for now, and will solve this issue.
When migrating to use packit we can revisit the man page generation and include it in the RPMs.

frenzymadness · 2024-09-07T05:58:25Z

I think we are mixing a lot of stuff here together. setuptools is not a legacy build system but it has to be configured correctly. And because setuptools has had many new releases in recent weeks, I'd suspect something changed there.
Stop building manual pages that are useless for Python packages is a reasonable workaround.

praiskup · 2024-09-08T08:12:00Z

Could we stop using six?

This drops the compatibility with Python 2.7, this should be OK (EPEL 7 is EOL). Fixes: devexp-db#149

This drops the compatibility with Python 2.7 which should be OK (EPEL 7 is EOL). Fixes: devexp-db#149

frenzymadness · 2024-09-08T12:51:58Z

Could we stop using six?

If we don't have to support Python 2 anymore, yes, sure.

SlouchyButton · 2024-09-09T06:50:24Z

Could we stop using six?

If we don't have to support Python 2

I'm all for removing support for python 2, which is potentially not needed anymore. There is no reason to hold onto it.

At the same time I am not particularly happy calling that a solution to this problem, because the problem seems to be in dependencies not being available during parts of the build and that is probably due to bad configuration of setuptools as @frenzymadness said, possibly due to an update either in pip, setuptools or something else.

Removing the dependency that isn't available despite it should be will very probably resolve the error message, but I feel like it doesn't really fix the issue.

praiskup · 2024-09-09T07:14:28Z

Yeah, I'm not personally interested in the build-system cleanup work now (even though it might be worth it, thank you for taking a look). I want to make it installable (fixing the original report).

As for the switch to Poetry, my (not important, really) vote is -1. Setuptools always seemed to be more powerful to me. Yet I'm +1 for moving the project from setup.py to pyproject.toml, if anyone can afford to spend some time on it!

SlouchyButton · 2024-09-09T10:37:11Z

As for the switch to Poetry, my (not important, really) vote is -1.

I think it was only supposed to be an example of pyproject.toml file, not a proposal to switch to poetry, rather a proposal to switch to pyproject.toml instead of setup.py.

I have this in my backlog and would like to address this issue together with the packit migration, just not sure when I will be able to do so.

Also, I agree that the package should be installable asap, and is a more pressing matter right now. Is the solution of removing six working in your repo? If so, I could do a quick PR review if you can make a PR. If not, I can just remove the manpage build for now and address it later as I proposed in comment above (#149 (comment))

duzda · 2024-09-09T11:22:43Z

Yes, I didn't mean to propose poetry, just showcase how sample pyproject.toml looks like and give a few references. I just used poetry for generating. I think moving to pyproject.toml would eventually be nice to have, but definitely not necessary (mostly because I wrongly interpreted that you have to either install wheel or use pyproject.toml).

Other than that, this is imo the correct solution.

Stop building manual pages that are useless for Python packages is a reasonable workaround.

SlouchyButton · 2024-09-10T08:54:29Z

Reopening because merging #150 didn't fix the issue, package is still not installable via pip.

We just replaced it failing on dependency six with it failing on importing dependency jinja2

praiskup · 2024-09-13T09:16:20Z

We just replaced it failing on dependency six with it failing on importing dependency jinja2

Yeah, we are probably unable to specify build deps. This code movement would probably fix the installation problem itself, don't you want to submit that as a separate pull-request?

SlouchyButton · 2024-09-13T09:43:32Z

We just replaced it failing on dependency six with it failing on importing dependency jinja2

Yeah, we are probably unable to specify build deps. This code movement would probably fix the installation problem itself, don't you want to submit that as a separate pull-request?

I feel like all the changes are kinda linked together. That code move will remove support for using dg as an executable, and in turn that will break some parts of the copr build afaik. It would mean also changing the old build system to create that script with the entry point as in the PR.

I feel like the migration to pyproject.toml is mostly ready, so I don't know if creating that change earlier is worth it.

praiskup · 2024-09-13T10:19:52Z

I meant the move of the argument parser into a separate file; that would itself fix the overall problem (IOW no jinja2 or any other third-party is needed when generating the manual page). Also, there's /bin/argparse-manpage utility in all the distributions, it is enough to build manpage in specfile (= as others suggested, stop building manpages in pypi). Making the build system modern is a separate thing.

Fixes: devexp-db#149

praiskup · 2024-09-13T10:28:21Z

Edited: Solutin in #154, the #153 goes totally orthogonal and I'm -1 there.

Fixes: devexp-db#149

praiskup added a commit to praiskup/distgen that referenced this issue Sep 8, 2024

drop the dependency on six

dfc5e8f

This drops the compatibility with Python 2.7, this should be OK (EPEL 7 is EOL). Fixes: devexp-db#149

praiskup added a commit to praiskup/distgen that referenced this issue Sep 8, 2024

drop the dependency on six

b640e1c

This drops the compatibility with Python 2.7 which should be OK (EPEL 7 is EOL). Fixes: devexp-db#149

praiskup added a commit to praiskup/distgen that referenced this issue Sep 8, 2024

Drop the dependency on python-six

4d651bf

This drops the compatibility with Python 2.7 which should be OK (EPEL 7 is EOL). Fixes: devexp-db#149

SlouchyButton closed this as completed in 956690b Sep 10, 2024

SlouchyButton mentioned this issue Sep 10, 2024

Migrate to pyproject.toml #152

Closed

SlouchyButton reopened this Sep 10, 2024

SlouchyButton mentioned this issue Sep 12, 2024

Rework of build system to use more modern standards #153

Merged

praiskup pushed a commit to praiskup/distgen that referenced this issue Sep 13, 2024

Fix support for automatic manpage generation

c4faaf1

Fixes: devexp-db#149

praiskup mentioned this issue Sep 13, 2024

Fix support for automatic manpage generation #154

Merged

praiskup pushed a commit to praiskup/distgen that referenced this issue Sep 13, 2024

Fix support for automatic manpage generation

5590110

Fixes: devexp-db#149

praiskup pushed a commit to praiskup/distgen that referenced this issue Sep 13, 2024

Fix support for automatic manpage generation

f5d8e6e

Fixes: devexp-db#149

praiskup closed this as completed in 00c94db Sep 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

installing distgen from PyPi failed with six traceback #149

installing distgen from PyPi failed with six traceback #149

phracek commented Sep 6, 2024

phracek commented Sep 6, 2024

frenzymadness commented Sep 6, 2024

duzda commented Sep 6, 2024 •

edited

Loading

SlouchyButton commented Sep 6, 2024

frenzymadness commented Sep 7, 2024

praiskup commented Sep 8, 2024

frenzymadness commented Sep 8, 2024

SlouchyButton commented Sep 9, 2024

praiskup commented Sep 9, 2024

SlouchyButton commented Sep 9, 2024

duzda commented Sep 9, 2024

SlouchyButton commented Sep 10, 2024

praiskup commented Sep 13, 2024

SlouchyButton commented Sep 13, 2024

praiskup commented Sep 13, 2024

praiskup commented Sep 13, 2024 •

edited

Loading

installing distgen from PyPi failed with six traceback #149

installing distgen from PyPi failed with six traceback #149

Comments

phracek commented Sep 6, 2024

phracek commented Sep 6, 2024

frenzymadness commented Sep 6, 2024

duzda commented Sep 6, 2024 • edited Loading

SlouchyButton commented Sep 6, 2024

frenzymadness commented Sep 7, 2024

praiskup commented Sep 8, 2024

frenzymadness commented Sep 8, 2024

SlouchyButton commented Sep 9, 2024

praiskup commented Sep 9, 2024

SlouchyButton commented Sep 9, 2024

duzda commented Sep 9, 2024

SlouchyButton commented Sep 10, 2024

praiskup commented Sep 13, 2024

SlouchyButton commented Sep 13, 2024

praiskup commented Sep 13, 2024

praiskup commented Sep 13, 2024 • edited Loading

duzda commented Sep 6, 2024 •

edited

Loading

praiskup commented Sep 13, 2024 •

edited

Loading