PyPi Standalone

[Gadgetoid]

These changes build a PY_LGPIO library that can be installed without any dependence on system libraries.

It does this by building the entire lg library into the Python C extension, and additionally lets setup.py handle the swig step (compiling the interface file into a C wrapper) so it does not need to be done manually.

For this to be optimal as a source distribution, we'd probably want to include lgpio_wrap.c in the distribution - via MANIFEST.in - and include some logic to only add lgpio.i as a source file if lpgio_wrap.c is missing. This would remove the dependency upon swig for users.

We should probably also convert this to some more modern packaging standard, though that is a secondary concern to just getting what we have working.

Edit: notable quirk for supplying a source distribution is that it's difficult to include the parent directory in the Python package. It would make more sense if the Python library files were another git repository which included lg as a submodule. That would also let packaging be a little more agile without spamming the base repo with Python-specific commits.

[bennuttall]

This is missing some sources - I can't build a wheel it from sdist:

$ python setup.py sdist
-> dist/lgpio-0.2.2.0.tar.gz
$ pip wheel dist/lgpio-0.2.2.0.tar.gz

[Gadgetoid]

This build should have the sources you need: https://github.com/Gadgetoid/PY_LGPIO

Grab the dist files from artifacts here: https://github.com/Gadgetoid/PY_LGPIO/actions/runs/6315704729

[Gadgetoid]

Note: I think that lgpio_extra.py and 'lgpio.i' should be inherited from lg in my Python-package-only setup. That would cleanly delineate the Python packaging from the stuff that's likely to change together.

[Gadgetoid]

Okay I'm still missing headers, let me sort that.

[Gadgetoid]

Right, the sdist package in PY_LGPIO.zip here should include everything needed to build - https://github.com/Gadgetoid/PY_LGPIO/actions/runs/6315882955

[Gadgetoid]

Tagged a release here, which I'm periodically updating if things need to change: https://github.com/Gadgetoid/PY_LGPIO/releases/tag/0.2.2.0

[Gadgetoid]

I've just backdated all releases, so you can grab the 0.2.0.0 version that works with gpiozero here: https://github.com/Gadgetoid/PY_LGPIO/releases/tag/0.2.0.0

Just installed that into my venv and did a basic test, seems to work! 🎉

[Gadgetoid]

Okay just to loop @joan2937 and @waveform80 back in here- sorry for the ceaseless git spam folks- I have:

1. Taken the Python bindings from lg and put them into a new repository
2. Added lg as a git submodule so that source files can be accessed
3. Linked lgpio.i and lgpio_extra.py to the lg versions, so they can continue to be maintained by @joan2937
4. Modified setup.py to build the bindings and lg into a single library, which is included in the package
5. Bumped the submodule through every version of lg and created releases for each (the 0.2.2.0 release linked above is compatible with gpiozero)
6. Updated the README to reflect installation from pypi is now possible

Upstream, potentially everything in PY_LGPIO is obsolete except lgpio.i and lgpio_extra.py which the packaging currently symlinks (these symlinks are resolved to static files when packaged into a dist).

The reason I have done it this way, is because the Python package needs to access the lg source files in order to include them in a source (buildable by PiWheels and users alike) distribution. For that to be possible (read: not a total hacky mess) the Python bindings need to be a parent of the source dir rather than a child.

Generating a new release is as simple as bumping the lg submodule to the new tag, updating the setup.py version number and tagging accordingly. GitHub then spits out a .tar.gz source file which can be uploaded straight to pypi.

Both .whl and source distributions are generated for pull request CI, so someone can raise a PR to bump the release and get back packages which they can test before it is merged.

Having the Python packaging separate also lets us work on making the packaging nicer - fleshing out the readme, adding qa, tests and examples, adopting new standards etc - without bothering Joan with Python nonsense 😆 .

[joan2937]

I need to make crystal clear that I have no understanding of how you guys do the packaging.

If you need me to do anything from my end (/joan2937/lg) just tell me what to do and I will try to do it.

[Gadgetoid]

I need to make crystal clear that I have no understanding of how you guys do the packaging.

That's fine, I just don't want to do anything that you might object to, but if you don't know what I'm doing you can't object 🤔😆

Part of the goal here is to make packaging standalone so you don't have to worry about it at all.

Though I have no understanding how the Debian packaging works and how this might work with it.

[Gadgetoid]

@bennuttall had a chance to test the linked packages on PiWheels?

I am planning to try this same trick for libgpiod so we can get a PiWheels-compatible distribution of the official Python bindings, too.

[Gadgetoid]

gentle nudge @bennuttall @waveform80

I am currently putting the finishing touches on a set of patches to libgpiod that accomplish effectively the same thing I'm proposing here. Some feedback as to if this approach will work for you, plus some good ol' peer review would be much appreciated.

[bennuttall]

Was talking to @waveform80 about it and we think there's a way around this - should be possible to make it buildable from source if you have the apt package installed - then it would be linked to the apt-installable version - which is generally the way piwheels works.

[Gadgetoid]

buildable from source if you have the apt package installed

This same general premise is why libgpiod is getting the library (optionally) vendored. Relying on distro package management to set the pace for Python package releases is... not great.

What happens if there's an API break between the Python bindings and the distro package? The latter cannot be readily updated, and the end-user doesn't really have control over the former since any arbitrary version can be specified as a dependency by a Python library, driver or project.

[waveform80]

buildable from source if you have the apt package installed

This same general premise is why libgpiod is getting the library (optionally) vendored. Relying on distro package management to set the pace for Python package releases is... not great.

When the library is under active development that's certainly true, and the gpiochip interface (which both libgpiod and lgpio depend upon) certainly did go through some upheaval fairly recently (to add pull-up/down facilities which finally made it useful on a Pi).

However, I'd hope at this point it's a pretty stable kernel interface and moreover, if we're in control of both the apt and PyPI packages (I currently do the apt packaging, and I think we've now got upload access on the PyPI side too) that shouldn't be an issue going forward (I say going forward, because we haven't bought the PyPI side up to date yet, but hopefully we should do that soon -- my apologies for the delay on this!).

What happens if there's an API break between the Python bindings and the distro package? The latter cannot be readily updated, and the end-user doesn't really have control over the former since any arbitrary version can be specified as a dependency by a Python library, driver or project.

The distro package is harder to bump than PyPI, certainly as we have to go through other people on both RaspiOS and Ubuntu, but if we're in control of both styles of packaging it should be possible to keep it synchronized.

[waveform80]

Edit: notable quirk for supplying a source distribution is that it's difficult to include the parent directory in the Python package.

Quick aside on this bit. There's an easy work-around for this (which I've used in my local experiments, and which I mention here in case it's of any use to you): ln -s .. PY_LGPIO/src. In other words, stick a sym-link called "src" to the parent dir within the python sub-directory and now you can pretend the C source is under the Python stuff.

[Gadgetoid]

if we're in control of both styles of packaging it should be possible to keep it synchronized

What about everyone else on other development boards, or Linux devices with GPIO access who might want to contrive a way to use our drivers or otherwise? Why even bother using the standard interface if we default to and recommend some incredibly specific, fortuitous combination of Pi-only packages in order to actually use it?

I guess by and large I don't really mind if the answer is "we only really care about Pi and other systems could kinda sorta work if they put in the leg work", but it does affect where and how I decided to leverage gpiozero + lgpio as a back end in our user-facing libraries.

By and large we (Pimoroni, since that's effectively who I represent here) are a Pi-only house, and we support only Raspberry Pi OS (though I see parity with Ubuntu is now so good that I'm strongly inclined to test everything there too) but if I'm making a transition from RPi.GPIO in my code, I'd like that to be the last one I ever have to make 😆 which means standards, baby, standards all the way down!

In other words, stick a sym-link called "src" to the parent dir within the python sub-directory and now you can pretend the C source is under the Python stuff.

symlinks sorta work, sure, but who wants to put condemn the Python bindings to a mostly C repository maintained chiefly by someone who neither knows nor cares about Python? It's just noise and busywork... not to mention you forfeit any ability to version and release the Python bindings without pulling in some development version of the C library.

You're speakin' to the guy who thrust WiringPi into the Python ecosystem and maintains the rpi_ws281x bindings for better or worse. I know a thing or two about how to structure these kinds of projects and have been doing this at least as long as rgpio, lgpio or whatever has been around.

[waveform80]

What about everyone else on other development boards, or Linux devices with GPIO access who might want to contrive a way to use our drivers or otherwise?

My understanding was there isn't anything raspi-specific in lgpio (or rgpio)?

There is in gpiozero, but that's stuff I'm actively trying to remove. The two last vestiges at the moment are the board-info stuff which I tried to make generic in 2.x and ran out of time, but there's still an dev branch somewhere with a mass of incomplete changes for that, and the "native" fallback driver (but I'd hope nobody really uses that going forward as it's incompatible with the Pi 5, and doesn't do PWM anyway).

By and large we (Pimoroni, since that's effectively who I represent here) are a Pi-only house, and we support only Raspberry Pi OS (though I see parity with Ubuntu is now so good that I'm strongly inclined to test everything there too) but if I'm making a transition from RPi.GPIO in my code, I'd like that to be the last one I ever have to make 😆 which means standards, baby, standards all the way down!

I'm 100% in agreement that that's the way to go. That's partly why I've been pushing Ubuntu towards stuff that sits atop gpiochip as it should be a more stable basis for the future than fighting the kernel over registers, and also more generic.

symlinks sorta work, sure, but who wants to put condemn the Python bindings to a mostly C repository maintained chiefly by someone who neither knows nor cares about Python? It's just noise and busywork... not to mention you forfeit any ability to version and release the Python bindings without pulling in some development version of the C library.

My (largely selfish) issue here is the complexity it adds to the Debian packaging situation. At the moment I can produce liblgpio1, librgpio1, python3-lgpio, python3-rgpio, etc. etc. from this single source and the fact it's a single source guarantees they're all in lock-step and compatible.

[waveform80]

Anyway, my apologies again for taking so long over this -- I'll try and get something finished on this, whether this solution or otherwise, as soon as I can.

[Gadgetoid]

Thanks for your detailed replies here, I’ll probably need some time (when I’m not in a mad dash around the kitchen) to give them due consideration but it’s useful to know what motivates you and where and why my approach won’t necessarily work (with “work” including “not cause you undue headaches”).

Sorry I got a bit aaaaah about the delays getting this sorted, but ultimately I totally appreciate that we’re all busy all the time 🤣 right now in particular.

Another evening of libgpiod patch revisions for me 😭

[Skidude240]

Just as a quick comment. i've spent about a day last week trying to get gpiozero to work with none system python on a Pi 5. It wasn't a wonderful experience. In the end the only solution that i got working was to use the repository @Gadgetoid has setup and pip install the lgpio module directly from there. I'd figured out that the lgpio version differed between the system install and the pyenv one but i couldn't figure out how to get the newer version from pip.