[WIP] dissociate python optional test wheels builds + python313

[th0ma7]

Description

Intent is to:

1. Remove wheel build testing from python310/python311 base Makefiles - This will ease updates
2. Create a defaul python-wheel testing package (or one per python version, tbd) - This will ease in-depth testing
3. Copy over all/most of our packaging wheel recipees into the new python-wheel test package. To help on regression-testing when upgrading python base package
4. Create a first python 3.13 package based on cleaned-up 3.11

Python 3.13 current build issue: python/cpython#125452

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• New Package
• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

Strategy

• Added a mk/crossenv/ directory containing wheel crossenv definitions, defaults to requirements-default.txt
• Can be called from the command line using:

$ make crossenv-x64-7.1

• To use a specific wheel definition:

$ WHEEL="aiohttp==3.8.5" make crossenv-x64-7.1

If resulting requirements-aiohttp==3.8.5.txt does not exists, falls back to requirements-aiohttp.txt then requirements-default.txt

[th0ma7]

@hgy59 I'll let this run but this patch in particular 9c764a7 needs to be in its own PR... interesting from not having noticed it earlier. I'll also mark as [WIP] as not even close to be merged, this was early work being push to initiate exchange with python github relatively to the build issue I had.

[th0ma7]

@hgy59 bind compiling tries to install packages ?!?!? I'll try to find where that comes from but in case it ring any bell...

===>  Patching for bind
===>  Configuring for bind
===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local/bind --localstatedir=/usr/local/bind/var --enable-full-report BUILD_CC=cc BUILD_CFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_CPPFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_LDFLAGS=-L/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath-link,/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath,/usr/local/bind/lib  BUILD_LIBS=
===>  - Install prefix: /usr/local/bind
===>  Install libtool binary
sudo apt-get update
[sudo] password for spksrc: 

[hgy59]

@hgy59 bind compiling tries to install packages ?!?!? I'll try to find where that comes from but in case it ring any bell...

===>  Patching for bind
===>  Configuring for bind
===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local/bind --localstatedir=/usr/local/bind/var --enable-full-report BUILD_CC=cc BUILD_CFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_CPPFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_LDFLAGS=-L/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath-link,/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath,/usr/local/bind/lib  BUILD_LIBS=
===>  - Install prefix: /usr/local/bind
===>  Install libtool binary
sudo apt-get update
[sudo] password for spksrc: 

@th0ma7 I will remove this in #6269. It is obsolete since spksrc image on debian 12.

[th0ma7]

@hgy59, @mreid-tt and other @SynoCommunity/developers I'd appreciate having some thoughts on this...

I always kept within spk/python3* examples to build from and test cross-compiled wheels. While it served its purpose it also led to complexifying the overall maintenance of default python makefiles. In hope to simplify this I first thought it would make sense moving all of that under a python-wheels or similar sub-package, sort of placeholder to test wheel building when upgrading python / pip / setuptools and al (around where this PR is atm).

Doing so I thought why not building them all and include other remanants dispersed into other sub spk ...

• While it may look neat on paper this becomes a burden to maintain as indivudual spk will evolve over time and this centralized copy won't serve it's purpose any longer...
• On the other hand I do believe we need a central location providing clear and up-2-date examples and to test new versions of complex wheels while upgrading python versions.

Then I recall someone mentioning a while back ago, why not having our own wheel repository?

• This could avoid maintaining cross-compiled wheels into individual packages + reducing package size as it would then download from our wheel repository at install time.
• This centralized python-wheels could potentially meet that purpose.
• While neat on paper I wonder what effect would lead upgrading dependent cross/* existing wheels ...

Thoughts on this would be much appreciated.

Also on my TODO (to which babysteps may be best):

1. make available a python 3.13 package (undergoing)
2. fix CMake based wheel building (requires sort of autodetection to allow passing proper parameters to at build-time)
3. fix meson based wheel building (similar issue to CMake type)
4. decouple crossenv from cross/python3* makefiles
5. allow either on-demand OR legacy+latest crossenv (for compatibility reasons)

[mreid-tt]

@th0ma7, I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:

1. Moving test code examples to "build from and test cross-compiled wheels" seems like a solid idea, aligning well with the purpose of the demoservice and demowebservice packages.
2. Setting up and maintaining our own wheel repository appears to be a significant long-term commitment.

I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code?

Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support?

I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions.

[th0ma7]

I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:

1. Moving test code examples to "build from and test cross-compiled wheels" seems like a solid idea, aligning well with the purpose of the demoservice and demowebservice packages.

Indeed. Package could even be renamed similarly.

2. Setting up and maintaining our own wheel repository appears to be a significant long-term commitment.

Not really. That would mostly be just a web page listing our wheels, that you can query through json to list our pre-compiled wheel packages. From there in our requirements.txt file provided with each package, we add a URL to fetch from. Then when wheels gets installed at package installation, it can download them from our source, similarly to using pypi.

I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code?

Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support?

Issue is, pypi provides tons of pre-compiled wheels but ppc, armv5, (and often armv7 and even aarch64) are always missing. Therefore, on our NAS at installation time pip will only find the source of the requested wheels and try to compile them locally... and failing miserably as our Synology NAS does not provide a pre-installed build system toolchain and we're not providing the include files of the various dependencies.

Therefore when we build our spk package using our spksrc framework, we pull source packages from pypi usng pip, then cross-compile them and generate a resulting *.whl file. This wheel file gets packaged withing the spk so at package installation installation time it is being fetched by pip after failing to find a valid source online.

I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions.

The challenge is that the python wheel build system is undergoing a lot of changes currently and impacting the overall wheel building approaches. As pypi modules documentation and maintenance varies from one to the next, this ends-up breaking things all over the place.

So question is, would it be easier to have a single location to manage all wheel cross-compiling, providing numerous versions online to ease package management? I've been thinking of this a little more and still unsure, further as this would probably mean statically linking wheels so it is compatible with any installation OR using rpath so dependencies are available from that python-wheel package for instance.

[Safihre]

It makes sense to have our own repository. Although it will be quite a lot of work to implement and significantly add to maintenance burden..
I saw there's applications available that allow this, for example https://github.com/pypiserver/pypiserver