Python support in future versions

[amanda11]

Starting an issue to discuss where to go next with Python versions and StackStorm, and how to distribute.

Current StackStorm distributes O/S specific packages with a specific StackStorm version. Currently:

• python 3.6 Ubuntu 18
• python 3.8 Ubuntu 20
• python 3.6 EL 7
• python 3.8 EL 8

Python 3.9 and Python 3.10 are available. Python 3.6 has gone EOL (however RHEL are supporting it until 2029).
EL 9 is available with python 3.9 and U22 is available with python ???

What python versions do we want to support?
Do we still want to provide O/S specific packages tied to a particular python, or want to move to a distribution mechanism that doesn't restrict to a prescribed set of python/OS combinations.

Would welcome opinions on:

• Do we continue to provide a package distribution tied to particular O/S or want to distribute at different level?
• What python versions do we want to support in next release, and perhaps longer term?
• Even if we distribute at lower level, do we still define a set of OS/python combinations that we then run through a full install at? (The distribution would probably support others, but we may want to have a subset that we actually run through)
• If we do run through CI on OS/python combinations - what combination do we want to go forward and support?
• If we distribute at a lower layer what is the preferred way of doing so?
• If we distribute at a lower layer do we still provide all 4 current non-containerised installers: manual, bash installer, ansible and puppet?

[amanda11]

Popping in some notes from Slack...

Just to throw another spanner into the python release discussions. RH 8 only supports python 3.8 until May 2023. Up to 2024 for python 3.9 and 2029 for python 3.6 - https://access.redhat.com/support/policy/updates/rhel-app-streams-life-cycle#rhel8_application_streams. RH 9 I believe have python 3.9.

So our upgrade from python 3.6->3.8 on RH, meant we moved to python 3.8 which python support until Oct 2024, but RH only support up to 2023, and actually support python 3.6 for longer.

I'm not sure what this means for Rocky 8, are they following the python or RH supported python versions?

Maybe, adds another argument for not doing the package generations? And perhaps a reason to keep python 3.6, and add 3.9 to the mix...

Just wondering if next release we could add support for python 3.9 but do that without a package build. Even starting with say running unit/integration tests on python 3.9, then publishing python 3.9 wheel and manual instructions. (By manual I mean producing equivalent of our manual instructions on site, but with the st2 package install replaced with a install of wheel etc - but hopefully we can script up much of that). If we have that process working, we can then work on putting that into installers...

Or alternatively if we published say the 3.6 or 3.8 wheel that gets generated as part of the current packaging, could we define instructions/scripts on how to install with the current wheel.
The wheel currently
generated by the ST2 packaging process that we throw away is labelled as "any" O/S - but not sure if that really is true and perhaps manylinux would be better....
I'm assuming (but not verified) that the wheel itself at that stage isn't O/S specific). But taking one of the existing python versions and working on it allows to verify the process knowing that it should work on all our current target O/S.

And the wheel and 3.9 could be done in parallel, don't block each other.

Or I suspect someone has a better plan. e.g do we benefit from wheel rather than source distribution?

But at least wanted to give a heads up on the early retirement of python 3.8 on RHEL, and that 3.9 is probably needed for any RH users from mid next year.

[amanda11]

I think Ubuntu 22.04 is shipped with python 3.10, so that is another one to add to the mix!

[nzlosh]

Here's a matrix of currently supported versions of Python on various distributions above and beyond the currently set we support.

	python3.6	python3.7	python3.8	python3.9	python3.10
ubuntu 18.04	X	X
ubuntu 20.04			X	X
ubuntu 22.04					X
debian 10		X
debian 11				X
centos 7	X
centos 8	X			X
centos 9				X
opensuse leap 15.3				X

From an ideological stand point, I'd like to see StackStorm tested against all current versions of Python 3.7-3.10. This would open the door to a lot of other Linux distributions.

[nzlosh]

If we are going to add Python 3.9, my gut feeling is the effort required to add 3.7 and 3.10 would be minimal (at least worth investigating the difficulty).

[amanda11]

@nzlosh Can we add python 3.9 and Centos 9 to your matrix?
We also do have 3.8 on Centos 8 - its just only until mid 2023.

My gut also is that 3.9 and 3.10 need adding. I'm just wondering if 3.7 is worth it. More on the basis of 3rd party libraries we pull in, as we'll be restricted to 3.7+ libraries... However, I'm not sure that really is an issue for us - as most of the newer 3rd party versions seem to be for python 3.7+.

However, that also brings into question as to when to drop 3.6. Dropping 3.6 allows us to move up versions, but 3.7 doesn't seem to restrict us. But looking at matrix and our 2 O/S previous idea of support, dropping 3.6 should mean adding either 3.7 + 3.9, or 3.9 and 3.10, or all 3.

If we have the self check running on the github actions, and we manage to do the install without packages, then it makes it probably easier to verify multiple python versions...

[nzlosh]

Added CentOS 9 to matrix.

You're right, Python 3.7 may not be worth adding since it's going to be EOL in a years time.

Release	Released	Security Support	Latest
3.1	8 months ago (04 Oct 2021)	Ends in 4 years (04 Oct 2026)	3.10.5
3.9	1 year and 8 months ago (05 Oct 2020)	Ends in 3 years (05 Oct 2025)	3.9.13
3.8	2 years and 8 months ago (14 Oct 2019)	Ends in 2 years and 4 months (14 Oct 2024)	3.8.13
3.7	3 years and 11 months ago (26 Jun 2018)	Ends in 1 year (27 Jun 2023)	3.7.13

Table sourced from https://endoflife.date/python

Regarding python 3.6 support, I seem to recall we already hit problems where upstream module dependencies have dropped Python 3.6 because it's EOL and we have to pin to older versions. This means St2 will miss bug and security fixes the longer the pinning is held. Our ultimate dependency hell destination will be when the pinned module causes a conflict in a cross dependency that blocks development completely.

I think we need to re-evaluate if it still makes sense for an open source project to limit itself to 2 distributions. I feel like this decision is a relic of StackStorm corporate days where support contracts were easier to respect on a constrained set of distributions.

[amanda11]

I was more thinking we want to support at least 2, rather than limiting to just 2! So if we dropped 3.6 now, we'd be limiting us to just only support one EL and one Ubuntu stream (if we don't support anything other than python 3.8)
And I agree I think we are already hitting problems with the python 3.6 on some dependencies...

[nzlosh]

If we drop 3.6 and add 3.9 now we'll get functionality on 5 Linux distributions. But we won't have OS packaging available for all those distributions. That could be added with time. The other question to be answered would be if these OS' have the correct versions packaged for our external services (mongo rabbitmq and redis). Or do we consider them to be outside the scope of StackStorm and it's the users responsibility to provide the services on or off the StackStorm host?

[amanda11]

If we drop 3.6 and add 3.9 now we'll get functionality on 5 Linux distributions.
Downside, only one Ubuntu version...

[amanda11]

Related - perhaps we should also update the st2client binary distribution - its python2.7 - https://pypi.org/project/st2client/#files

[nzlosh]

If Python 3.7 turns out to be straight forward to integration with 3.9 and 3.10 support, it would mean we can continue to support U1804 without keeping py3.6 support.

Poetry could be a good tool to help manage StackStorm dependencies. They way you can express module dependency versions in relation to python version would invaluable when attempting to manage dependencies across multiple versions of Python.

[cognifloyd]

For building wheels: I have a working / mostly-complete setup that can churn out wheels in my pants experiment. And I'd like some feedback.

Test procedure:

• Clone and checkout the pants branch of the st2sandbox/st2 repo.
  https://github.com/st2sandbox/st2/tree/pants/
• In the root of your checkout, run these (:: means everything. you can replace it with a path to a file or directory):
  • ./pants (to make sure bootstrapping pants works on your machine)
  • ./pants help (a key resource for learning about pants)
  • ./pants package st2common (to build only the st2common wheel and sdist and put it under dist/)
  • ./pants package :: (to build all the wheels and sdists and put them under dist/)
  • ./pants lint --lint-only=black :: (to lint with black and report if it would change anything across the repo)
  • ./pants lint --lint-only=black :: (to feel the power of pants' fine-grained caching)
  • ./pants lint :: (to lint everything - almost all of the lint targets from the Makefile have been implemented)
  • ./pants lint :: (again to show how pants has cached the results. Since no files have changed, the old results are still valid)
  • ./pants fmt :: (this one runs black to actually reformat files, and regenerate files like the api spec, the metadata schemas, and st2.conf.sample. And, because pants knows about dependencies, it will only regenerate if any of the python files involved in generating those have changed.)
  • ./pants lint fmt :: (you can also add multiple goals, and pants gladly runs all the tools intelligently - ie it will only run black once which covers both lint and fmt, instead of once for each goal)
  • ./pants export :: (to create virtualenvs under dist/virtualenvs - note how every tool gets its own venv so their deps don't mix/interfere with the st2 deps.)
• Check out the main lockfile at lockfiles/st2. Every dependency, including transient dependencies, is locked to a particular version! So, CI and local results should not drift any more! To regenerate the lockfile run:
  • ./pants generate-lockfiles --resolve=st2
  • Also, this is using the latest pip resolver

Now that pants is managing dependencies, some of the Makefile's dependency-related lint targets aren't needed anymore. That leaves only two lint tasks that I have not converted from Makefile over to pants:

• rst checking targets (tools: rst-check, readme_renderer, rst-lint)
• st2client-install-check: this should be reimplemented as a test as it is not really a lint task. With pants, we can make a test depend on the generated artifacts (wheels in our case) so it can install them and make sure the wheel is working correctly.

[cognifloyd]

Re poetry: I've played with poetry, but it doesn't work very well with monorepos (multiple python packages in the repo). So, I didn't have much luck getting poetry into the StackStorm repo.

[cognifloyd]

I would like to see us publish the wheels to pypi (for a variety of reasons, including easier pack dev environment setup). We can then use those wheels when we build the deb/rpm files to maintain all the current infrastructure as is, at least in the short/mid term.
Long term dropping deb/rpm support is a possibility, but I think publishing the wheels will go a long way towards allowing people to test StackStorm in more environments and see what python/os combos work and which don't.

[cognifloyd]

Here is a CI run where pants is handling all of the lint tasks: https://github.com/st2sandbox/st2/actions/runs/2543590121

https://github.com/st2sandbox/st2/actions/workflows/lint.yaml
linting with an empty pants cache: 6 min 34 sec
with a populated pants cache: 2 min 16 sec

[amanda11]

udatetime doesn't build on python 3.9. See StackStorm/st2#5346 where it was discussed moving to an alternative https://github.com/closeio/ciso8601

[nzlosh]

Part of supporting newer versions of Python is the projects dependencies. I took a look at MongoDB which has its own release life cycle. We test against MongoDB v4.x but v5 and v6 have been released. The last v4 minor release v4.4 is end of life next year and there needs to be code base updates to support the new major versions.

Looking at the pymongoengine matrix it looks like a move from pymongo==3.11.3 to pymongo==4.3 will give support from MongoDB 3.6 to 6.0 running on Python 3.7 to 3.11

A quick check of the documentation vs the code base shows the following matches:
https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#renamed-uri-options
https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#tz-aware-defaults-to-false
https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#database-collection-names-is-removed
https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#collection-save-is-removed
https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#collection-count-and-cursor-count-is-removed (perhaps, need to look closer at code base)
I don't have much more time to look at this for the moment. So I'll just put this here for awareness.

[nzlosh]

Working on U22.04 support, I've discovered that mongodb does not provide official packages for mongodb-server other than v6.0. If we want to officially support u22.04 in favour of u18.04, supporting mongodb v6 will be a hard requirement.