Pinning of dependency versions

[pinkwah]

SCOUT's main export is the Komodo stable distribution. In this, the entire environment is determined. In order to update the environment, the release manager has to manually modify the package versions (with the help of package maintainers).

However, in our public package sources, we tend to be a lot more hands-off when it comes to versioning. Whenever we pin a version, we make an issue to unpin it at a later date.

While this helps us stay up-to-date with the latest versions, possibly benefitting from bug-fixes and new software features, it has the downside of us stumbling onto incompatibility issues with no warning and being unable to merge into main. The latest incident occurred when the fastapi bumped their starlette version. The latest Starlette version dropped support for Python 3.6, which caused all of our CI pipelines to start failing until the issue was pinned. That is, instead of dealing with the day-to-day work, some people had to shift into fixing this particular issue.

In order to avoid this type of issue, I propose that we start pinning versions for all packages. Most programming language package managers support lock-files, which list all package versions, making it possible to have reproducible installs. Pip, however, does not have this functionality (yet). As such we need to resort to manual ways of specifying versions.

Python kind of has this functionality if one uses pip freeze to dump all the package versions in a single virtualenv. However, this dump would contain all packages, including transient dependencies, or any additional package that the developer happened to have installed in their environment. As such, we would still need to have a single source of truth, perhaps as a install_requires part of setup.py.

A less conservative alternative (which I personally support), would be to pin packages using semver. Any package that purports to use semantic versioning should be pinned with >= 1.2; < 2.0. Any package that does not use semantic versioning should be marked as such with a comment and pinned to an exact version. An example of a non-semver package is SQLAlchemy, which was somewhat backwards-incompatible between 1.3 and 1.4. Pinning like this includes testing dependencies like black and mypy.

In order to deal with the packages getting outdated, I propose adding a new SCOUT role. The "Update Manager" will be chosen alongside the Release Manager every month. Their goal will be to bump all packages in our repositories, perhaps once every Monday. Most weeks this would pass without any issues. Should a breaking change in a dependency occur, the Update Manager should make an issue in the relevant repository and pin the dependency to a version that does not fail, while bringing this issue up in the relevant Slack channel.

When bumping a SCOUT package in Komodo, the bumper can also update transient packages in Komodo to the version in the Github repositories. I think this will also help keep Komodo stable and git upstream packages much closer.