Skip to content

Release Process

Jump to bottom
Erik Jaegervall edited this page Nov 22, 2023 · 3 revisions

See also https://github.com/eclipse/kuksa.val/wiki/Release-Process

See also content of previous releases

Checklist

  • Make sure we are feature complete
  • Check links
  • Update dependencies
  • Create kuksa-client PyPI pre-release
  • Perform testing
  • Create local release tag
  • Upload "official" PyPI release
  • Update kuksa.val.feeder components to use new PyPI release and test them
  • Feeder test passed (Not Alpha components/features)
  • Create github tag for repo
  • Docker containers have been triggered and generated and tested for both this repo and feeders
  • Create github pre-release of this repo
  • Create github pre-release of kuksa.val.feeders
  • Merge release-branches (if used) back to master/main
  • Project lead informed
  • Official release approval received
  • Releases transformed to official releases

Release Process in short

  1. Merge all functional changes that should be included
  2. Create a PyPI pre-release of kuksa-python-sdk (needed for sanity checks)
  3. Perform testing according to https://github.com/eclipse/kuksa-python-sdk/wiki/Release-Testing
  4. When all findings are corrected and if needed retested, then tag and upload a tag
  5. Create an official PyPI release of kuksa-client
  6. Update kuksa.val.feeders to use the new official PyPI release (remove --pre, set version, regenerate from *.in`) and do some sanity check
  7. If it works create a Github pre-release
  8. Ask project owners to review and approve release, When done transform to real release

Make sure we are feature complete

No remaining important PRs or issues?

Check links

The commands below can be used to verify that links in markdown files works as expected

npm install -g markdown-link-check

find . -name \*.md -print0 | xargs -0 -n1 markdown-link-check

Or stricter:

markdown-link-check -q -a 403,200,404 *.md

To check all but ignore third party (submodule) code do from root repo folder:

find . -name \*.md -print0 | xargs -0 -n1 markdown-link-check -q -a 403,200,404

Update Dependencies

For all releases except patch releases we intend to update dependencies to latest. (Unless there is a good reason not to do it). The reason for this is that it is more likely that security vulnerabilities (if detected) will be fixed on recent releases.

cd kuksa-client
pip-compile --upgrade --resolver=backtracking setup.cfg
pip-compile --upgrade --extra=test --output-file=test-requirements.txt --resolver=backtracking setup.cfg

Trigger builds manually

image

In total four builds, below plus kuksa-val:

image

Do the same for feeder

Creating Releases/Pre-releases for kuksa-client

For testing purposes kuksa.val.feeders may need pre-releases of the kuksa-client Python package. This can be done as follows.

Pep 0440 describes Python versioning scheme. An alpha-release can be uploaded as a pre-release to pypi.

To create a pre-release a clean tagged repository is needed. Note that the whole repository needs to be clean even if you are to only create a kuksa-client pre-release.

cd ~/kuksa.val
# Or do a "git status" and check that it reports a clean status
git clean -fdx
git tag -a 0.4.0a1

The pre-release identifier (a1 above) shall follow PEP versioning rules. As of now we have started with a1, i.e. a0 has not been used.

You can check that you get the wanted version like below. If you get something saying post or dirty your repository is likely not clean or you have not tagged the latest commit. The warning comes from pyproject.toml but does not have any effect.

user@debian:~/kuksa.val/kuksa-client$ pip install setuptools-git-versioning
user@debian:~/kuksa.val/kuksa-client$ python -m setuptools_git_versioning
[2023-04-13 10:51:37,352]  WARNING: tag_filter is not a valid function reference: Wrong reference name: [0-9]+.[0-9]+.[0-9]+
[2023-04-13 10:51:37,481]  WARNING: tag_filter is not a valid function reference: Wrong reference name: [0-9]+.[0-9]+.[0-9]+
0.4.0a1

Then you can build the distribution like this. Make sure that the end-result will have names similar to this.

user@debian:~/kuksa.val/kuksa-client$ rm -rf dist 
user@debian:~/kuksa.val/kuksa-client$ python -m build
tag_filter is not a valid function reference: Wrong reference name: [0-9]+.[0-9]+.[0-9]+
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (grpcio-tools>=1.46.0, setuptools-git-versioning, setuptools>=42, wheel)
* Getting build dependencies for sdist...
tag_filter is not a valid function reference: Wrong reference name: [0-9]+.[0-9]+.[0-9]+
Matched 0.4.0a1
Matched 0.3.1
Matched 0.3.0

< lot of lines removed >

removing build/bdist.linux-x86_64/wheel
Successfully built kuksa_client-0.4.0a1.tar.gz and kuksa_client-0.4.0a1-py3-none-any.whl

The result can be uploaded to pypi or to the pypi test repository, if you have the rights to do so. Uploading to testpypi may have limited values for released packages.

python -m twine upload --repository testpypi dist/*
python -m twine upload --repository pypi dist/*

To be able to upload you need a config file, similar to this one:

user@debian:~/kuksa.val/kuksa-client$ more ~/.pypirc
[distutils]
  index-servers =
    pypi
    testpypi

[testpypi]
  username = __token__
  password = pypi-<token content>
  
  
[pypi]
  username = __token__
  password = pypi-<token content>

If successful it will be shown as a pre-release at pypi

image

By default pip does not install pre-releases, so you need to specify that with --pre. Just requesting kuksa-client to be installed will result in the latest stable release will be used:

user@debian:~/kuksa.val/kuksa-client$ pip install kuksa-client
Defaulting to user installation because normal site-packages is not writeable
Collecting kuksa-client
...
Successfully installed kuksa-client-0.3.1
user@debian:~/kuksa.val/kuksa-client$ pip install --pre kuksa-client
Defaulting to user installation because normal site-packages is not writeable
Collecting kuksa-client
...
Successfully installed kuksa-client-0.4.0a1

pip supports specifying multiple repositories, but if a package name exists in both only the official pypi repository will be checked.

# This will only check for kuksa-client in https://pypi.org/simple/
pip3 install --pre --index-url https://test.pypi.org/simple/  --extra-index-url https://pypi.org/simple/ kuksa-client==0.4.0a1

# This will use kuksa-client from test.pypi
pip3 install --pre --index-url https://test.pypi.org/simple/  kuksa-client==0.4.0a1

Using pre-releases in kuksa.val.feeder

To be able to use a pre-release two things must be modifed. The first version dependency like in requirements.in. There the dependency must be changed so it includes or possibly even require a newer version. For the 0.4.0a1 example the version requirement kuksa-client ~= 0.3.0 must be changed, for example to kuksa-client >= 0.4.0a1. That will accept any 0.4.* release (we hope), including pre-releases. Specifying kuksa-client ~= 0.4.0 does not work.

The requirements.txt file must be regenerated considering pre-releases.

pip-compile --upgrade --pre requirements.in

Finally any file using pip install must be updated to also include pre-releases, like Dockerfile

RUN /opt/venv/bin/python3 -m pip install --upgrade pip \
    && pip3 install --pre --no-cache-dir -r requirements.txt

Creating alpha PyPI packages is only needed if you cannot test with a locally installed kuksa-client package, for instance to build docker containers!

How to create kuksa.val.feeder releases.

See https://github.com/eclipse/kuksa.val.feeders/wiki/Release-Process

Create KUKSA.val Github release

  • Tag the repo and upload tag
  • Hopefully this will trigger builds of at least databroker and databroker-cli and publish new docker containers
  • Fetch *.zip binaries for databroker and databroker-cli

Take latest arm64/amd64 images from latest build on master (or the right tag)

https://github.com/eclipse/kuksa.val/actions/workflows/kuksa_databroker_build.yml

https://github.com/eclipse/kuksa.val/actions/workflows/kuksa_databroker-cli_build.yml

I.e. in total 4 *.zip files shall be downloaded and added to the release

  • Create a pre-release based on tag.
  • For now, as long as repo has not been split, the release name shall be KUKSA.val X.Y.Z Release
  • Manually select previous release and generate release notes
  • Copy instructions based on previous release (part above ## What's Changed), i.e. information on how to fetch from docker
  • Also say something about the included *.zip files
  • Select "Set as a pre-release "
Clone this wiki locally