Skip to content

MakingANewRelease

Jump to bottom
Brice Goglin edited this page Aug 14, 2018 · 13 revisions

How to Make a New Hwloc Release

There are several steps that are required to make a new hwloc release/prerelease and/or release series.

There are two cases:

  1. You are making a release on a brand-new release series (it's probably a pre-release, like an alpha, beta, or RC -- but hey, it could be a final release, too).
  2. You are making a release on an existing release series -- meaning that you've already followed the previous steps for initially creating the release series; now you're just putting a new release out on that existing series.

Announces

We usually announce the release to the following mailing lists:

Make a New Hwloc Release Series

This is the case where you need to make a release for version A.B.C where C==0 -- the release is probably a pre-release of some kind (e.g., an alpha, beta, or rc tarball). None of the infrastructure yet exists to make this new series. There are several systems that are affected:

  1. Build system
  2. Git
  3. Nightly / official tarball build server
  4. Web site

Let's go over each of these in order.

Build System

Before making a branch for the new release series, it's usually a good idea to evaluate if upgrading to newer versions of the GNU Autotools is a good idea. Specifically: we make an effort to pick a tuple of Autotools for use with a release series and try very hard to use those same exact versions for every official release in that series.

If you end up upgrading the version of Autotools that you're using, it is best to edit source:/contrib/dist/make_dist_tarball (on the master -- before you make the Git branch for the new series) to set these versions. Run the script to ensure that these versions work properly with the master branch. Tune up anything that you need on the master branch to make these versions work. Once it's all working, commit the new version of make_dist_tarball with those version numbers back to the master branch.

API Version, etc.

If the API has changed, increasing HWLOC_API_VERSION in hwloc.h is usually a good idea. This should only be needed for major releases. Do it in master before branching.

Also update HWLOC_VERSION in contrib/windows/private_config.h.

If struct hwloc_topology has changed, or #define used inside it, or one of its children structures (internal distances, support, bitmap), HWLOC_TOPOLOGY_ABI should be bumped too.

If the plugin ABI has changed (include/hwloc/plugins.h), bumping HWLOC_COMPONENT_ABI might be necessary too.

Git

The main thing to do is to create a release branch in Git. Once you get the master branch in a good state, branch it. Note that there is a naming convention for branches:

$ git branch vA.B

where "A.B" is obviously the numbers that are relevant for your new release series (e.g., "1.1").

Once you make this branch, check it out and edit the VERSION file. Ensure that its VERSION numbers are set to A.B and the greek is set to something relevant (e.g., "a1" or "rc1").

Set the .so library version to the appropriate value (see the Libtool documentation for instructions: page 1 and page 2. And commit these changes.

Back on the master branch, edit the VERSION file and ensure that it is now set to version A.(B+1) and its greek is set to something like "a1".

Once everything is ready, push everything to github, as well as a tag on the new branch so that the nightly script (git describe) uses vA.B instead of dev as the basename.

$ git tag hwloc-A.B-branch vA.B
$ git push origin vA.B
$ git push origin hwloc-A.B-branch
$ git push origin master

Build Server

Hwloc's build server is: mtt.open-mpi.org. The user that is used to create nightly tarballs is "mpiteam". Contact Jeff Squyres or the IU representative if you need to get the password (this password is pretty tightly controlled; you'll need a very good reason to have it).

NOTE: Be warned that this account is also used to build all Open MPI projects and subprojects. Much of the infrastructure used for hwloc is also used by Open MPI. So please share nicely.

Here's what to setup:

  1. Login as mpiteam on mtt.open-mpi.org via ssh.
  2. Edit the file $HOME/scripts/hwloc-nightly-tarball.sh. Around line 23 is a list of the Git branches to build nightly tarballs for (it also specifies the order in which these tarballs are built). Add the basedir of your new release series (e.g., /branches/vA.B).
  3. Check to see if an environment modulefile exists of the specific tuple of Autotools that this hwloc series requires.
    1. Look in $HOME/modules/autotools for a modulefile named <ac_version>-<am_version>-<lt_version>-<m4_version>
    2. If it does not exist:
      1. Go to $HOME/local.
      2. Download the relevant Autotools tarballs from ftp://ftp.gnu.org/gnu/ that are not already there.
      3. Make the directory $HOME/local/autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>.
      4. Copy the script build.sh from one of the other autotools-<ac_version>-<am_version>-<lt_version>-<m4_version> directories into your new autotools-<ac_version>-<am_version>-<lt_version>-<m4_version> directory.
      5. Edit your new build.sh to have the relevant version numbers at the very top.
      6. Run your new autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>/build.sh script (from within that directory).
      7. This script will expand the tarballs, build them, and install them, and create a modulefile.
    3. At this point, the modulefile for your autotools tuple must exist.
    4. Sym link that modulefile to hwloc-vA.B in the $HOME/modules/autotools directory (such that you can "module use $HOME/modules; module load autotools/hwloc-vA.B" to get the Right autotools tuple.
  4. Run the $HOME/scripts/hwloc-nightly-tarball.sh script manually (it may take several minutes; there will be no stdout because this script is normally run via cron).
  5. If all goes well, you'll eventually get a mail sent to the hwloc-devel list saying that a snapshot tarball was created for your new release series.

NOTE: Running the hwloc-nightly-tarball.sh will copy the nightly tarball to the live web site tree. The nightly tarballs are not committed to the web site git repository (because they change too fast), but they are placed in the live tree itself at the location /l/osl/www/www.open-mpi.org/software/hwloc/nightly/vA.B/. If the directory for your new series does not exist, the script will create it. This will create a problem if you have not already created the software/hwloc/nightly/vA.B/ directory in hwloc's web tree Git repo, because when you do, running git pull on the live web tree will run into a conflict: Git will not create a directory that already exists. As such, if you run hwloc-nightly-tarball.sh before you update the live web tree with an Git-controlled software/hwloc/nightly/vA.B/ directory, you should remove the script-created directory, create the directory in Git, run git pull on the live web tree (or let cron do it) and then run the hwloc-nightly-tarball.sh script.

Also remember that the hwloc-nightly-tarball.sh script is run nightly via mpiteam's cron on mtt.open-mpi.org. So you might want to setup the script for your new release series on the same day that you update the web site. Otherwise, the script will run that night, create the directory, and then you'll have the Git directory conflict whenever you finally get to updating the web site.

The script should create a new tarball every night whenever necessary (i.e., there have been Git commits in the relevant Git branch since the last snapshot tarball was made). As such, if you run the hwloc-nightly-tarball.sh script twice in a row, it is likely that nothing will happen the second time (because there will have been no new Git commits and therefore no need to make a new tarball).

Web Site

The web site requires several additions for a new release series.

You usually do not do this step until you have some kind of tarball to release for the new release series (e.g., an "rc" tarball, or some other prerelease tarball).

First, get a clone of the Open MPI web site (because the hwloc tree is part of that web site) from https://github.com/open-mpi/ompi-www/.

NOTE it is a very large tree. As of August 2014, a Git clone of this tree takes about 4.3GB of disk space.

Post the Tarballs

  1. Go into the software/hwloc tree.
    1. Copy the prior release series tree into a new tree: cp vA.(B-1) vA.B (e.g., cp v1.0 v1.1). Don't bother copying the directory recursively (i.e., with the -r flag). If you do, you'll just need to remove a bunch of stuff in the new vA.B tree.
    2. Cd into the new vA.B tree.
    3. Edit the version.inc file and change all instances of vA_(B-1) to vA_B and all instances of vA.(B-1) to vA.B.
    4. Update the $release variable with the new release.
    5. Check index.php for things to update too.
    6. Make a downloads subdirectory and copy latest_snapshots.txt from another release, and update it.
    7. Use the ompi-script clone to run something like
dist/upload-release-to-s3.py --project hwloc --branch v1.11 --version 1.11.10 --date 2018/03/26 --files ~/hwloc-*1.11.10*
  1. If all that worked, you should be able to surf to your tree in software/hwloc/vA.B/ and see your release tarballs listed.
    • Double check that the download links work
    • Double check that the link to the Git NEWS file works

Update Nightly Snapshots

  1. Cd to the software/hwloc/nightly directory.
    1. Copy the prior release series tree into a new tree: cp vA.(B-1) vA.B (e.g., cp v1.0 v1.1).
    2. Edit the vA.B/index.php file and change all instances of vA_(B-1) to vA_B and all instances of vA.(B-1) to vA.B.
  2. Edit the software/hwloc/nightly/index.php file and add some text about your new series, including a link to the vA.B directory.
    • IMPORTANT: Use a trailing slash in the HREF when you link to a directory. E.g., <a href="vA.B/">version A.B</a>.

Post Doxygen Documentation

  1. Cd to the projects/hwloc/doc directory.
    1. Put PDFs from your release tarball into that directory, obeying the naming conventions: hwloc-vA.B[greek]-letter.pdf and <same>-a4.pdf.
    2. Make a new directory vA.B<greek> for the new HTML documentation.
    3. Expand your release tarball and run configure on it. Cd into the doc directory.
    4. IMPORTANT: Ensure to have a recent version of Doxygen before continuing.
      • I know that version 1.3.9.1 (i.e., what is on mtt.open-mpi.org) is too old.
      • Doxyen version 1.7.1 (on my Mac via ports) is new enough.
      • I don't know the minimum version that is necessary.
    5. IMPORTANT: You must do this from your expanded and configured tarball. Do NOT generate the doxygen docs from a Git checkout. The idea here is the GUARANTEE that the generated docs exactly match the release tarballs.
    6. Run doxygen www.open-mpi.org.cfg. This will generate a directory named www.open-mpi.org.
    7. Run make www-doc-sed to remove the spurious qualifiers (inline, unused, etc.) from the generated html files.
    8. Copy the entire contents of the www.open-mpi.org/html directory to projects/hwloc/doc/vA.B<greek>.
    9. Also copy the www.open-mpi.org.inc files from the expanded tarball's doc directory to projects/hwloc/doc/vA.B<greek>.
    10. Edit the projects/hwloc/doc/index.php file. Find the definition of the $snapv_dir and change it to be the directory of your hwloc software directory (e.g., "vA.B", or "v1.1").
      • Browse to projects/hwloc/doc/ and ensure that your prerelease PDF and HTML links show up and work properly.
      • Browse to projects/hwloc/doc/vA.B/ and ensure that the HTML documentation shows up properly.

Update Left-Hand Navigation

  1. Edit the projects/hwloc/nav.inc file.
  2. Add a line for the new series. You might want to label it "(prerelease)", or somesuch.

Check Everything

  1. Browse all the pages that you just added and ensure that they are all working properly, including all links.

Git Commit

  1. Make sure to git add all new files.
  2. Then Git commit and push to Github when you're sure that everything is correct.
  3. REMEMBER:
    • Several sites around the world mirror the entire Open MPI site and synchronize via Git. We have no control over when then run git pull to update their mirror.
    • Hence, as soon as you commit, some mirror somewhere may run git pull immediately and see your new changes. So treat git pushing as "make it immediately live on the web site".
    • The main Open MPI web site runs git pull every hour or so. So even if your changes aren't immediately visible on the main site, check back in an hour.

Make a New Hwloc (Pre)Release on an Existing Release Series

This is the case where you are making a release on a release series that has already been created (see below). Hence, all the infrastructure for the release series is already in place and you just need to make a new point release (potentially even the first formal release of the series, version A.B.C, where C==0, because the release series was created with a pre-release such as an alpha, beta, or rc tarball).

Make the Tarballs

  1. Edit the VERSION file and change at least 2 things:
    • Set the greek value to something relevant (e.g., "b1" or "rc1" or whatever -- keep bumping the number as you make successive pre-releases towards the final release).
    • Set the .so library version to the appropriate value (see the Libtool documentation for instructions: page 1 and page 2
    • Also update the HWLOC_VERSION in contrib/windows/private_config.h
    • You must commit the modified VERSION values back to Git
  2. NOTE: Each hwloc release branch requires a specific tuple of Autoconf/Automake/Libtool. You are strongly discouraged from changing this tuple for the life of a given series. The idea is to keep the build system fairly constant for the life of a series and only change the tuple at the next release series.
  3. Double check your Git tree and ensure that "git status" returns no locally-modified files.
  4. From the top-level source directory, run contrib/dist/make_dist_tarball.
    • The process take anywhere from 10 to 30 minutes, depending on your machine.
    • You will need to have a recent version of doxygen installed.

If the process succeeds, you'll have 4 tarballs:

  • tar.gz and tar.bz2 for the pre-release/greek version.
  • tar.gz and tar.bz2 for the final version (i.e., exactly the same, but with the greek value set to the empty string).

The rationale is that you can test the greek tarballs, and if they're good, you have the final tarballs ready to ship. They're guaranteed to be exactly the same because they were all made at the same time.

Web Site

There are several steps to updating the web site.

Post the Tarballs

  1. Get a clone of the Open MPI web site. See instructions about how to do this, above.
  2. Update software/hwloc/vVERSION/version.inc
  3. Use the ompi-script clone to run something like
dist/upload-release-to-s3.py --project hwloc --branch v1.11 --version 1.11.10 --date 2018/03/26 --files ~/hwloc-*1.11.10*

Post Doxygen Documentation

Follow the same steps as above: [#PostDoxygenDocumentation for posting the Doxygen documentation]. If you are releasing a production version, <greek> will be empty.

Additionally, if this is the first production release, then Git remove all the pre-release documentation directories. We do not continue hosting pre-release documentation after a final release is made.

Update Left-Hand Navigation

  • If this was the first production release of a series:
    • Set the $dir variable in software/hwloc/current/version.inc to point to the directory of the new release series. E.g., "v1.1".
    • Edit the projects/hwloc/nav.inc file.
    • Label the new series as "(stable)".
    • Update the labels on the prior series to indicate that they are "(older)", or "(ancient)", or whatever is relevant.
  • If this is not a pre-release
    • Edit projects/hwloc/doc/index.php
    • Add a new link for the new release series.

Check Everything

Follow the same steps as above for checking everything.

Git Commit

Once you have Git removed all relevant files and/or directories, follow the same steps as above to commit all the web site changes back to Github.

Git

If this was a production release, then the release needs to be tagged in Git and the VERSION file needs to be updated.

For example:

$ git tag -a hwloc-A.B.C

where "A.B.C" is obviously the numbers that are relevant for your new release series (e.g., "1.1.0"). Note that we include the C value in the tag, even if it's zero.

NOTE: Be sure to use the -a option so that an annotated Git tag is created. See git-tag(1) for more details.

Once you have made this tag, edit the VERSION file on the branch and increment the version number so that it is now "A.(B+1).0a1". This ensures that we don't make new nightly snapshot tarballs that have the "A.B" version number. Specifically, version A.B has been released -- so we shouldn't make any more snapshot tarballs with that version number.

Clone this wiki locally