Skip to content

MakingANewRelease

Jump to bottom
Jeff Squyres edited this page Sep 3, 2014 · 13 revisions

How to Make a New Hwloc Release

There are several steps that are required to make a new hwloc release/prerelease and/or release series.

There are two cases:

  1. You are making a release on a brand-new release series (it's probably a pre-release, like an alpha, beta, or RC -- but hey, it could be a final release, too).
  2. You are making a release on an existing release series -- meaning that you've already followed the previous steps for initially creating the release series; now you're just putting a new release out on that existing series.

Announces

We usually announce the release to the following mailing lists:

Make a New Hwloc Release Series

This is the case where you need to make a release for version A.B.C where C==0 -- the release is probably a pre-release of some kind (e.g., an alpha, beta, or rc tarball). None of the infrastructure yet exists to make this new series. There are several systems that are affected:

  1. Build system
  2. Git
  3. Nightly / official tarball build server
  4. Web site

Let's go over each of these in order.

Build System

Before making a branch for the new release series, it's usually a good idea to evaluate if upgrading to newer versions of the GNU Autotools is a good idea. Specifically: we make an effort to pick a tuple of Autotools for use with a release series and try very hard to use those same exact versions for every official release in that series.

If you end up upgrading the version of Autotools that you're using, it is best to edit source:/contrib/dist/make_dist_tarball (on the trunk -- before you make the Git branch for the new series) to set these versions. Run the script to ensure that these versions work properly with the trunk. Tune up anything that you need on the trunk to make these versions work. Once it's all working, commit the new version of make_dist_tarball with those version numbers back to the trunk.

API Version

If the API has changed, increasing HWLOC_API_VERSION in hwloc.h is usually a good idea. This should only be needed for major releases. Do it in trunk before branching.

Git

The main thing to do is to create a release branch in Git. Once you get the trunk in a good state, branch it. Note that there is a naming convention for branches:

$ git branch vA.B

where "A.B" is obviously the numbers that are relevant for your new release series (e.g., "1.1").

Once you make this branch, check it out and edit the VERSION file. Ensure that its VERSION numbers are set to A.B and the greek is set to something relevant (e.g., "a1" or "rc1").

Set the .so library version to the appropriate value (see the Libtool documentation for instructions: page 1 and page 2. And commit these changes.

Back on the trunk, edit the VERSION file and ensure that it is now set to version A.(B+1) and its greek is set to something like "a1".

Build Server

Hwloc's build server is: mtt.open-mpi.org. The user that is used to create nightly tarballs is "mpiteam". Contact Jeff Squyres or the IU representative if you need to get the password (this password is pretty tightly controlled; you'll need a very good reason to have it).

NOTE: Be warned that this account is also used to build all Open MPI projects and subprojects. Much of the infrastructure used for hwloc is also used by Open MPI. So please share nicely.

Here's what to setup:

  1. Login as mpiteam on mtt.open-mpi.org via ssh.
  2. Edit the file $HOME/scripts/hwloc-nightly-tarball.sh. Around line 23 is a list of the Git branches to build nightly tarballs for (it also specifies the order in which these tarballs are built). Add the basedir of your new release series (e.g., /branches/vA.B).
  3. Check to see if an environment modulefile exists of the specific tuple of Autotools that this hwloc series requires.
    1. Look in $HOME/modules/autotools for a modulefile named <ac_version>-<am_version>-<lt_version>-<m4_version>
    2. If it does not exist:
      1. Go to $HOME/local.
      2. Download the relevant Autotools tarballs from ftp://ftp.gnu.org/gnu/ that are not already there.
      3. Make the directory $HOME/local/autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>.
      4. Copy the script build.sh from one of the other autotools-<ac_version>-<am_version>-<lt_version>-<m4_version> directories into your new autotools-<ac_version>-<am_version>-<lt_version>-<m4_version> directory.
      5. Edit your new build.sh to have the relevant version numbers at the very top.
      6. Run your new autotools-<ac_version>-<am_version>-<lt_version>-<m4_version>/build.sh script (from within that directory).
      7. This script will expand the tarballs, build them, and install them, and create a modulefile.
    3. At this point, the modulefile for your autotools tuple must exist.
    4. Sym link that modulefile to hwloc-vA.B in the $HOME/modules/autotools directory (such that you can "module use $HOME/modules; module load autotools/hwloc-vA.B" to get the Right autotools tuple.
  4. Run the $HOME/scripts/hwloc-nightly-tarball.sh script manually (it may take several minutes; there will be no stdout because this script is normally run via cron).
  5. If all goes well, you'll eventually get a mail sent to the hwloc-devel list saying that a snapshot tarball was created for your new release series.

NOTE: Running the hwloc-nightly-tarball.sh will copy the nightly tarball to the live web site tree. The nightly tarballs are not committed to the web site git repository (because they change too fast), but they are placed in the live tree itself at the location /l/osl/www/www.open-mpi.org/software/hwloc/nightly/vA.B/. If the directory for your new series does not exist, the script will create it. This will create a problem if you have not already created the software/hwloc/nightly/vA.B/ directory in hwloc's web tree Git repo, because when you do, running git pull on the live web tree will run into a conflict: Git will not create a directory that already exists. As such, if you run hwloc-nightly-tarball.sh before you update the live web tree with an Git-controlled software/hwloc/nightly/vA.B/ directory, you should remove the script-created directory, create the directory in Git, run git pull on the live web tree (or let cron do it) and then run the hwloc-nightly-tarball.sh script.

Also remember that the hwloc-nightly-tarball.sh script is run nightly via mpiteam's cron on mtt.open-mpi.org. So you might want to setup the script for your new release series on the same day that you update the web site. Otherwise, the script will run that night, create the directory, and then you'll have the Git directory conflict whenever you finally get to updating the web site.

The script should create a new tarball every night whenever necessary (i.e., there have been Git commits in the relevant Git branch since the last snapshot tarball was made). As such, if you run the hwloc-nightly-tarball.sh script twice in a row, it is likely that nothing will happen the second time (because there will have been no new Git commits and therefore no need to make a new tarball).

Web Site

The web site requires several additions for a new release series.

You usually do not do this step until you have some kind of tarball to release for the new release series (e.g., an "rc" tarball, or some other prerelease tarball).

First, get a clone of the Open MPI web site (because the hwloc tree is part of that web site) from https://github.com/open-mpi/ompi-www/.

NOTE it is a very large tree. As of August 2014, a Git clone of this tree takes about 4.3GB of disk space.

Post the Tarballs

  1. Go into the software/hwloc tree.
    1. Copy the prior release series tree into a new tree: cp vA.(B-1) vA.B (e.g., cp v1.0 v1.1). Don't bother copying the directory recursively (i.e., with the -r flag). If you do, you'll just need to remove a bunch of stuff in the new vA.B tree.
    2. Cd into the new vA.B tree.
    3. Edit the version.inc file and change all instances of vA_(B-1) to vA_B and all instances of vA.(B-1) to vA.B.
    4. Edit index.php.
      1. Do the same version substitution that you did for version.inc.
      2. Comment out the line defining the $t_stable and $t_older. Commenting out these lines will prevent showing the grid of the stable release and older releases. It is not necessary to comment out any of the other PHP code.
    5. Make a downloads subdirectory and cd into it.
      1. If you recursively copied from a prior release version, remove all the release files (tarballs, SRPMs, etc.) from the downloads directory.
      2. Place your vA.B release (or prerelease) tarballs/SRPMs/whatever into this downloads directory.
      3. Make a file named latest_snapshot.txt and put in just the version number of the tarballs you just put there. For example, if you have a vA.Brc1 tarball, put "A.Brc1" in the latest_snapshot.txt file.
      4. Generate MD5 and SHA1 sums on your tarballs:
        • csh syntax: sha1sum hwloc-* >! sha1sums.txt; md5sum hwloc-* >! md5sums.txt
        • sh syntax: `sha1sum hwloc-* > sha1sums.txt; md5sum hwloc-*

          md5sums.txt`

    6. If all that worked, you should be able to surf to your tree in software/hwloc/vA.B/ and see your release tarballs listed.
      • Double check that the download links work
      • Double check that the link to the Git NEWS file works

Update Nightly Snapshots

  1. Cd to the software/hwloc/nightly directory.
    1. Copy the prior release series tree into a new tree: cp vA.(B-1) vA.B (e.g., cp v1.0 v1.1).
    2. Edit the vA.B/index.php file and change all instances of vA_(B-1) to vA_B and all instances of vA.(B-1) to vA.B.
  2. Edit the software/hwloc/nightly/index.php file and add some text about your new series, including a link to the vA.B directory.
    • IMPORTANT: Use a trailing slash in the HREF when you link to a directory. E.g., <a href="vA.B/">version A.B</a>.

Post Doxygen Documentation

  1. Cd to the projects/hwloc/doc directory.
    1. Put PDFs from your release tarball into that directory, obeying the naming conventions: hwloc-vA.B[greek]-letter.pdf and <same>-a4.pdf.
    2. Make a new directory vA.B<greek> for the new HTML documentation.
    3. Expand your release tarball and run configure on it. Cd into the doc directory.
    4. IMPORTANT: Ensure to have a recent version of Doxygen before continuing.
      • I know that version 1.3.9.1 (i.e., what is on mtt.open-mpi.org) is too old.
      • Doxyen version 1.7.1 (on my Mac via ports) is new enough.
      • I don't know the minimum version that is necessary.
    5. IMPORTANT: You must do this from your expanded and configured tarball. Do NOT generate the doxygen docs from a Git checkout. The idea here is the GUARANTEE that the generated docs exactly match the release tarballs.
    6. Run doxygen www.open-mpi.org.cfg. This will generate a directory named www.open-mpi.org.
    7. Run make www-doc-sed to remove the spurious qualifiers (inline, unused, etc.) from the generated html files.
    8. Copy the entire contents of the www.open-mpi.org/html directory to projects/hwloc/doc/vA.B<greek>.
    9. Also copy the www.open-mpi.org.inc files from the expanded tarball's doc directory to projects/hwloc/doc/vA.B<greek>.
    10. Edit the projects/hwloc/doc/index.php file. Find the definition of the $snapv_dir and change it to be the directory of your hwloc software directory (e.g., "vA.B", or "v1.1").
      • Browse to projects/hwloc/doc/ and ensure that your prerelease PDF and HTML links show up and work properly.
      • Browse to projects/hwloc/doc/vA.B/ and ensure that the HTML documentation shows up properly.

Update Left-Hand Navigation

  1. Edit the projects/hwloc/nav.inc file.
  2. Add a line for the new series. You might want to label it "(prerelease)", or somesuch.

Check Everything

  1. Browse all the pages that you just added and ensure that they are all working properly, including all links.

Git Commit

  1. Make sure to git add all new files.
  2. Then Git commit and push to Github when you're sure that everything is correct.
  3. REMEMBER:
    • Several sites around the world mirror the entire Open MPI site and synchronize via Git. We have no control over when then run git pull to update their mirror.
    • Hence, as soon as you commit, some mirror somewhere may run git pull immediately and see your new changes. So treat git pushing as "make it immediately live on the web site".
    • The main Open MPI web site runs git pull every hour or so. So even if your changes aren't immediately visible on the main site, check back in an hour.

Make a New Hwloc (Pre)Release on an Existing Release Series

This is the case where you are making a release on a release series that has already been created (see below). Hence, all the infrastructure for the release series is already in place and you just need to make a new point release (potentially even the first formal release of the series, version A.B.C, where C==0, because the release series was created with a pre-release such as an alpha, beta, or rc tarball).

Make the Tarballs

  1. Edit the VERSION file and change at least 2 things:
    • Set the greek value to something relevant (e.g., "b1" or "rc1" or whatever -- keep bumping the number as you make successive pre-releases towards the final release).
    • Set the .so library version to the appropriate value (see the Libtool documentation for instructions: page 1 and page 2
    • You must commit the modified VERSION values back to Git
  2. NOTE: Each hwloc release branch requires a specific tuple of Autoconf/Automake/Libtool. You are strongly discouraged from changing this tuple for the life of a given series. The idea is to keep the build system fairly constant for the life of a series and only change the tuple at the next release series.
  3. Double check your Git tree and ensure that "git status" returns no locally-modified files.
  4. From the top-level source directory, run contrib/dist/make_dist_tarball.
    • The process take anywhere from 10 to 30 minutes, depending on your machine.
    • You will need to have a recent version of doxygen installed.

If the process succeeds, you'll have 4 tarballs:

  • tar.gz and tar.bz2 for the pre-release/greek version.
  • tar.gz and tar.bz2 for the final version (i.e., exactly the same, but with the greek value set to the empty string).

The rationale is that you can test the greek tarballs, and if they're good, you have the final tarballs ready to ship. They're guaranteed to be exactly the same because they were all made at the same time.

Web Site

There are several steps to updating the web site.

Post the Tarballs

  1. Get a clone of the Open MPI web site. See instructions about how to do this, above.
  2. Place the tarballs in software/hwloc/vVERSION/downloads/ and Git add them (don't bother committing yet; there's a bunch more Git changes coming).
  3. Get a Windows build of the same tarball; place it in the same directory and Git add it.
    • Samuel has made all of the Windows builds; I'm not sure of the process he follows to make them.
  4. Re-generate the md5sums.txt and sha1sums.txt:
    • md5sum hwloc-* >! md5sums.txt
    • sha1sum hwloc-* >! sha1sums.txt
  5. Edit latest-snapshot.txt to reflect the latest release number (e.g., "1.0.3rc3", or "1.0.4", or whatever version number is relevant).
  6. Edit software/hwloc/vVERSION/index.php:
    • If this is a pre-release (this step is unnecessary in v1.3 and beyond; the PHP has been made smarter to just see if last_snapshot.txt contains strings like "rc", and will automatically show the pre-release if so):
      • Ensure $t_prerelease is uncommented in software/hwloc/downloads/vVERSION/index.php.
      • That should be sufficient for the PHP to read the latest-snapshot.txt file, see that it's a pre-release, and add the appropriate pre-release box / download links on the web page.
    • If this is not a pre-release:
      • Edit software/hwloc/vVERSION/version.inc. Set the $ver_vX_Y variable to the version. E.g., "1.0" or "1.1.3", or whatever is relevant.
      • Git remove all pre-release tarballs and zip files. We do not continue hosting pre-release versions after a final release is made.
      • Edit software/hwloc/download/vVERSION/index.php:
        • Comment out $t_prerelease. This is sufficient to remove the "pre-release" box / download links from the web page.
        • If this is the first production release of the series:
          • Ensure $t_stable is uncommented.
          • Ensure $t_older is commented out.
        • If this is not the first production release of the series:
          • Ensure $t_older is uncommented.
          • Expand the array of version numbers in $t_older to include all old version numbers.

Post Doxygen Documentation

Follow the same steps as above: [#PostDoxygenDocumentation for posting the Doxygen documentation]. If you are releasing a production version, <greek> will be empty.

Additionally, if this is the first production release, then Git remove all the pre-release documentation directories. We do not continue hosting pre-release documentation after a final release is made.

Update Left-Hand Navigation

  • If this was the first production release of a series:
    • Set the $dir variable in software/hwloc/current/version.inc to point to the directory of the new release series. E.g., "v1.1".
    • Edit the projects/hwloc/nav.inc file.
    • Label the new series as "(stable)".
    • Update the labels on the prior series to indicate that they are "(older)", or "(ancient)", or whatever is relevant.
  • If this is not a pre-release
    • Edit projects/hwloc/doc/index.php
    • Add a new link for the new release series.

Check Everything

Follow the same steps as above for checking everything.

Git Commit

Once you have Git removed all relevant files and/or directories, follow the same steps as above to commit all the web site changes back to Github.

Git

If this was a production release, then the release needs to be tagged in Git and the VERSION file needs to be updated.

For example:

$ git tag -a hwloc-A.B.C

where "A.B.C" is obviously the numbers that are relevant for your new release series (e.g., "1.1.0"). Note that we include the C value in the tag, even if it's zero.

Once you have made this tag, edit the VERSION file on the branch and increment the version number so that it is now "A.(B+1).0a1". This ensures that we don't make new nightly snapshot tarballs that have the "A.B" version number. Specifically, version A.B has been released -- so we shouldn't make any more snapshot tarballs with that version number.

Clone this wiki locally