REP: 132 Title: Incorporation of Changelogs into Package Source Tree Author: Tully Foote, Thibault Kruse, Mirza Shah, Dirk Thomas, William Woodall Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 22-Jan-2013 Post-History: 10-April-2013
This REP describes incorporating package changelogs (i.e. a list of changes made to the package in each release) as part of the package source tree rather than being maintained separately on the ROS wiki. This is to address shortcomings with maintaining a separate changelog list.
In ROS, package changelogs have traditionally been maintained as their own page within a package's ROS wiki page. These changelogs have had to be maintained separately from their code bases. This approach has had some drawbacks:
- Hard to Find - As changelogs are isolated from code, they can be difficult to find and correlate with particular versions of code. By making them part of the source tree, you can determine what has changed by inspecting the package itself.
- Lack of Maintenance - As changelogs are isolated from code and are optional, the motivation to maintain the lists is very weak. The result is that only a minority of package maintainers keep good changelogs. Especially due to packaging packages instead of stacks the number of changelogs which need to be maintained has increased. The hope is that when changelogs are kept with the package source, developers will be more likely to update the changelog file.
- Syntactic Inconsistency - Even among well-maintained changelogs there is an inconsistency in the layout of these lists across the ROS wiki. The lack of consistency makes it harder for both people and machines to parse the data in these logs. This is an opportunity to constrain the structural and syntactic format of the changelogs so that they are consistent across packages.
- Needed for Packaging - Virtually all package formats including .deb and .rpm require changelogs as part of the package. Scraping this information from the wiki is not feasible and therefore not performed. A changelog file with a specified parseable format can be automatically converted into these package formats.
The format of the changelog should have the following properties:
- In Source - The changelog is part of the package source rather than its own wiki page.
- Wiki Integration - The changelog is displayed on the wiki as in the past, but now pulled from SCM rather than being directly edited -- just like package wiki headers.
- Syntax - Simple, easily parseable syntax.
- Packaging Compatible - The changelog, along with information from the package.xml, should satisfy the required fields for .deb/.rpm changelogs.
- Non-redundant - Does not contain redundant information found elsewhere such as package.xml.
- Optional but Recommended - Should not be required, but highly recommended with tools such as bloom giving warnings when the changelog is missing.
- Generated Documentation - It is good practice to also include changelogs in generated static documentation. As we use Sphinx, a format that it can process would be ideal.
- GitHub Integration - Since ROS packages are being moved to GitHub (and GitHub practices are copied by other hosting solutions), it would be good to comply with GitHub display of files. [5]
- Flexibility - While standardized syntax is preferred, changelogs should be as flexible as is reasonably possible.
- Changelogs will be written in ReStructuredText (RST)[4]_ with additional constraints.
- Each package (catkin package, containing a package.xml) may contain a single CHANGELOG.rst file used to list changelog entries.
- The CHANGELOG.rst file for a package must be a peer of the package.xml for that package, i.e. in the root of the package directory.
- The CHANGELOG.rst file may contain a RST section for each version of the package released. The sections are detected based on a section title format convention. The sections should be listed in descending version order by convention, but the ordering will not be enforced.
- The RST section title for each version will follow the required format or it will be ignored. The format will simply be
<version> (<timestamp>), where both the canonical version (major.minor.patch) and timestamp are required.
The changelog document must be valid RST. The document will be processed by an RST parser (ignoring file inclusion directives, as those also do not work on GitHub).
The changelog file will use one section without subsections for each changelog entry, and within that other elements from the RST specification [4] are allowed:
- paragraphs
- transitions
- blank lines
- references
- comments
- bullet lists
- enumeration lists
- inline markup
- directives (may be invisible in processed changelog)
Inline markup may not be preserved when transforming the log entries for deb/rpm format. The following elements of the RST specification [4] are explicitly forbidden in the version sections of the changelog:
- definition lists
- field lists
- option Lists
- literal blocks
- line blocks
- block quotes
- doctest blocks
- tables
- subsections
A changelog entry is defined by a heading which contains a version number followed by the timestamp in parenthesis.
The version number consists of three positive integers separate by single dots, e.g. 1.2.3 as specified in [8].
The timestamp must be parseable by the Python dateutil module, i.e. dateutil.parser.parse(...).
The timestamp should at least contain a full date (YYYY-MM-DD, ISO 8601 format).
The rest of the elements in the timestamp are optional, but should be included in order, e.g. you should not specify seconds without minutes and hours. Additional elements are: hours and minutes (together), seconds, and timezone offset.
A changelog version section may not contain subsections, but it may be a subsection itself.
The following example shows a few valid changelog entries:
1.2.3 (2013-01-23) ------------------ * Change 1 Optional Text . . . * Change n Optional Text 1.2.2 (2013-01-22 08:35) ------------------------ * Change 1 * Change 2 1.2.1 (2013-01-22 06:35 +0100) ------------------------------ * Change
This is a standard example, but more variations are possible with the allowed RST elements. The following example uses a lot of variations to show what possibilities exist:
^^^^^^^^^^^^^^^^^^^^^^^^^ Changelog for package foo ^^^^^^^^^^^^^^^^^^^^^^^^^ 0.1 === Free form text about this minor release. 0.1.27 (forthcoming) -------------------- * Great new feature 0.1.26 (2012-12-26) ------------------- * Utilizes caching to improve query performance (fix https://github.com/ros/ros_comm/pull/2) * Simplified API calls based on (https://github.com/ros/robot_model) Note that these changes are based on REP 192 * Fixed synchronization issue on startup .. not mentioning secret feature on purpose 0.1.25 (2012-11-25) ------------------- - Added thread safety - Replaced custom XML parser with `TinyXML <http://www.grinninglizard.com/tinyxml/>`_. - Fixed regression introduced in 0.1.22 ---- The library should now compile under Win32 0.1.0 (2012-10-01) ------------------ First public *stable* release 0.0 === 0.0.1 (2012-01-31) ------------------ 1. Initial release 2. Initial bugs
In the above example, the header (enclosed by ^ characters) and the sections with only MAJOR.MINOR are additional visual markup to make the changelog more readable / structure for humans to read, but will be ignored by tools which process the changelogs, e.g. bloom will not insert these elements into the debian changelog file.
Similarly the section 0.1.27 (forthcoming) does not match the format of the version section title, so it ignored when being parsed by tools like bloom. This is recommended practice to state changes which are already committed but not yet released.
The ROS wiki shall have a macro which users can point directly to the URL of the CHANGELOG.rst file. The ROS wiki will also have the changelog automatically embedded by rosdoc-lite when the package is parsed by the documentation indexer, such that the macro should only be needed for packages which are not indexed by the documentation system.
There are several rules which are good ideas and strongly encouraged, but either shouldn't be or cannot be enforced.
One rule is to have a valid version section for each version of a package released. For example, it is considered bad form to have changelog sections for 1.0.0 and 1.0.1 but not 1.0.2. This is confusing to anyone who is reading the raw CHANGELOG.rst file and requires tools like rosdoc-lite and bloom to fill the gap. This rule is highly recommended and if the developer chooses to skip a version, the release tools will protest, but allow it. This rule comes up because packages which share a single source repository must maintain the same version, so when changes in one package cause the version to increase in its peers, the peers may have no new changes to report. In this case the recommendation is to have an empty section like this:
0.3.5 (2013-05-09 16:36:55 -0700) --------------------------------- 0.3.4 (2013-04-09 16:36:55 -0700) --------------------------------- - More changes - Changed a thing
Alternatively you can give a simple message why this section is blank:
0.3.5 (2013-05-09 16:36:55 -0700) --------------------------------- - Released along with other packages, no changes 0.3.4 (2013-04-09 16:36:55 -0700) --------------------------------- - More changes - Changed a thing
This rule is a "nice to have" rule which comes up when someone other than the normal maintainer releases a package. To clarify this is the person who: updated the version in the package.xml, updated the CHANGELOG.rst (adding the date of the release and ensuring the log is complete), and tagged the version in the VCS. This is not neccissarily the person who ran the package through the "release pipeline" using tools like bloom.
This is informal and not required at all, the entry might look like this:
0.3.4 (2013-04-09 16:36:55 -0700) --------------------------------- - Released by: Sally <[email protected]> - More changes - Changed a thing
This is really a convenience for people reading the CHANGELOG.rst because this information can be obtained by the information in the VCS history of the package. It is a good idea to include this information if someone other than one of the normal maintainers (listed in the package.xml) released the package.
This rule comes up when multiple people are maintaining a single package. The idea is to "tag" some or all of the changes in the CHANGELOG.rst with authors of those changes. It would look something like this:
0.3.4 (2013-04-09 16:36:55 -0700) --------------------------------- - More changes @steve - Changed a thing - John Doe <[email protected]>
The format of the "tagging" is not formal and is in no way required or utilized by ROS infrastructure tools. This recommendation is just a "nice to have" as all of this information is obtainable from the VCS using tools like git annotate or git blame.
The proposed format has the following properties that help to meet the design requirements:
- Changelogs will be in-source while remaining optional.
- Wiki integration is simple to realize.
- Simple markup and very similar to how changelogs are typically written on the wiki and other open source projects.
- Can reuse RST parsers. See [6], [9]
- Can be embedded in sphinx docs via include directive.
- When combined with the corresponding package.xml, enough information is provided to meet the full requirements of .deb and .rpm changelog formats (timestamp, package name, etc...).
- Minimal redundant information from package.xml
Concerns have been discussed on ros-developers ([3]) and in the Buildsystem SIG ([7]).
- Can the timestamp of a changelog entry be optional?
While the toolchain could use the current time when a release is made the information is missing when reading the changelog in the source repository. Without a timestamp it is also not clear if the version has already been release (but the maintainer did not provided a timestamp) or is forthcoming (where the maintainer did not add an annotation for that).
- Can we allow free form text in the changelog entry headline?
This would make the decision if a headline is a valid changelog entry more difficult. On the other hand free form text could be either placed before a changelog entry or inside the changelog entry which should be sufficient to add additional information. Therefore the specification does not allow that.
How can a full changelog with multiple versions be generated?
The available information from the current package.xml and changelog file can be used to generate the changelog for the current version. Older changelog can not be generated since the information from the package.xml file at that point in time might have been different (i.e. other maintainer). Anyway a full changelog can be constructed based on the changelog of the previous changelog and the changelog of the current version if desired.
How to link to tickets/issues in bug tracker without having to give full URL?
Would be nice if GitHub did this for us on their website, but currently it does not.
- How much of RST should be supported?
- Outside section entries, no reason to forbid full RST
- Inside section entries, we only want to support things that can easily be transformed into deb/rpm format, though some loss of quality might be acceptable. Things to consider:
- Substitutions http://docutils.sourceforge.net/docs/ref/rst/directives.html#replacement-text
- References http://docutils.sourceforge.net/docs/ref/rst/directives.html#references
- Inclusion of other files (disabled on GitHub)
- Nested lists
- Definition lists (could also be used for version!)
- Directives, such as . note:: foo
REP now states some definitely allowed and forbidden elements. More may be allowed if users demand that and they can be easily supported.
- Other markup language support. See [5]
Not urgent, leave out for now.
- Name and placement
An early suggestion "ChangeList.txt" was rejected due to similarity to CMake "CMakeLists.txt".
The RST extension makes it possible for GitHub to render the file, and allows us to later possibly also support other markup flavors.
The package root is a common default way for such meta information, a "doc" sub folder is useful for static documentation. Sphinx does not allow to refer to documents outside the doc folder via toc-trees, but it does allow inclusion of files like this:
.. include:: ../CHANGELOG.rstSo we went for CHANGELOG.rst in root as ideal place. Alternatives are not planned to have a single location to check for the existence of a changelog.
- README.rst fall back: When users have a small package, it may be more convenient to put changelog into the README.rst. Could changelog tooling(bloom) fall back to try README.rst for changelog entries?
The prototype library could handle such complex README files. Though no technical reason is known that would prevent this, there was too much doubt on possible unknown problems with that approach, and user confusion over multiple alternatives, so for now it was decided to not go ahead with this.
- inline markup transformation rules: When creating deb/rpm changelogs from RST, a problem is how to deal with unicode and complex inline markup. Alternatives:
- Forbid all inline markup
- Support some inline markup nicely, forbid all that we do not transform
- Support some inline markup nicely, treat other markup as raw source
- Support all inline markup nicely
The actual transformations to happen are for other tools to decide. For now, we shall support some markup nicely (references), and treat other markup as raw source.
- Wiki display: We could display the changelog in the wiki as raw text, try to render the RST, display what goes into the deb, or merely link to the source file in its home repo.
- raw display is quickest for the users and easiest for us, maybe
- rendered display is nicer to the eye, allows following embedded references
- link to the source location is a bit worse for the users (navigating separate sites, but may be least effort)
- rosbuild stacks and package support?
For rosbuild stacks the CHANGELOG.rst file could be placed beside the stack.xml file. However, that won't be a priority to implement in the near future and might require contributions from the community.
For reference, here are the changelog formats for .deb [1] and .rpm [2] packages. Both package formats expect a changelog as prerequisite to creating a package.
package (version) distribution(s); urgency=urgency
[optional blank line(s), stripped]
* change details
more change details
[blank line(s), included in output of dpkg-parsechangelog]
* even more change details
[optional blank line(s), stripped]
-- maintainer name <email address>[two spaces] date
The Debian Policy manual [1] goes further to describe the maintainer as:
The maintainer name and email address used in the changelog should be the details of the person uploading this version. They are not necessarily those of the usual package maintainer.
* Fri Jun 23 2006 Jesse Keating <[email protected]> - 0.6-4 - And fix the link syntax. * Fri Jun 23 2006 Jesse Keating <[email protected]> 0.6-4 - And fix the link syntax. * Fri Jun 23 2006 Jesse Keating <[email protected]> - 0.6-4 - And fix the link syntax. * Wed Jun 14 2003 Joe Packager <joe at gmail.com> - 1.0-2 - Added README file (#42).
A prototype implementation of a library that parses any RST document and extracts changelog entries as described here is provided as ongoing effort here [6].
Since then an official implementation has been merged into catkin_pkg which will be used by rosdoc-lite and bloom: [9]
| [1] | (1, 2) Debian Package Changelog Requirements (http://www.debian.org/doc/debian-policy/ch-source.html) |
| [2] | Fedora RPM Package Changelog Requirements (http://fedoraproject.org/wiki/Packaging:Guidelines#Changelogs) |
| [3] | Tully Foote Proposal for Stack Changelogs (9-03-2010) (http://code.ros.org/lurker/message/20100903.213420.d959fddc.en.html) |
| [4] | (1, 2) reStructuredText (RST) (http://docutils.sourceforge.net/rst.html) |
| [5] | (1, 2) Github Markup languages (https://github.com/github/markup) |
| [6] | (1, 2) Prototype python script (https://github.com/tkruse/changelog_rst.git) |
| [7] | Buildsystem SIG discussion (https://groups.google.com/d/msg/ros-sig-buildsystem/L3nE9X0T2Jk/ML_1JsHLuF0J) |
| [8] | REP 127 Specification of package manifest format (https://github.com/ros-infrastructure/rep/blob/master/rep-0127.rst#version) |
| [9] | (1, 2) Implementation in catkin_pkg (https://github.com/ros-infrastructure/catkin_pkg/blob/master/src/catkin_pkg/metapackage.py) |
This document has been placed in the public domain.