Report on Plone Upgrades for EDRN

This document represents the results of experimentation and research conducted from late November through December 2018.

Background

Plone is an enterprise content management system (CMS) based on the Zope application server, both written in a mixture of the Python and C programming languages. Plone is considered widely to be the most robust and secure of the enterprise CMSs, but also one with the steepest of learning curves in terms of customization. Customization of a CMS typically refers to the addition of new content types and publication states.

Basic content types include the familiar page (a collection of HTML), image (JPEG, PNG, and so forth), news item (headline, lead image, body text), event (start and end date and time, location, detail text), and so forth. Custom content types include anything in the application domain.

Enterprise CMSs are distinguished in that given the schema for a content type, they can generate database schema, add & edit forms, validation, and so forth. Plone is no exception. Oddly, we hardly leverage this feature.

Additional integrations for enterprise CMSs include RSS syndication, faceted navigation, IFTTT integration, CAPTCHA support, social media integration, LDAP authentication, and so forth.

Content Types and LDAP

Two of the most important customizations and integrations the Informatics Center has leveraged in support of our customer, NCI, include the creation of custom content types and the integration of LDAP for authentication and group membership. Plone, as an enterprise CMS, supports both of these.

For content types, Plone has two implementations:

• Archetypes. Archetypes appeared in Plone 2.1 and was the preferred content type framework until Plone 4. ATContentTypes is a package of content types based on Archetypes that include the familiar types: Page, Image, News Item, Event, and so forth.
• Dexterity. Dexterity appeared in Plone 4 and was to replace Archetypes moving forward as it was easier to use and allowed end-users to design content types through-the-web, coding optional.

The Plone team understands that the transition between Archetypes and Dexterity can be difficult, which is why Dexterity first appeared in Plone 4 and became official in Plone 5, while Archetypes is "supported" in Plone 5 but will be phased out entirely by Plone 6.

For LDAP integration, Plone also has two implementations:

• plone.app.ldap. This integration provides LDAP and Active Directory lookup of users and groups and authentication of passwords. It is recommended for Plone 3 and 4.
• pas.plugins.ldap. This integration also provides LDAP and Active Directory lookup of users and groups plus authentication of passwords. It is recommended for Plone 5 and up. This add-on is intended for the Zope application server itself, not necessarily Plone, and thus can provide LDAP integration for any Zope-based application.

To summarize (read each column downwards):

Plone Version	Archetypes	Dexterity	plone.app.ldap	pas.plugins.ldap
3	✅	❌	✅	❌
4	✅	✅	✅	❌
5.0	✅	✅	❌*	✅
5.1	?	✅	❌	✅
6 (speculative)	❌	✅	❌	✅
7 (speculative)	❌	✅	❌	✅

Plone at NCI

The Informatics Center provided a Plone portal to the Early Detection Research Network (EDRN) since Plone 2.1. As capabilities expanded and grew, we implemented custom content types implemented in an object-oriented style from ATContentTypes. For example, each collaborative group is represented by a CollaborationsFolder which is derived from the ATContentTypes' ATFolder. A collaborative group's highlight is class Highlight, derived from ATNewsItem. And when the EDRN Directory Service came online, we installed plone.app.ldap.

The Informatics Center also provided a Plone portal to the Consortium for Molecular and Cellular Characterization of Screen-Detected Lesions (which you might expect to be CMCCSDL but is in fact simply MCL). MCL was immediately based on Plone 5 and as such leverages the Dexterity framework but also plone.app.ldap, hence the asterisk in the table above. Sean Kelly simply was not aware of pas.plugins.ldap because of its Zope-focus and not Plone-focus. Strangely, he found a way (or stumbled onto a way) to make plone.app.ldap work with Plone 5.0. But not Plone 5.1.

Upgrading Plone Versions

Upgrading the Plone core is an automated process. The new Plone software detects an older version of both the schema and contents of the Plone database and applies a series of transformations to make it compatible with the new software. If the process succeeds, the database state is committed and everything proceeds as normal.

Except for the add-ons.

Upgrading Add-Ons

Individual Plone add-ons require their own upgrade procedures unless they expect to be compatible with a newer Plone version. Modern "well behaved" add-ons will provide upgrade steps, and each one can be individually upgraded through Plone's control panel or programatically. This is accomplished by assigning each add-on a "profile version" where "profile" refers to the state of the database (schema and contents) similarly to the way Plone upgrades itself.

A newer version of an add-on simply provides a newer profile version and a series of "upgrade steps" to go from one version to the next. Most "well behaved" add-ons do this.

Uninstalling add-ons follows a different process: instead of a series of "downgrade steps" there is an "uninstall profile" that simply removes all of the database changes in one fell swoop. You can visit the Plone Control Panel and click "Uninstall" or even uninstall an add-on programatically.

It is important to note that the database contents in this case are not simply integers, strings, floating point values, and the like; Zope is an object database, and stores serialized Python objects. Because of this, the Python code that implements each object must be present in memory in order to manipulate them: this includes upgrading them when upgrading an add-on as well as uninstalling them when removing an add-on. That last part must be emphasized: to uninstall an add-on, the code for that add-on must still be available, and it must provide an uninstall profile. If either are missing, all sorts of useless objects will be left in the database.

The consequence is that whenever Plone (Zope) processes a URL request, it traverses the URL and passes through each component; most add-ons install "persistent utilities" in the root of a Plone site, though. So every add-on gets a chance to access the request in some way, even out-of-date add-ons, even add-ons whose code has been removed. This has implications for both MCL and EDRN.

The MCL Problem

As we can see from the table above, MCL (running Plone 5.0) is currently using plone.app.ldap. However, plone.app.ldap does not work with Plone 5.1. We must shift to pas.plugins.ldap.

Yet plone.app.ldap does not provide an "uninstall profile". There is no (easy) way to uninstall it. Simply removing the code will leave unimplemented objects in the database, and every URL request will cause stack traces to appear in the log as Zope attempts to give these unimplemented "persistent utilities" a chance to access the URL request. Ultimately, this will slow down every request: for each page, every stylesheet, each JavaScript, images, and so forth. It will also fill the log files quite quickly.

It is possible to uninstall an add-on that fails to provide an "uninstall profile" or whose code is unavailable, but this is considered wizard-level.

The EDRN Problem

EDRN also uses plone.app.ldap and therefore suffers the same fate as MCL. But EDRN faces an additional issue: transitioning its custom content-types from the Archetypes Framework to the Dexterity Framework.

Plone 5 promises to maintain compatibility of Archetypes in order to ease this transition. By simply including the Archetypes and ATContentTypes packages in a site's buildout, Plone 5 will apparently happily welcome these custom content-types. In practice this works for simpler content types. In my experiments so far I have not successfully gotten all of our custom content types to function properly in a Plone 5 environment with Archetypes compatibility enabled. We have a deep class hierarchy of custom content types. For example:

• class BiomarkerPanel
  • class QualityAssuredObject
  • class Biomarker
    • class ResearchedObject
    • class ATFolder
    • class KnowledgeObject
      • ATContentType

Many of these classes are supported by custom code. Class KnowledgeObject, for example, has a URIValidator so its RDF Subject URI can be confirmed and also generates a vocabulary so it and its child classes can be used in faceted navigation. Plone 5 provides an automatic upgrade script for older Archetypes-based classes, but in practice these work for the simplest of implementations.

This sounds like a complete rewrite may be in order; however the Plone Transmogrifier may also be a solution. This will require more research.

Conclusion

The suggestion to "simply upgrade to Plone 5" as a way to solve a wide swath of Section 508 issues was extremely optimistic. Once I began down the path of actually doing so, I ran into issue after issue. I asked for CBIIT to provide implementation data. I retrieved database snapshots from NCI. I created smaller and ever expanding test cases. I engaged the Plone community. I ended up at lot of dead ends. I went through log file after log file. I checked out different versions of packages. I tried setting the versions of individual components both up and down. And I felt as if my body had aged 10 years in just one and a half months.

I have used the "wizard-level" techniques before to remove a misbehaving add-on, but that was many years ago. I have saved the code, but it will take some time to re-familiarize myself with it and get it up to speed to uninstall plone.app.ldap from both MCL and EDRN. The Transmogrifier, however, is new technology to me and may take some time to fully comprehend and configure for EDRN's aging content-types.

In the short term, it may be simpler to manually edit the EDRN Theme to address the Section 508 issues and later face the upgrade issues.