Support Distributions to be built on top of Plone

[ericof]

PLIP (Plone Improvement Proposal)

Responsible Persons

Proposer: Érico Andrei [@ericof]

Seconder: Timo Stollenwerk [@tisto]

Abstract

Over the years, the Plone community has developed numerous "distributions" of the Plone CMS, even without official support from the core codebase. These include SENAITE, Quaive, Portal Modelo, Portal Padrão, CastleCMS, and ioComune. These distributions often require patching Plone or necessitate manual steps to create a customized Plone site. This PLIP aims to establish a clear protocol for creating and managing new Plone distributions, including the incorporation of example content during site creation.

Talk about plone.distribution at the Plone Conference 2023

Motivation

As one of the creators of Portal Padrão and a contributor to Portal Modelo, I have experience in modifying the Plone site creation process to include additional steps and customize the resulting site. However, each distribution, including Quaive and CastleCMS, has its unique implementation. The absence of a standardized approach and official support can lead to compatibility issues when changes occur in core packages like Products.CMFPlone.

Assumptions

We assume that the existing options in Plone’s Add Site page are basic distributions. Their content creation processes need unification to ensure consistency and ease of use.

Proposal & Implementation

We propose the development of an API to streamline Plone site creation. This API will facilitate site creation through Python code and RESTAPI, enabling the development of frontends for default Plone or SaaS offerings. A standardized method for importing content into new websites will also be introduced to enhance user experience and consistency across distributions.

Deliverables

• A new package, plone.distribution, which:
  • Supports Plone 6.0.
  • Implements basic content distributions, mirroring the content in plone.volto and plone.app.contenttypes, until these packages provide their distributions.
  • Incorporates ZCML support for registering new distributions and managing site creation.
  • Features an extensible "wizard" utilizing JSONSchema to facilitate user input during site creation.
• Elimination of conflicting site creation methods in:
  • Products.CMFPlone: Removal of browser/admin.py.
  • plone-backend: Removal of scripts/create-site.py.
• Modifications to:
  • plone.volto: Introduction of a distribution registration to manage default site creation.
  • plone.classicui: Implementation of a distribution registration to oversee site creation, replacing the existing method in plone.app.contenttypes.
  • plone.app.contenttypes: Removal of the existing content setup implementation.

Risks

Supporting Plone 6.0 necessitates that plone.distribution duplicates content from plone.volto and plone.app.contenttypes. It also requires the implementation of a unified site creation function and RESTAPI services, ideally situated in Products.CMFPlone and plone.restapi, respectively.

Participants

• Érico Andrei

[sneridagh]

+1 to the proposal. I subscribe to every single line.

Maybe it's worth to add the i18n limitation and that we need to find a good way to overcome it.

@ericof maybe you can also point to your Plone Conference talk this year, talking about them? I think there are really good points in there and it does expose the whole picture in an extraordinary way.

[tisto]

@plone/framework-team since this is a PLIP that covers both Classic UI and Volto I guess we need to schedule a call, or at least have a vote if we think that this is a good idea. Since I am seconding this idea, I am +1.

[mauritsvanrees]

I obviously like this idea and would love to have this in Plone 6.1.

Some details are not yet clear to me though, especially around the dependencies, and possibly circular dependencies:

• plone.distribution depends on collective.exportimport, which depends on plone.restapi. plone.restapi is currently not a dependency of Products.CMFPlone. So when you pip install Products.CMFPlone you should not get plone.distribution. But this means there is duplication of code between plone.distribution on one hand, and Products.CMFPlone, plone.volto, and plone.app.contenttypes on the other.
• plone.distribution depends on Plone. But don't we want Plone to depend on plone.distribution?

[ericof]

@mauritsvanrees Excellent points.

Dependencies

My original vision was to move site creation logic to Products.CMFPlone and keep a slim plone.distribution focusing on high-level API to register new distributions and provide a UI and rest endpoints.

Of course, we would need to address how to handle plone.distribution with Plone 6.0 and the code-duplication we would have until 6.1 is out.

Export / Import

I would love to see Products.CMFPlone (or GenericSetup) shipping with a default way to serialize and deserialize content to JSON and collective.exportimport, focusing on handling all possible edge cases and conversions used in migrations.

I want to discuss implementing this and where it should be done.

@pbauer, opinions here?

[MrTango]

+1 for the whole idea
could you please explain this point a bit more in detail:

Implements basic content distributions, mirroring the content in plone.volto and plone.app.contenttypes, until these packages provide their distributions.

[ericof]

@MrTango, In Plone 5.2 and 6.0, we have the example content being implemented by setuphandlers in plone.volto and in plone.app.contentypes.
To support plone.distribution being usable in Plone 6.0, we need to mirror the content created in plone.volto and plone.app.contenttypes, but for Plone 6.1, both packages should replace their setuphandlers implementation by distributions.

[ericof]

@mauritsvanrees, @fredvd, @pbauer : After some digging, it seems the plone.restapi serializers and deserializers would be very hard to be refactored into another package.
Any opinions?

[mauritsvanrees]

• The Classic distribution has portlets.json . This could just be a standard portlets.xml .
• For each distribution automatically register a profile directory as GS profile. Instead of profiles.json you can have a metadata.xml . (But the json also has a content key.) Uninstall should not be needed: you would just delete the Plone Site.
• How would we translate the default content into other languages? One profile per language? Run a translate function over the json right before importing it?
• I doubt ordering.json is needed: we import the items in a specific order, so that should be fine already, at least when we export them in the correct order in the first place.
• Volto/default: is 1.json really enough? Or does other content still need to be added?
• Could we add is_default_page: true to the classic jsons that need it? The importer would then set this as default page of the folder. If two items have this in the same folder: the last one wins. Then we would not need defaultpages.json , at least not in plone.distribution.
• Is the headless option needed? Can’t we support the portlets and defaultpages export steps automatically only when they are needed? Even if they give an empty file, it would be okay. Or we could automatically leave an empty file away from the export.

[jensens]

I added a PLIP configuration to buildout.coredev - @ericof please check the branches!

run buildout -c plips/plip-distributions.cfg in coredev

[tisto]

I can not approve a PLIP where I am a seconder I guess. However, the @plone/release-team team discussed this and approved the PLIP.

[davisagli]

@ericof @mauritsvanrees I've tested this using plips/plip-distributions.cfg in buildout.coredev, and I think we are close to being ready to merge the following PRs for Plone 6.1:

• Refactor site creation #3981
• Soft dependencies on p.a.{discussion,multilingual} plone.exportimport#31
• Move internal classic distribution to plone.classicui  plone.distribution#90
• Drop support for Plone 5.2 and Plone 6.0 plone.volto#156
• Add distribution tests, sync dependencies plone.classicui#13

I've also successfully run a 6.1 PLIP build on jenkins with these PRs.

Remaining work:

• documentation
• update addPloneSite in CMFPlone to support passing a distribution when plone.distribution is installed (not a blocker)

[mauritsvanrees]

Since this is close to merge, I have updated coredev 6.0 to use the 4.x.x branch of plone.volto that I just created from main.

I have added plone/Plone#31 to add plone.classicui as dependency of the Plone package. I have added this branch to the sources in the PLIP config.

[mauritsvanrees]

Remaining work:

• update addPloneSite in CMFPlone to support passing a distribution when plone.distribution is installed (not a blocker)

I have added this in bea0b3d.

[mauritsvanrees]

I think I am done with all the changes that I want.

TODO: documentation.

@gforcada Could you add a PLIP job for this on Jenkins please? Some have been testing this by pasting the PR urls, but the PLIP config also adds some test-eggs, so that does need a special job:

test-eggs +=
    plone.classicui[test]
    plone.distribution[test]
    plone.exportimport[test]

[gforcada]

Let me have a look :)

[gforcada]

Jobs are created (see plone/jenkins.plone.org@5230f6f):

• https://jenkins.plone.org/view/PLIPs/job/plip-plip-plone-distribution-3.12
• https://jenkins.plone.org/view/PLIPs/job/plip-plip-plone-distribution-3.8

[mauritsvanrees]

Thanks Gil. Updated to 3.10 and 3.12 meanwhile.

But there are three test failures on all runs. See 3.10 and 3.12

• Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_search.SearchControlPanelFunctionalTest.test_search_control_panel_link
• Products.CMFPlone.tests.test_robot.RobotTestCase.Scenario Enable anyone to view about information in the Security Control Panel
• Products.CMFPlone.tests.test_robot.RobotTestCase.Scenario Exclude Content Types from Search

Do those failures ring a bell? Did anything get fixed recently that still needs to be pulled in to the various PLIP branches?

[mauritsvanrees]

I have fixed the three failing tests. See ec4601c

Running the Jenkins PLIP jobs now.

[mauritsvanrees]

Robot tests on Jenkins are a bit flaky lately. Some people are porting robot tests to Playwright, which may help there.

But the 6.1 PLIP job at least passes.

@plone/framework-team and others: last call for reviews. If we merge everything this week, then maybe the Salamina sprinters can fix some bugs that we overlooked. ;-)

There are some details left like plone/plone.distribution#95, but we can polish later.
And documentation needs to be added.

[mauritsvanrees]

I created a PR for the 6.1 upgrade guide, please review:
plone/documentation#1733

With that under way, it is time for me to merge all related PRs. There will probably be some breakage on Jenkins for a bit, as there is no correct order in which to merge this.