Pico 4.0 and beyond

[PhrozenByte]

This issue is intended to collect and discuss ideas about Pico 2.0 3.0 4.0, especially on how we can make Pico's plugin system more flexible than ever. The changes are very substantial and far reaching, therefore they can't make it into the soon-to-be-released Pico 2.0 3.0. The changes break BC, therefore they can't make it into Pico 1.1 2.1 3.1, but Pico 2.0 3.0 4.0. Features planned for Pico 2.0 can be found in #270. Feedback is appreciated! 😃

General

• Move Pico to the \picocms\Pico namespace and use a PSR-4 autoloader. Use PicoDeprecated to also provide all classes in the root namespace (using class_alias()), otherwise this would break all existing plugins.
• Add Unit Tests.
• Split Pico::run() into multiple public methods ("phases"), but still call them through Pico::run() (init, request URL, loading contents, evaluating contents (YAML + Markdown), page discovery, page rendering (Twig)).
• Use PicoMarkdownParser, PicoYamlParser and PicoTemplateEngine instead of \Parsedown, \Symfony\Component\Yaml\Parser and \Twig_Environment as type hints and add appropriate wrapper classes for Parsedown, Symfony YAML and Twig, allowing one to extend/replace them completely.
  • Enable some useful Symfony YAML flags by default (like converting keys to string automatically).
  • Also see cancelled "replace YAML/Parsedown/Twig" ToDo below.
• Drop all remaining pre-v1.0 behavior from PicoDeprecated, also remove v1.0+ behaviour with a notable performance impact
• Think about how users can easily install plugins with dependencies when they aren't using composer, but a pre-bundled release
• Think about refactoring Pico's plugin discovery process in general: having a lot of (partially duplicate) code for filesystem-based and composer-based plugins is bad, furthermore there are issues with the current system like Error running Pico on Synology Diskstation #538; maybe use extensible iterators?
• Allow theme developers to register meta headers and change Twig's default config (maybe using a config.yml in the theme's dir?) Implemented with Pico 2.1
• Allow theme developers to register static pages in Pico's pico-theme.yml (e.g. a virtual contact page is added to the pages list not requiring a contact.md with e.g. Template: contact to exist)
• Support relative page URLs in Markdown files
  • We must likely hook into Parsedown and make them absolute
  • Relative paths to other files pages (e.g. assets/image.jpg) should be supported, too
  • Relative paths must be interpreted so that content dir breakouts are stripped, i.e. ../page from content/sub/page.md ends up as https://example.com/pico/page, whereas ../../assets/index.jpg ends up as https://example.com/pico/assets/index.jpg (instead of https://example.com/assets/index.jpg)

Page management

• Implement lazy loading using page objects (i.e. use simple objects instead of $page arrays)
  • Performance, Performance, Performance
  • Implement ArrayAccess; not only to maintain BC, but also to make the object's data easier to use (especially for non-experienced developers)
    • Also switch a page's meta data to ArrayAccess objects and allow case insensitive keys (i.e. it no longer matters whether a meta value is Title or title, one can always access it with Title, title, or even tItLe)
    • Only propagate title, date, template and hidden as page data; everything else (e.g. description, author, robots) gets ordinary YAML metadata and can be accessed via page.meta
  • Pico initially loads no file contents, only page.id and page.url are available at first; as soon as something else is accessed, the file's contents are read; depending on what has been accessed, Pico processes either the YAML frontmatter or the Markdown contents (i.e. $page['content'] won't be deprecated anymore).
  • Plugin developers should be able to implement dynamic values with callbacks (e.g. PicoPage::addDynamicValue(string $key, callable $callback)). This also allows plugin developers to implement lazy loading for custom values.
  • Open question: Should we still allow regular page arrays in the $pages array? Probably not: plugin developers will have to differentiate both cases otherwise, what makes the whole feature a pain in the ass for them. Thus we need a conversion method, otherwise we would break all existing plugins...
  • Page objects don't know their respective next and previous pages, but parent and child pages (see below)
• Implement a lazy page tree (using iterable objects) as better performing alternative to the regular pages array
  • Performance, Performance, Performance!
    • Rather than always loading the whole page tree (a Traversable, ArrayAccess object like the pages array, see above), nothing is loaded by default.
    • When accessing the page tree for the first time, Pico discovers only pages directly inside content/ (with the second level of lazy loading as elucidated above). Pages in sub-directories are accessible through a children key. However, the contents of directories aren't discovered until they are explicitly requested by accessing said children key.
    • Most themes build their page menu by iterating over pages on the first level anyway. This allows Pico to discover only the pages it needs to know, i.e. just the pages directly inside the content/ dir and without any sub directory. This should heavily improve performance when a Pico instance is supposed to serve hundreds or thousands of pages.
    • Even the regular pages array is actually empty in the beginning. However, by accessing the variable (i.e. by iterating over it or by accessing a key) the whole page tree is being loaded (what shatters our efforts). We should encourage plugin/theme developers to use the page tree instead.
  • Pico manages a global sorted page tree
    • Drop support for sorting the pages list globally and rather use concomitant alphabetical sorting; i.e. Pico first discovers content/index.md only, then all other pages in content (e.g. content/foo.md) and sorts them alphabetically, then checks for a content/foo/ directory and discovers all pages in it (e.g. content/foo/bar.md), etc. and slices the just discovered pages in
  • Drop support for the $pages variable and force users to use the pages() function
    • Open question: PicoDeprecated should implement it as some magic variable to still support lazy loading (is this even possible?)
    • The pages() function should return an object instead of an array with sortable page objects wrapping the requested page objects; the pages list object can then be sorted and set a page's respective previous and next page to the page wrapper
      • It should be possible to perform both shallow and recursive sorting
  • Open question: How do we make this backwards-compatible with PicoDeprecated?
    • Use ArrayObject or ArrayIterator instead of a regular $pages array. This allows us to convert page arrays to objects as soon as they are added.
    • However, PHP's built-in array functions (e.g. array_keys()) won't work anymore... According to that we must pass a regular array to the onPagesLoaded event for older plugins, otherwise we would pretty likely break many of them. PicoDeprecated could then iterate through the $pages array and convert them appropriately.
    • Use event system versioning?
  • Refactor the pages() function to use offset and length parameters instead of depth, depthOffset and offset; see Show (grand)parent-page title #627 (comment) and earlier comments
    • Utilize theme API versioning to actually drop depth and depthOffset, but use PicoDeprecated to inject the old function for old themes
    • Thinking about using API versioning anyway, we could also rename length to depth - it better reflects what it does... But it's not to confuse with the "old" depth
    • Taking page objects into consideration we should use those in most cases, i.e. there should be keys to specifically not just access a page's child pages, but ancestors until a user-defined depth

Plugin event system

• Don't trigger all events on all plugins. Let plugins register the events they want to use instead. This heavily increases performance with a large number of plugins, because method_exists calls are comparatively expensive compared to a simple foreach per event.
  • Open question: Either introduce a new onSinglePluginLoaded event or a new PicoPluginInterface::getEvents() method.
    • The latter breaks BC, but with this new approach, we must refactor AbstractPicoPlugin::handleEvent() anyway, therefore we probably can't implement it without breaking BC one way or the other.
    • We can circumvent this by letting \AbstractPicoPlugin and \picocms\Pico\AbstractPlugin differ in functionality (i.e. the first mentioned implements the getEvents() method in a BC way by examining a ReflectionClass of the plugin, or by letting \PicoPluginInterface lack the getEvents() method entirely).
• Allow plugins to return false on preliminary events (e.g. onContentLoading) to prevent Pico from performing a specific processing step (Pico::run() skips Pico::loadFileContent()). Returning true or null works as with Pico 1.0 and changes nothing. The subsequent event is still triggered (onContentLoaded), but the payload variable ($rawContent) is empty. The event is triggered with special priority on this plugin (regardless of the regular processing order), so it can set the variable before any other plugin receives the event.
  • Example:
    • A markdown cache plugin returns false during onMetaParsing and onContentParsing to load both meta data and the parsed contents from its cache.
  • Affected Events:
    • onContentLoading (completely skips on404Content… events) and onContentLoaded
    • on404ContentLoading and on404ContentLoaded
    • onMetaParsing and onMetaParsed
    • onContentParsing (simulates onContentPrepared), onContentPrepared and onContentParsed
    • onPagesLoading (onSinglePage… events will be simulated) and onPagesLoaded
    • onSinglePageLoading (new event) and onSinglePageLoaded
    • onPageRendering and onPageRendered
• Allow plugins to return false on the onRequestUrl or onRequestFile events to completely skip Pico's processing. The only remaining event to trigger is onOutput (new event that is only triggered when Pico's processing is skipped) right before Pico returns $output.
  • Example:
    • A static HTML cache plugin returns false during onRequestFile, bypasses Pico's processing completely and returns the cached contents during onOutput.

New official plugins

• Markdown cache
• Static HTML cache
  • Save rendered output of pages to static HTML files
  • Rely on OS to detect file changes (last modification time of .md files)
  • Add Cache: No meta header to prevent pages from being cached
  • Add a event to let plugins "register" non-content pages for caching (should be triggered right after onConfigLoaded to allow plugins to change their behavior when caching is requested)
  • Explicitly allow combining statically cached and dynamic pages
  • Open questions
    • What happens when a page is added (i.e. page navigation changes)?
    • What happens when a plugin or theme is added/updated/removed?
    • How to determine all URLs that need to be parsed? Markdown files don't necessarily have a 1:1 relation to pages, just think of collections or hidden meta files
      • Ignore files and directories starting with a _
      • Allow users to explicitly specify the URL of a page
  • Use this feature to allow Pico to act as a static website generator (allow plugins to distinct between "static HTML cache" and "static website generator" mode)
    • Use Pico (with Travis, PHP's development server and wget -r) for our website rather than Jekyll
  • Plugin plugins (:smile:):
    • Add a URLs sequence meta header to support alternative URLs (like Jekyll's Redirect From plugin
• Search (using Lucene?)
  • Problem: How to determine the URL of a found Markdown file? Markdown files don't necessarily have a 1:1 relation to pages, just think of collections or hidden meta files
  • Possible solution: Use a static HTML cache and search in the HTML files
  • Possible solution: Do the exact same things as the static HTML cache (see above)
• Multilanguage (i18n)
  • Contents: use language-specific content directories or file extensions?
  • Themes: use PHP intl and Twig_Extensions_Extension_Intl
  • References: How to create multilingual web site? #335
• Performance statistics: See Pico 4.0 and beyond #317 (comment)
• Import plugins to import contents of other CMS (e.g. WordPress, see https://github.com/gilbitron/WordPress-to-Pico-Exporter)
  • Use a HTML to Markdown converter?
• Data Files
  • Support independent (meta) data files (e.g. content/catalog.yml)
  • The files are accessed similar to pages (e.g. {{ data.catalog }}).
  • A data file named after a markdown file (e.g. both content/catalog.yml and content/catalog.md exist) is non-recursively merged into the page's meta data (i.e. into {{ pages.catalog.meta }}). However, the YAML frontmatter takes preference and the data file can still be accessed via {{ data.catalog }}.
  • The same happens for all pages (non-recursive) in a directory if there's a data file with the same name as the directory (e.g. _collection.yml and _collection/ directory). You can enforce recursion for e.g. _collection/subdir/ by creating _collection/subdir.yml.
• Redirect pages (like Jekyll's Redirect From plugin)

Not planned anymore

• Allow a single plugin to hook into Pico to basically replace YAML/Parsedown/Twig with something different. Rather than hooking into the instantiation of \Symfony\Component\Yaml\Parser in Pico::parseFileMeta(), Pico::registerParsedown() and/or Pico::registerTwig(), it should be possible to replace the Pico::parseFileMeta() method, the Pico::prepareFileContent()/Pico::parseFileContent() methods (+ the markdown Twig filter in PicoTwigExtension) and/or the call of Pico::$twig->render(). Otherwise the plugin needs to re-implement the internal structures and workings of the YAML parser/Parsedown/Twig, what isn't desirable. I'm currently not sure about how this interacts with the $twig parameter of the onPageRendering event (maybe drop the parameter and add a new onTwigRegistered event?). The plugin needs to be registered explicitly in config/config.php to work.
  • Example (quite a stretch):
    • Instead of parsing Markdown, parse MediaWiki syntax.

[Crank1d]

I have idea on how to make pages more flexible - in other words, having "custom page types" based on XML schema file (or YAML, maybe?) in content directory without writing specific meta data into every .md file.

This would make Pico more CMS-like and structured, so for example, if someone would like to have "blog" or "products" type of pages, each containing alot of "child pages" (subpages), he should create XML schema file called "blogposts.xml", which contains info/data describing how subpages should behave, like order of pages, pagination count etc. and any pages associated with schema put into subfolder in content directory. This would probably branch content directory structure into several subfolders for specific page type so structure would be something like "default" - for default pages, "posts" - for blog posts or any other page types. Index pages for those page types could also contain page loop/query of all of its child pages - just like blog has, similar to Wordpress but still being simple.

Also, maybe having another XML file for post categories which would contain something like page IDs under specific category section ("Animals", for example) and when category URL is requested, category page would do the loop and display all pages associated with this category page.

I think this would be possible but would require to rewrite core Pico files, since creating Blog plugin alone would not have access to way/order pages are displayed since Pico searches folders recursively for .md files in content directory.

This would increase complexity of Pico but it would be a welcome feature which could possibly draw more people to use Pico for their website, especially those having smaller blog dedicated to specific subject, like personal travel blog, for example.

[PhrozenByte]

Thank you for your feedback @Crank1d! 👍

I did not really understand what data you're suggesting to store in these XML files. Pico leaves page presentation (i.e. how a page in a specific folder (e.g. blog) looks an behaves) completely up to themes. For example, @smcdougall's NotePaper theme interprets content files differently based on YAML headers (e.g. NotePaper's widget feature or blog articles). NotePaper specifically also allows you to set a Tag meta header and lists all pages containing a specific tag (so implementing categories shouldn't be a problem either).

As far as I understand your suggestion right, Pico already allows you to do something like this - it just doesn't require you to create a separate XML file, you can simply store that information in the YAML header of the content file.

[Crank1d]

Something like "Page Blueprints", implemented in GRAV - http://learn.getgrav.org/forms/blueprints

Scheme file would contain custom defined meta data for all its child pages/subpages, so there would be no need to define custom meta from plugin.

Maybe Im wrong, but Pico currently doesnt support "parent-children" relationship of pages?

[PhrozenByte]

As far as I understand Grav's documentation right, blueprints are about metadata for plugins and themes, not about pages. Pico has no administration backend. We're indeed working on a editor plugin for Pico 1.1 (see PhrozenByte/pico-admin), but that's rather a "enhanced" text input field. Much more "administration backend" is in contradiction to Pico's "stupidly simple" goal (what isn't limited to the user-side, but also Pico's sourcecode).

Pico doesn't have a "parent-child" relationship of pages in the narrow sense. When it comes down to "folder-specific meta data defaults": Pico absolutely allows you to do something like this, e.g. by creating a _meta.md file and adding its meta data to all other pages in this directory by using the onPagesLoaded event with a very simple plugin. Apart from that, Pico leaves the directory structure completely up to the user and maybe a custom theme that interprets the directory structure to do something special (like listing all pages in categories).

[Crank1d]

OK, thanks for explanation.

[mayamcdougall]

Just a thought... If you wanted to add the feature sooner, it'd probably be easier to make the "static website generator" a manual process for the initial implementation. The automation (presumably the harder part) could be developed in stages afterward.

Maybe you'd want them to be two separate components anyway. A "worker" that builds the static site and an optional "watcher" daemon/process that provides the file tracking and automation. This would also allow users the freedom to decide it they wanted the extra automation or whether they'd rather rebuild manually.

Anyway, just some stray thoughts from having dealt with my odd, manual Jekyll setup.

Even if it had to be rebuilt manually each time, it might get Pico to a state where you'd want to use it for the website sooner. 😉

[mayamcdougall]

Another random thought. Not really big enough for it's own issue.

What about having some kind of performance monitoring in Pico? I'd call it separate from the "static website generator" because it could be useful for regular, dynamic Pico sites.

Basically, I've noticed with Jekyll that whenever it rebuilds the site, it tells me how long it takes. I've noticed that certain actions (like jsonify) take a LOT longer than others. I think it would be interesting for Pico theme and plugin developers to be able to measure the impact of their actions.

It would probably be a toggle or a plugin, since it wouldn't really have a production use.

A detailed breakdown of performance timings could be cool too.

Pico Core.................................10ms
Plugin 'something_low_level.php'..........20ms
Parsedown.................................80ms
Plugin 'pagination.php'...................50ms
Twig using template 'NotePaper'..........200ms
----------------------------------------------
Total Rendering Time.....................360ms

The timings here are of course made up.

Again, just an idea I had for the future. I can make it a new issue if you'd like.

[PhrozenByte]

Just a thought... If you wanted to add the feature sooner, it'd probably be easier to make the "static website generator" a manual process for the initial implementation.

Actually it's already possible to use Pico as a static website generator - simply use wget -r to recursively get a static version of the website. Whether the result is usable or not depends on what plugins have been "installed". The primary target of this feature actually is caching the HTML results to boost performance, using Pico as static website generator is just a side benefit.

The decision to use Jekyll was made before I knew about what power Travis gives us. So, if you want to do a little research whether we can use Pico 1.0 instead... That is something which I would greatly welcome. 😉 😃

Maybe you'd want them to be two separate components anyway. A "worker" that builds the static site and an optional "watcher" daemon/process that provides the file tracking and automation.

I haven't thought about a concrete implementation yet. So, yeah, maybe 😉

What about having some kind of performance monitoring in Pico?

We'll keep this in mind (I've added it to the list above), but at the moment there are some ToDos which will boost performance significantly (Markdown cache, lazy loading of pages...), so it IMHO doesn't make much sense to publish performance stats when we already know that we can do much better.

[mayamcdougall]

The primary target of this feature actually is caching the HTML results to boost performance, using Pico as static website generator is just a side benefit.

Yeah, I'm interested in seeing how it plays out. I've played with Nginx's caching abilities and seen my pages go from 500ms+ down to near instant loading times.

Maybe I should go write a script that wget's into a static folder on demand. 😆

For the performance stats, I'm more curious from a theming perspective. It would be interesting to be able to track performance changes between different ideas. If I change the way something works, and all of a sudden it's taking an extra 100ms or more, I'd know that my implementation was poor.

It would actually be kind of interesting to see how NotePaper performs next to a barebones theme. There's probably a way I could send a request to php directly from a shell, then time the response using some command line trickery. I might see what I can do if I'm every working on NotePaper. Far too busy for that at the moment though.

[Chris2011]

What is happened with the Static HTML Cache? Is this the Pico Cache plugin: https://github.com/glumb/pico_cache?

[PhrozenByte]

Welcome back @mayamcdougall! 🎊 😃

Sure, looks good, go ahead 👍

[Nepose]

Not sure whether this topic is still alive, but I'm going to try 😉
IMHO the 3.0 version should finally use all goods PHP7 gives us and leave support for 5.3.6 version.

[bricebou]

Hi @PhrozenByte ,

Thanks for all the good work you are doing on Pico CMS !
I've seen the release of the alpha version of Pico 3, and it's a great news.

But, my major concern is that I need a tag functionnality... I tried to work on a plugin a long time ago (and you saw how ugly was my code, but hey, I'm not a developper and haven't any PHP basics...) ; I see there is a plugin for version 2 but why not make tags a native functionnality in Pico 3 ?

Thanks again !

[PhrozenByte]

Tagging is a rather basic functionality and one of the perfect examples of what can (and should) be done with plugins. The most basic tagging functionality can even be realized with pure Twig. So if there's going to be something official it will rather be a plugin. Since time is very limited I'm afraid I can't promise that 😒

Anyway, maybe it's a good approach to think a little about what is needed for a Twig-only approach... There should be just some rather small hurdles I assume. We'll see... 😃

[mayamcdougall]

I mean, if you want to be technical, I came up with a Twig-only tags implementation back when I wrote NotePaper. It's very basic, and I never went back to really improve it, but it might be a good starting point if you're looking to avoid writing a proper plugin.

NotePaper as a whole is kind of unmaintained at the moment. It probably still works fine on the current version of Pico, but just be aware that it hasn't been properly maintained in a bit.

Edit: You can check out the NotePaper Site for an example of the two twig-based tag widgets I wrote.

[bricebou]

Thanks a lot for your answers ! I'll check Pico's development from time to time and I'll try to upgrade my website.
Thanks again :)

[KoljaL]

Hi @PhrozenByte,
i`m playing around with a new theme for my documentation website.
Now it is time to make the menue and i found a lot of ways to do it.
Like with the NestedPages, the Category plugins or the pages() function of pico itselfs.

But what will the (best or easiest) way to to make it fitted for pico3?

My need is a menu with all pages in their folders (without a limitation in the depth), highlighting the current page and the way through the folders from the current page to un-collapse them.

[PhrozenByte]

You should use Pico's new pages() Twig function @KoljaL, it already works the same way as the new page discovery will work 🚀

[KoljaL]

@PhrozenByte, thanks for the hint.
Is there another way for nested menus than the one in this example: #383 (comment)?
As I said, I would like to have more levels.

[PhrozenByte]

This example isn't even utilizing the pages() function... See http://picocms.org/in-depth/features/pages-function/

[KoljaL]

oh, now i know that there is a difference between pages and pages().
But i found only examples for the pages variable, like you wrote in the old themes, and nothing with the pages() function.

[PhrozenByte]

{% macro rnav(startPage, currentPage) %}
    {% import _self as macros %}
    {% for page in pages(startPage) %}
        {% if not page.hidden and page.title %}
            <li{% if page.id == currentPage.id %} class="current"{% endif %}>
                <a href="{{ page.url }}">{{ page.title }}</a>

                {% set childNav = macros.rnav(page.id, currentPage) %}
                {% if childNav|trim %}<ul>{{ childNav }}</ul>{% endif %}
            </li>
        {% endif %}
    {% endfor %}
{% endmacro %}

{% import _self as macros %}
{% set index = pages["index"] %}
<ul>
    <li{% if index.id == current_page.id %} class="current"{% endif %}>
        <a href="{{ index.url }}">{{ index.title }}</a>
    </li>
    {{ macros.rnav(index.id, current_page) }}
</ul>

[KoljaL]

Thank you for the example. I have already played with it and noticed that folders without "index.md" are not displayed at all. No matter whether they contain other files or subfolders. Is it correct that way?

[PhrozenByte]

Correct. If you wanna list them you'll have to use the page tree directly, see https://phrozenbyte.github.io/Pico/in-depth/features/page-tree/

[KoljaL]

No, it`s okay.
I will put index files in every folder and do the rest in CSS.
Maybe it is good to have the chance of this content possibility.

[mhzawadi]

Good evening lovely Devs,

Thank you so much for this super simple CMS, I have it running as both my main site and some sites in nextcloud.
Would love to send some money if it would help.

I would also like to help test and move v3 forward, I have seen the Alpine Linux 3.16 drop and that drops support for PHP7.
My docker image is built on Alpine and so cant bump to 3.16 just yet, what would be most helpful in doing any testing?

[mhzawadi]

FYI: have bumped my image to PHP8.1 and using the v3 alpha build, no errors seen. Buts its a very simple site.

[Thatoo]

I think it would be nice to integrate ActivityPub into PicoCMS? Maybe offering a plugin?

So far, to write blog that integrate within the fediverse through ActivityPub, there is only https://writefreely.org (no theming possible and versy difficult to install) or thanks to this wordpress plugin https://wordpress.org/plugins/activitypub/ .

[lloydsargent]

This is really basic: moving user plugins/content/themes all under the content directory. I’m a fan of keeping user and system files separate. ATM, it’s pretty loosey-goosey in this regards. It would also assure several things:

o Updates are simpler (whether you hand cart them like I do or use Composer (which I have never managed to get working).

o If you don’t use Composer (which I do not having never had it work), it makes updates even simpler.

o Keeps me from having to figure out what I do and don’t need .gitignore on. Frankly, it’s enough to deal with my own stuff. Just let me go “oh, that directory is magic… don’t touch it”.

i think simplifying this to PicoCMS directory and User Content directory would clean things up significantly.