Experiment: A simple Modules API with no server dependency graph

[luisherranz]

What?

This is the first of a series of experiments to understand the requirements of a hypothetical Modules API. More info in the Tracking Issue:

• Tracking Issue: Script Modules API #55942

This experiment aims to implement the simplest possible API, with these characteristics:

• An API to register modules using:
  • A module identifier (required)
  • The full URL or the path (required)
  • Whether the module is for the admin, the frontend or both (required)
  • A version (optional)
• An API to enqueue modules.
• No server graph of dependencies.
• All registered modules add an entry in the import map.
• Enqueued modules add a <script> tag in the head.

Why?

To understand the requirements, we can start with the simplest possible API and test it out with the interactive blocks.

How?

By leveraging a new Gutenberg_Modules class that acts as a singleton and a couple of functions to expose the functionality:

• gutenberg_register_module to register modules and add entries to the import map.
• gutenberg_enqueue_module to enqueue modules.

Then, we print two times using the wp_head action:

• The import map.
• The enqueued modules.

Testing Instructions

• Add some interactive blocks: Navigation, Query or Image block.
• Check that in the frontend, there is an import map script.
• Check that scripts with type="module" are loaded by the interactive blocks.
• Check that the @wordpress/interactivity module is loaded by the other modules using import ... from "@wordpress/interactivity".

[youknowriad]

Do you think we can avoid the file source from the enqueue function? I expect enqueued modules to be registered before hand no? (I know the current scripts API also allows this but it feels weird to me)

[luisherranz]

Sure. I'll do that.

One thing that feels a bit weird to me, though, is to add an entry to the import map for the enqueued modules because they rarely export anything. So I wonder if register and enqueue should be separate things, one adds entries to the import map and the other adds <script> tags. Something like:

• gutenberg_include_module: adds the module to the import map
• gutenberg_enqueue_module: adds the module to the header using <script>

[youknowriad]

I see what you mean, I'm not sure at the moment. I also wonder if there are scripts that could be entry points but also dependencies at the same time.

[luisherranz]

I also wonder if there are scripts that could be entry points but also dependencies at the same time.

I think it's a possibility, but it's going to be uncommon.

[youknowriad]

For the basic version (no server graph), I was wondering if we could support some kind of "flags" or something to say that a script is a backend script, frontend script or both. (It's a small optimization as there's a lot of backend script that are not meant to be loaded in the frontend)

[luisherranz]

Sure. I don't think it makes sense to have a version of the API that doesn't do that.

[luisherranz]

Added in 11b4e1b.

[youknowriad]

I think the same script can be probably be used in both frontend and backend. I also wonder if we'd have more "usage" in the future. Do you think a string is enough or should it be an array? I'm not very opinionated, just asking the question I guess.

[luisherranz]

Makes sense. I changed it so now it accepts a string or an array of strings: e0201ce

[youknowriad]

This is cool. Thanks for the work here. I'm looking forward to seeing this IRL.

[tomalec]

By leveraging a new Gutenberg_Modules class that acts as a singleton and a couple of functions to expose the functionality:

• gutenberg_register_module to register modules and add entries to the import map.

also related to the thread above

I wonder if register and enqueue should be separate things, one adds entries to the import map and the other adds <script> tags. Something like:

• gutenberg_include_module: adds the module to the import map

WDYT, of making it more explicit. Instead of making up a new word for a high-level concept of us "registering"/"including", make the function state what we actually do.
As this is a new API, and the first step in the experiment to build a more complete solution, we could start with low-level principles. This would also allow newcomers to pick up the concepts quicker.
There are more JS developers than Gutenberg developers - therefore, I assume (eventually) more people are familiar with ESM + import maps and their flow than with Wordpres + Gutenberg flow.

gutenberg_(add|register)_imort_map_entry - for the sake of import map, it does not have to be anything "module" related. Import maps process identifiers and paths. Like

 {
   "imports": {
     "lodash/": "/node_modules/lodash-es/",
     "https://www.unpkg.com/vue/": "/node_modules/vue/"

(No module was referenced above)

Personally, I find "include" more relevant to the behavior of PHP include statement or HTML includes - meaning it actually stamps/includes/pastes the content to be processed. What we do here - registering an entry in import - is just adding an entry to a map for a potential future reference, which may never be used.

---

I don't mean we need to cover all the aliasing, remapping, and all features of import maps in the first iteration. But let's not limit ourselves and users without a good reason.

For example, just by changing the name, and leaving the implementation more-or-less the same we can have unlock use-cases like:

• DEWP exceptions

gutenberg_register_import_map_entry( '@automatewoo/bundled/', '/wp-content/plugins/automatewoo/build/...');

// Somewhere in AutomateWoo codebase:
import { Button } from '@wordpress/components';
import { Button as PatchedButton } from '@automatewoo/bundled/@wordpress/components';

To cover the bundled-vs-extracted logic and developer experience currently covered by the Dependency Extraction Webpack Plugin. #35630

• static imports

gutenberg_register_import_map_entry( '@automatewoo/static-data/', '/wp-content/plugins/automatewoo/data/');

// ...
import { version, minWP } from '@automatewoo/static-data/extension-info';
console.log( `You're running AutomateWoo ${ version }, it requires at least ${ minWP }.` );

Without a need to "register" every single file under data/.

[luisherranz]

Hey Tomek, that's indeed an interesting approach. Thanks for sharing 🙂

However, I see some issues for which I don't think it would be good to expose an API like that in WordPress. Remember that WordPress is a platform where code ownership is divided among different people (core, different plugins, theme…), which affects how we need to think about these APIs. Let me elaborate.

Cache Invalidation

When you update WordPress or a plugin, they might include newer versions of some of the JavaScript files. Other JavaScript files might rely on those updated versions to work correctly.

Most hosting providers configure their servers to treat JavaScript files as immutables, and they send those cache headers to the browser. This means that in WordPress, the URLs of each JavaScript file need to be unique, or your site might not function correctly when a user who has visited previously the site, visits it again but doesn't get updated URLs.

If we expose a low-level API and people can add entire folders to the import map, the filenames need to be hardcoded in the JavaScript files, where we cannot make them unique:

// PHP
gutenberg_register_import_map_entry(
  "@automatewoo/",
  "/wp-content/plugins/automatewoo/build/"
);

// JS
import { something } from "@automatewoo/some-folder/some-file.js";  // filename hardcoded

The only way to invalidate the cache would be to include a build hash or version in the build folder name. But forcing every plugin author to use that complex build configuration doesn't seem like a good idea, especially when people are already used to delegating that management to WordPress.

Filename Backward Compatibility

By hardcoding file paths in the JavaScript files, we would lock them into those specific paths/names. In the future, if someone decides to rename or relocate a file, any JS code that references the old path/name will break. That practice will lead to a fragile architecture where the risk of breaking changes is high, as the system lacks the flexibility needed for maintaining and refactoring the codebase.

For example:

// Original import
import { utilFunction } from "@automatewoo/some-folder/utilities.js";

// If we move utilities.js to another folder or rename it, this path is invalid.

Dependency Preloading

Preloading is important for performance, as it helps avoid initial waterfalls by allowing the browser to start downloading static dependencies early on. Introducing a low-level API that could be used to register folders (as opposed to files) might complicate this process if dependencies (and dependencies of those dependencies) are unknown by the server.

<link rel="modulepreload" href="/path/to/dependency.js" />

Exposing an additional API exclusively for preloads may work fine for direct static dependencies, but if those dependencies have other dependencies, preloading them becomes non-trivial. You'd have to know and manage the entire dependency tree manually, which can become cumbersome and error-prone, as any update in an external plugin could change their dependencies without notice.

This is not a problem exclusive to the low-level API you suggested, as this first experiment is also not able to handle preloads, but it's another issue that will make it difficult to adopt it in more complex implementations like in the second or third experiments.

---

So, while I agree that the idea of using a low-level API for managing import maps might seem appealing for its simplicity and directness, and would be an excellent choice in other platforms, I believe that, in WordPress, it might give the wrong incentives to people and introduce several significant maintenance and usability issues regarding cache management, backward compatibility, and dependency preloading.

Let me know what you think 🙂

[tomalec]

Thank you for the elaborate answer :)
I agree there are valid concerns.

Forgive me that I'd start philosophically ;) My main idea was that as we're still in the early stage of experimentation, we could allow ourselves to start simple. Exposing APIs that may be abused and used in suboptimal ways if someone would like to go in a different direction than we imagine.
Allowing them and us to go in other directions than we initially envisioned builds an opportunity for greater improvement and designing a solution not limited to our biases today.

Bringing it back to our ground: if we deliver (experimental) wp_add_import_map_entry, we can still use that build convenient, optimized, and high-level wp_register_module as we plan. We don't have to guarantee that …_add_import_map… will make you use of all the benefits of the WP Platform and cover the problems you mentioned. We don't need to try to make folder-mapping work. What I suggest is just not to block experimenting in those and other directions at this point.

---

Re: Cache Invalidation

in WordPress, the URLs of each JavaScript file need to be unique, or your site might not function correctly when a user who has visited previously the site, visits it again but doesn't get updated URLs.

If we expose a low-level API and people can add entire folders to the import map, the filenames need to be hardcoded in the JavaScript files, where we cannot make them unique:

My point was not to make the API to allow folders in import maps but to make a low-level API that would interact with the import map more directly. Including allowing solving versions in a self-descriptive way. Adding import map entry can be used precisely to map a file name, to the versioned one:

// PHP
gutenberg_register_import_map_entry(
  "@automatewoo/some/file.js",
  "/wp-content/plugins/automatewoo/build/some/file" . get_version()
);

// JS
import { something } from "@automatewoo/some/file.js";

But forcing every plugin author to use that complex build configuration doesn't seem like a good idea, especially when people are already used to delegating that management to WordPress.

I cannot agree more. Especially, that as a Woo extension developer, I feel the pain of a specific build process being forced on me.
I hope, with a low-level API, we could allow users to either use it to adapt to their own build processes, as well ad delegating that to WordPress either by writing a bit more code, or using the higher-level API that is about to come as well.

Re: Filename Backward Compatibility

By hardcoding file paths in the JavaScript files, we would lock them into those specific paths/names. In the future, if someone decides to rename or relocate a file, any JS code that references the old path/name will break.

My proposal did not introduce any more hardcoding paths in JS than the original.
You can as well do or not do it with both APIs:

// Hardcode path with original API
gutenberg_register_module(
		'@automatewoo/source/folder/file.js',
		'/wp-content/plugins/automatewoo/source/folder/file.min.js'
);
// JS
import { utilFunction } from "@automatewoo/source/folder/file.js";

// Do not hardcode with the proposed one.
gutenberg_register_import_map_entry(
		'@automatewoo/library/block,
		'/wp-content/plugins/automatewoo/build/blocks/folder/file.min.js'
);

One can argue, that hardcoding JS file paths in PHP is equally a bad practise, as it makes the code hard to maintain.
Paraphrasing: "if someone decides to rename or relocate a JS file, any PHP code that references the old path/name will break".
So we are just talking about moving the responsibility between areas.

Speaking of a "platform where code ownership is divided among different people … which affects how we need to think about these APIs" When Modules were introduced to the Web ;) They were also introduced without bare identifiers, separately from import maps, AFAIK to allow a faster implementation and more flexibility. Maybe we can follow that path.

Re: Dependency Preloading

That's the problem the API proposed by me will definitely not help to solve. But well, that's the point of being "low-level" it does not have to include all the features and solve all the cases.

Whereas

that will make it difficult to adopt it in more complex implementations

Is definitely an important point.

However, personally, I'm not a fan of handling the dependency graph ourselves. It's pretty complex and expensive work. It means we'd be re-implementing in PHP what JS engines already do. (I see a point and value, of preloading, and don't have anything better to propose. Maybe except of employing actual already made implementation).
It also puts a lot of responsibility on consumers of their API.

---

I also see the importance of "it might give the wrong incentives to people". So whatever we actually ship should support the right incentives and maintainability. But are when there already?

---

Said all the above, I see a list of valid reasons not to follow my proposal. I just hope that our high-level-first approach will not prematurely limit WP Platform and community possibilities, innovation, and it will not actually impose too many constraints and inconveniences on the developers.

If we are already trying to support high-level things like "issues regarding cache management, backward compatibility, and dependency preloading.". I'd strongly vote to add the dependency extraction and its exceptions & extensions to the list.

In my understanding import maps, by design are meant to solve the problems DEWP was created to solve.
In my opinion, import maps solve them in straight forward, and easy-to-explain way.
Whereas, the way DEWP works and is configured is still a confusing surprise to some of the folks from my team. Smart, experienced web developers, with a few years of tenure in WooCommerce, and they still struggle to get it. I can only imagine how hard it is for 3PDs.

It's more sustainable to expect developers in the WordPress ecosystem to understand the native Web feature of import maps. Instead of forcing them to use Webpack, with our custom, complex build process, and expect to learn our own tool.

Plus, we would not have to document, and maintain it ;)

[gziolo]

This is now part of WordPress core 🎉

The final approach is documented in the dev note: Script Modules in 6.5.