Layer 0 Spec: Module & Module Source

[caridy]

This PR define the specs for layer 0.

The readable version of this PR can be found here.

Details

1. Introduces Module
2. Introduces ModuleSource
3. Modifies Runtime Semantics for import() to accept a Module instance

[Jack-Works]

This algr looks very similar to ParseModule. Can you use <ins> and <del> to indicate the changes?

[caridy]

I have added a note below this operation related to this. I don't it will be a modifcation, but a refactor that splits ParseModule into two operations.

[Jack-Works]

Can we do ? Call (%Object.assign%, *null*, < _importMeta_, _meta_ >)?

[caridy]

being more conservative here since we don't use Object.assign anywhere.

[Jack-Works]

Why it will call the host hook? This will prevent virtualization if the host did something observable.

[caridy]

HostFinalizeImportMeta: https://tc39.es/ecma262/#sec-hostfinalizeimportmeta

Most hosts will be able to simply define HostGetImportMetaProperties, and leave HostFinalizeImportMeta with its default behaviour. However, HostFinalizeImportMeta provides an "escape hatch" for hosts which need to directly manipulate the object before it is exposed to ECMAScript code.

Basically, the creation of the meta object involves two steps, one of them needs virtualization (HostGetImportMetaProperties), the other one HostFinalizeImportMeta doesn't.

[Jack-Works]

No, since it does not call the HostGetImportMetaProperties, it should not call HostFinalizeImportMeta either.

[caridy]

@nicolo-ribaudo @Jack-Works, I have two issues here:

1. if the result of calling [[ImportHook]] function must always be a Module Instance, then how can we resolve to fs or some other module instance that delegates the resolution to the browser? That seems like a problem. I don't know if the issue is here, or somewhere else, but definitely needs to be addressed.
2. where do we go into the recursive resolution? in HostImportModuleRecordDynamically we only talk about the "unique ModuleSpecifier associated to the moduleRecord", never about the transitive dependencies. any suggestion?

[guybedford]

One option here might be to support having import hook return a ModuleSource which is then instanced against its primary default instance for the host context? Would need to think through the edge cases though....

Recursive resolution happens through HostResolveImportedModule calls in InnerModuleLinking and InnerModuleEvaluation. I really liked @nicolo-ribaudo's approach in https://nicolo-ribaudo.github.io/modules-import-hooks-refactor/ of having this being a sync lookup against the resolution list on the module record itself though to more clearly separate the sync and async iterations.

[nicolo-ribaudo]

1. My hope is that import reflection could also give Modules other than ModuleSources (the syntax is just an idea):

   import module jsM from "x.js";
   import source jsS from "x.js";
   import module wasmM from "y.wasm";
   import source wasmS from "y.wasm";
   
   jsM instanceof Module;
   jsS instanceof ModuleSource;
   wasmM instanceof Module;
   wasmS instanceof WebAssembly.Module; // WebAssembly.Module respects the "module source interface", whatever that means
   
   jsM.source === jsS;
   wasmM.source === wasmS;

   Then, an import hook might be implemented like this:

   function importHook(specifier, importMeta) {
     if (specifier === "fs") return import("fs", { reflection: "module" });
     
     const source = // get the module source
     const innerImportMeta = // get the import.meta
     return new Module(source, importHook, innerImportMeta);
   }

   @guybedford The problem is that, at least for built-in modules, we might not want a ModuleSource (module.source should be null) because you wouldn't want to be able to, for example, create multiple instances of a "fs" module.

2. As @guybedford said, the current ECMA-262, we explicitly handle the recursive resolution: we hide the fact that "real resolution" can be asynchronous, and we pretend that the resolution happens in HostResolveImportedModule. We call it recursively for every transitive dependency in InnerModuleLinking, which is triggered by an (implicit) call to .Link() of the top-level imported module in HostImportModuleDynamically.
   We probably can keep hand waving in this proposal spec (we need a "complete spec text" only when asking for stage 3, and this PR is already very good for asking for stage 2), but a proper solution will look a lot like https://nicolo-ribaudo.github.io/modules-import-hooks-refactor/#sec-LoadRequestedModules (or, I push for merging https://nicolo-ribaudo.github.io/modules-import-hooks-refactor without waiting for a proposal, and then you just edit that in this proposal to call [[ImportHook]] instead of HostLoadImportedModule)

[guybedford]

@nicolo-ribaudo on (1), this is an interesting proposal although does introduce more complexity. That said it very much resolves some current concerns through the separation. Would the module instance reflection be unexecuted if it hasn't already been executed? But would the dependencies be fully preloaded? If so, that may well make it suitable for deferred evaluation mechanisms....

Furthermore, passing an instance to another realm in this way with linkage might enable an easy solution to resolution sharing without having to re-instrument a loader on the other realm...

Would be interested to hear more details if it might be able to address the above.

[guybedford]

Seems roughly on the right track to me.

[guybedford]

Is it possible, to overload HostImportModuleDynamically to accept either a module source or a string specifier as it's "specifier"?

[guybedford]

Perhaps add something like a note here on the topic that the Instance constructor will create a bi-directional mapping between an instance object and its corresponding SourceTextModule record, while internal SourceTextModule construction leaves this undefined forming a distinction between user-instrumented and non-user-instrumented code?

[guybedford]

Should this be [[Module]] or [[ModuleSourceInstance]]?

[guybedford]

Also, should we also be looking up if _referencingScriptOrModule_ is itself a SourceTextModule (pretty sure we can assume it always be?) that we should look up its [[Instance]] slot in turn for the parent reference if there is one? Would that be useful information for an import hook even at the top level?

[kriskowal]

Pardon this late response; I’m sure our shared understanding has evolved in 22 days. I believe we’re expecting Module instances to represent not just modules constructed by the Module constructor, but any reflected module and any module block, which suggests they may or may not have a [[ModuleSourceInstance]], that they will always have a [[Module]] Module Record. We can only be sure that [[Module]] is a Source Text Module Record if the Module Instance has a [[ModuleSourceInstance]].

We may want to make that invariant explicit.

[caridy]

Should this be [[Module]] or [[ModuleSourceInstance]]?

It should be [[Module]] because the instance provided must already have a qualifying module record associate to it, and that's the currency of all these operations.

[caridy]

We may want to make that invariant explicit.

Any suggestion on how to make that more explicit?

[guybedford]

One option here might be to support having import hook return a ModuleSource which is then instanced against its primary default instance for the host context? Would need to think through the edge cases though....

Recursive resolution happens through HostResolveImportedModule calls in InnerModuleLinking and InnerModuleEvaluation. I really liked @nicolo-ribaudo's approach in https://nicolo-ribaudo.github.io/modules-import-hooks-refactor/ of having this being a sync lookup against the resolution list on the module record itself though to more clearly separate the sync and async iterations.

[erights]

At https://raw.githack.com/tc39/proposal-compartments/e897e4a5707154f4af67ac89ec7bf7cfef966636/0-module-and-module-source.html 4.1 step 7 I see

If 7.b completes abruptly, shouldn't 7.c be skipped?

[erights]

Should https://raw.githack.com/tc39/proposal-compartments/e897e4a5707154f4af67ac89ec7bf7cfef966636/0-module-and-module-source.html specify that ModuleSource.prototype.constructor is the ModuleSource constructor? Or is that taken care of by a generic preamble?

[erights]

Step 3

3. If Type(importHook) is not Callable, throw a TypeError exception.

seems redundant with step 6

6. If IsCallable(importHook) is false, throw a TypeError exception.

[nicolo-ribaudo]

@erights

If 7.b completes abruptly, shouldn't 7.c be skipped?

IfAbruptRejectPromise has an early return if the completion is an abrupt completion, so 7.c is skipped.

[caridy]

@erights

Should https://raw.githack.com/tc39/proposal-compartments/e897e4a5707154f4af67ac89ec7bf7cfef966636/0-module-and-module-source.html specify that ModuleSource.prototype.constructor is the ModuleSource constructor? Or is that taken care of by a generic preamble?

it was missing for both, I just added.

Step 3

3. If Type(importHook) is not Callable, throw a TypeError exception.

seems redundant with step 6

6. If IsCallable(importHook) is false, throw a TypeError exception.

Fixed.

[naugtur]

I made an attempt to visualize this change for the benefit of my own mental model. Sharing in case it's useful:
https://gist.github.com/naugtur/daf2aaf415e03e6c3ae5030e06f7b20e

[caridy]

I made an attempt to visualize this change for the benefit of my own mental model. Sharing in case it's useful: https://gist.github.com/naugtur/daf2aaf415e03e6c3ae5030e06f7b20e

Very nice, I added a note in the gist.

[nicolo-ribaudo]

We have always discussed about a string referrer, but this spec text allows referral to be any ECMAScript value. I very much prefer this behavior, rather than restricting to strings (I was going to open a new issue proposing to relax the restriction, before noticing that it's already not restricted).

I realized that passing the referrer to importHook (and thus passing referral to new Module) is not necessary, but it's just an ergonomic improvement. If the spec didn't call importHook passing the referrer, I could just do this:

function buildImportHook(referrer) {
  return async function importHook(specifier) {
    const url = new URL(specifier, referrer);
    const sourceText = await fetch(url).then(r => r.text());
    const source = new ModuleSource(sourceText);
    return new Module(source, buildImportHook(url));
  };
};

const entryModule = new Module(entrySource, buildImportHook(entryURL));
await import(entryModule);

(or I could use .bind rather than manual currying)

Passing the referrer through the Module constructor to importHook is just an ergonomic/performance improvement to avoid creating closures that capture context: it's similar to the thisArg parameter of all the array methods that take a function.

Since we are "just" improving ergonomics, we can look at other use cases for providing some context to the importHook function. One example (taken from the HTML spec) is to provide some "fetch options" that describe how the dependencies should be fetched:

async function importHook(specifier, { baseURL, referrerPolicy }) {
  const url = new URL(specifier, baseURL);
  const sourceText = await fetch(url, { referrerPolicy }).then(r => r.text());
  const source = new ModuleSource(sourceText);
  return new Module(source, {
    baseURL: url,
    referrerPolicy,
  }, importHook);
}

const entryModule = importHook("./main.js", { baseURL: document.URL, referrerPolicy: "same-origin" });
await import(entryModule);

// That is roughly equivalent to
//   <script type="module" src="./main.js" referrerpolicy="same-origin" />

Allowing any object aligns this "context parameter" with the current [[HostDefined]] slot on Module Records: it's a general-purpouse place used to share loader-level information between modules.

So 👍 to not checking that Type(_referral_) is string, and maybe we could rename it to _context_ (or _thisArg_ and pass it as this to importHook, like array methods do).

[caridy]

@nicolo-ribaudo in principle I agree with this. Question: in the example above I suspect you are missing importHook argument when calling new Module() right?

[nicolo-ribaudo]

Yes, fixed!

[caridy]

Resolution: We will look into this in another PR.

[Jack-Works]

importHook should be optional argument. Inherit the current evaluator's import hook.

const x = new Module(source, 'spec') // uses host
const ev = new Evaluators({ importHook: myHook })
new ev.Module(source, 'spec') // uses myHook

[caridy]

Resolution: we will look into the default hook in another PR/issue.

[caridy]

@kriskowal feel free to merge this. It seems that we are all in agreement according to the last SES meeting.

[Jack-Works]

It should call ? HostEnsureCanCompileStrings(_currentRealm_) before ParseText.

[caridy]

This is the wrong place to do so... creating an instance of module source should be possible even when CSP prevents evaluation of code. Perhaps the host hook has the wrong name... it is not really about compilation, but evaluation. I'm fixing this in #76.

[Jack-Works]

This step should do a coherent to convert sourceText to a String.

[caridy]

fixed by #76

[kriskowal]

Pardon this late response; I’m sure our shared understanding has evolved in 22 days. I believe we’re expecting Module instances to represent not just modules constructed by the Module constructor, but any reflected module and any module block, which suggests they may or may not have a [[ModuleSourceInstance]], that they will always have a [[Module]] Module Record. We can only be sure that [[Module]] is a Source Text Module Record if the Module Instance has a [[ModuleSourceInstance]].

We may want to make that invariant explicit.

[kriskowal]

I believe “referrer” is the more correct word-choice than “referral”, but that is not important yet.