Make module system more robust

[jeaye]

More details to be filled in soon.

JIT

Add dynamic var for loaded libs

Clojure uses a clojure.core/*loaded-libs* dynamic var, which contains a set of loaded libs. It's initialized in clojure.core, like so:

(defonce ^:dynamic
  ^{:private true
     :doc "A ref to a sorted set of symbols representing loaded libs"}
  *loaded-libs* (ref (sorted-set)))

We can do the same with jank but make it an atom and leave a TODO for ref usage. Then follow all of the usages of *loaded-libs* in clojure.core and make sure jank is doing the same work in the various functions.

Add cyclical dep check

Clojure also uses another dynamic var, *pending-paths*, to track cyclical deps. It's defined like so:

(defonce ^:dynamic
  ^{:private true
     :doc "A stack of paths currently being loaded by this thread"}
  *pending-paths* ())

We'll want to add the same thing, along with the check-cyclic-dependency function and its usages. Manually test this to ensure it's working well.

Add transactionality

Based on my testing, Clojure will still mark a namespace as loaded if you throw an exception while requiring it. This means we don't need anything clever other than thread safety when updating the loaded libs set. Please do your own testing (check loaded libs, try to load a ns which isn't loaded, have it fail, and then check again) and ensure jank matches Clojure's behavior. The loaded libs var is private, but you can get to it like so:

(deref (intern 'clojure.core '*loaded-libs*))

Again, an atom will do fine here, as far as I can tell. I think it'd be good to ask about why a ref is used here, compared to an atom, in the Clojurian Slack; we might learn something neat.

Add ability to look up module by source

I have recently added an origin enum to load_module. It allows us to explicitly load from source or load from the latest, where latest is either binary or source depending on timestamps (timestamp checking not yet implemented).

For future functionality, it would be helpful to extract some of the behavior we have in load_module into a find_module function which also takes an origin. It can then return some data containing both the entry and which part of the entry should be used (based on the origin).

Add reloading support

When a pass :reload along to require, we want to load the module from the latest origin again, even if it's already loaded. This will require passing in a flag to ignore the early exit for skipping modules which are already loaded. This also needs to work with :reload-all, which reloads the specified module and all of its dependencies from their latest origins. Look into how Clojure does this in the load-lib function.

AOT

Ensure module dependencies are compiled and loaded properly

We can AOT compile a module to an object file right now. If that module requires other modules, we need to fork off and compile each of them into their own object files. This has not been tested and may not be working. Start simply, with a.jank and b.jank, where a requires b. Then tell jank to compile a with jank compile a (ensure that a.jank is on the module path).

Firstly, we want to be sure that both a and b get generated separately, with the correct LLVM IR modules. You can comment out the print in runtime::context::write_module to see the IR that's getting written for each module.

Secondly, we want to be sure that when we load a, b gets loaded as well. You can verify this by putting a println at the top of both of them, starting a jank repl, and requiring a. You should see both prints.

Load binaries only if the source isn't newer

When loading a module and the origin is latest, we can consider binaries for loading. If a binary is present, we need to check its timestamp against the source file. If the source file is missing, we need to not load the binary, since we always require source distributions. If the binary is newer or at least as new as the source, we can load the binary. If the source is newer, we need to load the source. For timestamps, we want to check the last modified time.

Skip module compilation based on timestamp

When compiling, if a module has a binary which has a sufficient timestamp, we can skip compilation of the source.

[Samy-33]

@jeaye, I am on this.