Skip to content

roadmap v1

Jump to bottom
Jeremie Dimino edited this page Apr 16, 2019 · 5 revisions

ppxlib 1.0.0 roadmap

This document is a work in progress.

This document describes the roadmap for ppxlib 1.0.0. The goal is to make ppxlib a good foundation for the ppx ecosystem, as currently the ppx stack is complicated and not very resilient to compiler upgrades.

The expectation is that Ppxlib 1.0.0 will effectively supersede everything else. In particular, all other projects such as ocaml-migrate-parsetree, ppx_deriving, ppx_tools, ppx_tools_versioned and ppxlib 0.x.

As such, we must prepare ppxlib to be a solid foundation for the whole ppx ecosystem, and soothe the rough edges that might get in the way of its adoption. The main difficulty is backward compatiblity, and this document describes how we are going to solve this problem.

While all of this happen, we will also communicate the plan and advancement on discuss.ocaml.org so that people who care about ppx can follow what's happening, understand how this work will affect them and what benefit they can expect from it.

If you have any questions or concerns about this plan, please contact @diml.

Backward compatiblity

Currently, ppxlib uses one selected version of the AST, for instance 4.07. When OCaml 4.08 comes out, the following events happen:

  • ocaml-migrate-parsetree learns how to convert between the 4.07 and 4.08 AST. This allows ppxlib to work with OCaml 4.08 and code using ppxlib to build against 4.08, however users of ppx rewriters cannot use the new 4.08 features as the 4.07 AST cannot represent them
  • ppxlib is updated to use the 4.08 AST. This is a breaking change
  • all authors of ppx rewriters have to upgrade to the new version of ppxlib, based on the 4.08 AST. If they are lucky, they don't need to upgrade

There are several issues with this approach:

  • it is a lot of churn for a lot of people
  • we can't test the new versions of OCaml until all of the above has happened

The plan is to arrive in a situation where:

  • ppxlib builds at all time, even against OCaml trunk
  • existing OCaml files are always parsed and preprocessed the same way, allowing to test OCaml trunk at any time

We plan to achieve this by the introduction of a new Astlib library that will sit between the compiler libraries and ppxlib.

Astlib

Astlib will be a small connector library between the compiler and ppxlib. It will provide a minimal stable API for ppxlib to rely on. The plan is to make it live in the compiler directly. However, in order to make the transition to the new ppx world easier for users, we will backport this library to older compilers.

Effectively, we are going to create a repository github.com/ocaml-ppx/astlib. This reposiotry will contain a package astlib-v1 and a library astlib-v1 exposing an Astlib_v1 toplevel module. When building this library, we will inspect the current compiler version. If it is a compiler that provides Astlib_v1, then the astlib-v1 library will just redirect to compiler library. Otherwise, we will use the implementation embed in the reposiotry.

API of Astlib

Astlib will expose the following functionality:

  • parsers for a few public entry point of the grammar: structure, signature, expression, ...
  • printers for the same entry points
  • a stable AST

The stable AST

The stable AST exposed by Astlib will be an AST that is different from the one used inside the compiler. The goal is that a given file parses to exactly the same AST value, not matter the version of the compiler.

While this AST must be stable, it must also be extensible so that new versions of OCaml can add new features.

To achieve that, we will follow the following rules:

  • all types will be variants types. Records will only be allowed as inline records (except if we want 4.02 compatibility, to be discussed)
  • any AST fragment will always be represented in the oldest possible way

To understand better the last point, let's consider the following change that happened in a recent version of the compiler. In the past, the grammar of let open <x> in <y> was:

let open <identifier> in <expression>

Now it is:

let open <module-expr> in <expression>

Let's assume the old expression was represented as Let_open_in of ident * expression. In order to represent the new forms, we would add a constructor Let_open_in_expr of module_expr * expression. However, we would forbid values of the form:

Let_open_in_expr (Ident id, e)

Instead, these should represented as:

Let_open_in (id, e)

This will ensure that existing OCaml files wouldn't be affected by this change and would still parse the same way that they used to. To enforce this, all the types will be private and Astlib_v1 will provide functions to construct ast values.

If one day we really want to change the types in a way that doesn't fit in this scheme, we will mint Astlib_v2 and re-implement Astlib_v1 on top of Astlib_v2.

Astlib and Ppxlib

Ppxlib will use the AST provided by Astlib directly. In addition, it will provide viewers (as in ppx_view viewers) for convenience so that users do not have to deal with all versions of a constructor.

When traversing the AST, Astlib will fail when it encounters a constructor it doesn't know about. This will ensure that it doesn't silently ignore parts of a file because it cannot interpret the new features.

Effectively, when the language evolves, the following will happen:

  • a new version of OCaml will be released
  • we will update ppxlib to handle the new constructors
  • we will make a new release of ppxlib
  • users will be able to enjoy both the new language features and ppx rewriters

During all this time, existing OCaml files will still be interpreted and preprocessed in exactly the same way they used to.

Ppxlib 1.0.0

Ppxlib 1.0.0 will be released as a package ppxlib-v1. This package will contain two libraries: Ppxlib_unstable, which will be the current Ppxlib library, and Ppxlib_v1 which will be a thin layer on top of Ppxlib_unstable exposing a stable API. This will allow us to do breaking changes in the future without breaking the world. This will also allow ppxlib 0.x and ppxlib-v1 to co-exist, allowing authors of ppx rewriters to gradually upgrade.

Dependencies

The current dependencies of ppxlib are:

  • dune
  • base
  • stdio
  • ocaml-compiler-libs
  • ocaml-migrate-parsetree
  • ppx_derivers

The new dependencies will be:

  • dune
  • astlib-v1

Dropping the dependency on ocaml-migrate-parsetree

This means that users will not be able to mix ppx rewriters based on ocaml-migrate-parsetree and ppxlib-v1. This is expected as we expect that ocaml-migrate-parsetree will eventually disappear.

Dropping the dependency on ppx_derivers

This library existed to allow users to mix derivers based on ppx_deriving and derivers based on ppxlib 0.x. Users of ppx_deriving will be able to switch to Ppxlib_v1.Deriving, which offer a similar functionality.

Dropping the dependency on Base

We will drop the dependency on Base as I believe it would slow down the adoption of ppxlib-v1. Indeed, so far Base hasn't convinced a everybody, and I have seen people who didn't want to depend on ppxlib because of Base.

This will also make things easier for Jane Street developers as currently the situation between Base and ppxlib is a bit complicated. Indeed, Base is developed inside Jane Street, ppxlib is developed on github and ppxlib depend on Base. This causes a very weird workflow for people working on Base and/or ppxlib.

Instead, we will use the OCaml standard library and have a small overlay inside ppxlib to add the few things that are missing, a bit like what is done in dune.

Preparing Ppxlib_v1

In Ppxlib_v1, we will expose only the functionality that we are ready to support for a long time. In particular, we should hide all functions that are only exposed for the purpose of other modules of ppxlib.

We must start with the minumin amount of features and add them one by one as they are needed.

Roadmap

Items under the same bullet can be done in parallel

  • preparation:
    • write github.com/ocaml-ppx/astlib
    • drop dependency on ocaml-migrate-parsetree and ppx_derivers
    • drop dependency on base and stdio
  • delete ppxlib.ast (ast folder in ppxlib) and use astlib-v1 instead
  • rename the ppxlib package to ppxlib-v1
  • rename the ppxlib library to ppxlib-v1.unstable
  • write the Ppxlib_v1 API
  • release ppxlib-v1 1.0.0
Clone this wiki locally