In order to reach an MVP where we are able to get some kind of return on our engineering investments, we'll need an initial benchmark runner, initial set of candidate programs, and finally a simple analysis that can tell us whether a change is statistically significant or not. This is enough to be useful for some ad-hoc local experiments. The MVP does not mitigate measurement bias, it does not have a representative and diverse corpus, and it does not integrate with GitHub Actions or automatically detect regressions on the main branch.

See the RFC for more details.