Build a run evaluation engine in Python

[PGijsbers]

The evaluation engine is a component on the server which handles multiple tasks. This is currently implemented in Java and we want to rebuild it in Python, and compartmentalised per each function, for easier maintenance/more accessible to new contributors. One of its tasks is evaluating run results.

So we want an engine which can take in any run result, and produce a number of metrics of those results. It should be easily extendable towards new task types, and cover many (all?) of the currently available metrics - or at least ensure that those that share a name produce identical results. It would be best to have a base implementation that could be inherited from for separate evaluation engines that are specific to a task type.

[joaquinvanschoren]

This is a nice standalone project, assuming we can build this on top of the Python API. What would make a lot of sense is to sit together for an hour during the hackathon to design the overall architecture and concrete next steps.