Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Choose one model from the Pareto front as our champion #35

Open
jmmcd opened this issue Apr 4, 2019 · 0 comments
Open

Choose one model from the Pareto front as our champion #35

jmmcd opened this issue Apr 4, 2019 · 0 comments
Assignees

Comments

@jmmcd
Copy link
Collaborator

jmmcd commented Apr 4, 2019

FFX creates a Pareto front of good models, trading off numBases against accuracy. For some applications we would like to be able to put up just one model as the champion, for example when FFX is being benchmarked against other techniques for symbolic regression, eg https://github.com/EpistasisLab/regression-benchmark.

Our current option (see api.py/FFXRegressor) is just to choose the model of highest accuracy/highest numBases. But there are at least two other options:

  1. Try to find an "elbow" in Pareto front (idea: we are willing to give up a little accuracy for simplicity)
  2. reserve some training data to use as a validation set, and choose the model with best accuracy on the validation set (idea: the more complex models may be overfitting and our goal is to avoid that).
@jmmcd jmmcd self-assigned this Apr 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant