"There are 14 competing standards"

[tornikeo]

I've just finished reading the beyond-jupyter, and while I appreciate the effort, I don't understand this:

How would I convince someone that writing this:

    @classmethod
    def create_logistic_regression_orig(cls):
        return SkLearnLogisticRegressionVectorClassificationModel(solver='lbfgs', max_iter=1000) \
            .with_feature_generator(FeatureGeneratorTakeColumns(cls.COLS_USED_BY_ORIGINAL_MODELS)) \
            .with_feature_transformers(DFTSkLearnTransformer(StandardScaler())) \
            .with_name("LogisticRegression-orig")

Is more maintainable, and easier to learn and extend for new developers than writing a custom fastapi->pandas->sklearn->fastapi pipeline for each new experiment? Wouldn't there be a lot less boilerplate code in that kind of codebase?

[opcode81]

Hi @tornikeo,

thanks for your interest, allow us to elaborate on the points you raised.

It is easy to maintain and extend, because

• it contains zero boilerplate. It is an entirely declarative, compact definition of a model and its input processing pipeline.
  Every line serves a very concrete purpose:
  • define the model to be used and its parameters
  • define the set of features being used
  • define the representation/transformation being used for the features
  • define the name of the model.
• it is a component in a larger code base which follows the principle of designing individual, resusable components that each serve a single, well-defined purpose, enabling all aspects of an experiment to be easily changed or extended whilst maintaining full compatibility with past experiments.

Whether it is easier to maintain and extend than a custom pipeline you design based on the libraries you mentioned depends entirely on your concrete implementation. Your implementation could be as easy to maintain or vastly more difficult to maintain. In Beyond Jupyter, we do not claim that the sensAI implementation is the only one that achieves the goals we put forth; we just showcase it as an example of one implementation that satisfies the properties we put forth.

Whether it is easier to learn is debatable and this isn't something we would claim. In fact, for someone new to the field, who is not used to thinking in abstractions, this approach has a learning curve. But what we observe is that this additional learning effort is worthwhile and is quickly amortised, as one gains flexibility, maintainability and, importantly, development speed.

Dominik & Kristof