Skip to content

Machine learning [how build a working model from scratch]

Jump to bottom
fab edited this page Dec 29, 2023 · 36 revisions

I built a custom DevGPT to write most of the code I'm currently publishing on GitHub and I am using it to build a model suitable for domain safety ranking from scratch.

I will use the rank score to filter out safe predicted FQDNs from the release blacklist to provide additional accuracy and reduce false positives.

Since I can feed the machine learning pipeline with fresh data anytime (dataset with millions of blacklisted and whitelisted domains and subdomains aka FQDNs), I planned to build a model to predict badness score for new submitted FQDNs. The rank score is in the 1-100 range where 1 means really safe and 100 means really bad.

I then started by using a subset of the entire dataset (25000 good + 25000 bad items instead of millions of them).

Afterthat I built a simple ensemble pipeline to find the most accurate method for training and inference.

I tested the most popular and easy-to-implement methods in this context like RandomForest, GradientBoosting, ExtraTrees, LogisticRegression and SVC:

    classifiers = {
        "RandomForest": RandomForestClassifier(random_state=42),
        "GradientBoosting": GradientBoostingClassifier(random_state=42),
        "ExtraTrees": ExtraTreesClassifier(random_state=42),
        "LogisticRegression": LogisticRegression(random_state=42, max_iter=2000),
        "SVC": SVC(probability=True, random_state=42)
    }

Let's describe all those methods one by one:

  1. RandomForest Classifier
  • Type: Ensemble Learning Method
  • Description: RandomForest is a type of ensemble learning method that constructs a multitude of decision trees during training. For classification tasks, it outputs the class that is the mode of the classes of individual trees.
    • Strengths:
      • Handles both numerical and categorical data well.
      • Robust to overfitting as it averages the results of many decision trees.
      • Good performance in a wide range of problems.
    • Weaknesses:
      • Can be less interpretable compared to a single decision tree.
      • Performance may degrade with very noisy data.
  1. GradientBoosting Classifier
  • Type: Ensemble Learning Method
  • Description: GradientBoosting builds an additive model in a forward stage-wise fashion, allowing optimization of an arbitrary differentiable loss function. It builds the model in a stage-wise fashion like other boosting methods do but generalizes them by allowing optimization of an arbitrary differentiable loss function.
    • Strengths:
      • Often provides predictive accuracy that cannot be trumped.
      • Lots of flexibility as it can optimize different loss functions and provides several hyperparameter tuning options.
    • Weaknesses:
      • Can overfit if the number of trees is too large.
      • Sensitive to noisy data and outliers.
      • Requires careful tuning of parameters and may take longer to train.
  1. ExtraTrees Classifier
  • Type: Ensemble Learning Method
  • Description: ExtraTrees (Extremely Randomized Trees) Classifier fits a number of randomized decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control overfitting.
    • Strengths:
      • Reduces variance more effectively than RandomForest by using random thresholds for each feature rather than searching for the best possible thresholds.
      • Typically faster to train than RandomForest.
    • Weaknesses:
      • Like RandomForest, can be less interpretable.
      • Might not perform well on data with strong linear relationships.
  1. Logistic Regression
  • Type: Regression-based Classifier
  • Description: Despite its name, Logistic Regression is used for binary classification problems. It models the probability of a default class (e.g., class labeled '1').
    • Strengths:
      • Simple, efficient, and easy to implement.
      • Performs well with linearly separable classes.
      • Outputs probabilities, which can be a useful feature.
    • Weaknesses:
      • Assumes linearity between dependent and independent variables.
      • Can struggle with complex relationships in data.
      • Vulnerable to overfitting if the data is highly dimensional.
  1. SVC (Support Vector Classifier)
  • Type: Kernel-based Classifier
  • Description: SVC is a powerful, versatile machine learning algorithm, capable of performing linear or nonlinear classification, regression, and even outlier detection. It is one of the best out-of-the-box classifiers.
    • Strengths:
      • Effective in high-dimensional spaces.
      • Versatile as different kernel functions can be specified for the decision function.
    • Weaknesses:
      • Can be inefficient on large datasets.
      • Requires careful tuning of parameters and selection of the kernel.
      • The choice of kernel and regularization can have a large impact on the performance of the algorithm.

The best method is choosed by performing RandomSearch instead of GridSearch:

        random_search = RandomizedSearchCV(clf, params[name], n_iter=20, cv=5, scoring='f1_macro', verbose=1, n_jobs=-1, random_state=42)
        random_search.fit(X_res, y_res)

by using all of the following parameters (I'm running this project on a Dell R620 48 cores, 128GB ram, no GPU server):

    params = {
        "RandomForest": {'n_estimators': sp_randint(100, 500), 'max_depth': sp_randint(10, 50), 'min_samples_split': sp_randint(2, 11)},
        "GradientBoosting": {'n_estimators': sp_randint(100, 300), 'learning_rate': uniform(0.01, 0.2), 'max_depth': sp_randint(3, 10)},
        "ExtraTrees": {'n_estimators': sp_randint(100, 500), 'max_depth': sp_randint(10, 50), 'min_samples_split': sp_randint(2, 11)},
        "LogisticRegression": {'C': uniform(0.01, 100), 'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']},
        "SVC": {'C': uniform(0.1, 10), 'kernel': ['linear', 'rbf', 'poly']}
    }

to find the most suitable approach.

I then focus on the elected approach to increase accuracy.

The best model:

Best model: SVC(C=7.319987722668247, probability=True, random_state=42), with parameters: {'C': 7.319987722668247, 'kernel': 'rbf'}

Let's train the model with such approach:

Loading and preprocessing data...
Vectorizing domains...
Applying SMOTE for class imbalance...
Training the SVC model...
......................................*...............*
User
optimization finished, #iter = 53354
obj = -10549.506943, rho = -0.657859
nSV = 27740, nBSV = 87
Total nSV = 27740

Here's what each part of the message means:

  • optimization finished, #iter = 53354 This shows that the optimization process within the SVM training has finished. #iter = 53354 indicates the number of iterations the algorithm took to converge. In this case, it went through 53,354 iterations.

  • obj = -10549.506943 This is the final value of the objective function that the SVM was minimizing. In SVM training, the goal is often to minimize a loss function, which in this case settled at approximately -10549.51.

  • rho = -0.657859 rho is a parameter in the decision function used by the SVM. It's involved in the calculation of the decision boundary for the classifier.

  • nSV = 27740, nBSV = 87 nSV stands for the number of support vectors. Support vectors are the data points that lie closest to the decision surface (or hyperplane) and are critical to defining the position and orientation of the hyperplane. Here, your model has 27,740 support vectors. nBSV stands for the number of bounded support vectors, which are support vectors that lie exactly on the margin of the classifier. There are 87 of them in your model.

  • Total nSV = 27740 This reaffirms the total number of support vectors used in your model.

What This Means for Your Model

  • The large number of iterations and support vectors suggest that your model is dealing with a complex, high-dimensional dataset.
  • The successful completion of training is good news, but it's crucial to evaluate the model's performance on a hold-out test set to understand how well it generalizes to unseen data.

Now that the model is trained, I should test it with my test dataset to evaluate its performance metrics (like accuracy, precision, recall, F1-score). These metrics will give a better understanding of how well the model is likely to perform in a real-world scenario.

Clone this wiki locally