Tackling imbalance

[manumacc]

This is a very imbalanced problem, as the number of samples with reply is much lower than the number of samples without any reply (the same applies to retweet, retweet with comment and, to a lesser extent, like).

Possible approaches: see https://stats.stackexchange.com/questions/28029/training-a-decision-tree-against-unbalanced-data

• Undersampling or oversampling (probably best to undersample the majority class since we have a huge amount of data, instead of introducing synthetic data? Either way, needs testing)
• Modify cost of misclassification
• Balanced accuracy metric (standard accuracy is a pretty bad choice here)

Keep in mind that over/undersampling applies to the training set only, to reduce bias towards the majority class. Validation and test should be kept imbalanced.