Question of training on conversational datasets

[qibin0506]

I trained with 1000 pieces of conversational data, and the loss was very small, but the effect was poor. I took 10 of data for training, and after about 10 epochs, the effect was very good. What is the reason for this and how to solve it?

My config like this

CFG = {
    'vocab_size': 21128,
    'ctx_len': 512,
    'embed_dim': 768,
    'n_heads': 12,
    'n_layers': 12,
    'drop_rate': 0.1
}

[rasbt]

Hi @qibin0506 ,

Let's see what's going on here... first, a few questions:

1. You mention

I took 10 of data for training,

does this mean you only had 10 training examples? In this case, when you say the loss was very small, do you mean the training or validation loss? One possible explanation is that with only 10 training examples, you model was overfitting to the training data a lot.

2. Looking at

'vocab_size': 21128,

it seems you have decreased the vocabulary size. Did you still use the gpt-2 tokenizer? If you change the vocabulary, you'd have to create a new tokenizer and pretrain the model with it before you do finetuning. This could be another explanation that would explain the bad performance.

[qibin0506]

Thank you for your reply @rasbt .

1. Is training loss, and I also think it was overfitting, but when I train on a large dataset, the loss may also be small, but the effect was poor.
2. Yes, I am using BERT's tokenizer, and I pretrained on this dataset from scratch.

[rasbt]

Is training loss, and I also think it was overfitting, but when I train on a large dataset, the loss may also be small, but the effect was poor.

Do you have a portion of the data for validation? How do the training/validation loss curves look like?

Yes, I am using BERT's tokenizer, and I pretrained on this dataset from scratch.

I think in this case the dataset of 10k conversational examples may be too small for pretraining. Unless you used a different dataset for pretraining?

[qibin0506]

OK, maybe I need to pretrain on large document dataset and then fine tune on the conversation dataset, right?

[rasbt]

Yes, that would be one day. Or, alternatively, I suggest starting with a pretrained model.