Question regarding evaluation dataset and paper results

[zeyihou]

Thank you for your great work!
From the provided YAML files for the evaluation dataset, it appears that only the last prediction_length of each dataset is used for evaluation. This means the evaluation is based on a very small portion of the dataset (i.e., the last segment).
For example, if prediction_length is 24 and the dataset length is 1000, only the last 24 time steps are used. Are the results reported in the paper based on this testing method (i.e., using only the last prediction_length)

[lostella]

@zeyihou yes, results in the paper are based on this setup

[zeyihou]

Thank you for your prompt reply!
I still have some questions regarding this setup. If the last segment window is used for testing instead of validating across the entire test set, how can you ensure the generalizability of the results? Did the other alternatives used for comparison also employ the same evaluation setup in your paper?

[lostella]

Did the other alternatives used for comparison also employ the same evaluation setup in your paper?

Yes, all models were evaluated with the same setup (otherwise the comparison would not be meaningful)

how can you ensure the generalizability of the results?

I’m not sure I understand, could you be more specific?