Unable to reproduce univariate and multivariate time series forecasting

[jannikbach]

Hello everyone,

I was trying to replicate the results from the paper of Table 13 and 14. Therefore I used the configurations of configs/experiment/forecasting/s4-informer-etth.yaml and adjusted the prediction window inside /configs/dataset/etth.yaml to the values found in the table.
Particularly I changed the third parameter in "size" inside configs/dataset/etth.yaml, e.g.

size:
  - 384
  - 96
  - 24

For this setup I got an final MSE of 0.2863. The paper states a MSE of 0.061.
For all the setups the MSE values I got never came close to the ones stated in the paper.

Was someone able to reproduce the results and can share the configuration to the setup?

Best Regards

[albertfgu]

Try using dataset.size=[720,168,24] and model.timeenc=1. Seems like our old results use these settings. Sorry it isn't better documented.

[maisuiqianxun]

I changed the default setting in "configs/dataset/etth.yaml" from
size = [384, 96, 96], timeenc = 0 to size = [720,168,24] timeenc = 1. But it still cannot obtain the expected 0.061. Instead, the obtained min val MSE is 0.074 in epoch 0. The corresponding test MSE is 0.095.
@jannikbach Do you repeat the MSE of 0.061?

[jannikbach]

When I used the configs @albertfgu provided, I came close to the reported test MSE of 0.061.
But it was also in the first epoch. The dataset is very prone to overfitting.

The following run shows my results. Unfortunatley I was using a visualisation pytorch-lightning callback, I wrote, but was made for another usecase and does not work with this data. So the run crashed but the results were logged nevertheless. https://wandb.ai/jannik-bach/hippo/runs/37cp8mpr?workspace=user-jannik-bach

[maisuiqianxun]

ok, i will recheck my settings, thanks.