Test Reproduction Issue using command from a Closed Issue

[abhiPL07]

Hello, I'm trying to reproduce the test from a closed issue.
When I try to run the following:

python src/main.py --output_dir experiments --comment "pretraining through imputation" --name pretrained --records_file Imputation_records.xls --data_dir "datasets/Monash_UEA_UCR_Regression_Archive/AppliancesEnergy/" --data_class tsra --pattern TRAIN --val_ratio 0.2 --epochs 700 --lr 0.001 --optimizer RAdam  --pos_encoding learnable --num_layers 3  --num_heads 16 --d_model 128 --dim_feedforward 512 --batch_size 64

I get the following error:

2023-10-16 01:00:54,681 | INFO : Loading packages ...
2023-10-16 01:00:57,006 | INFO : Stored configuration file in 'experiments/pretrained_2023-10-16_01-00-57_vCC'
2023-10-16 01:00:57,006 | INFO : Running:
src/main.py --output_dir experiments --comment pretraining through imputation --name pretrained --records_file Imputation_records.xls --data_dir datasets/Monash_UEA_UCR_Regression_Archive/AppliancesEnergy/ --data_class tsra --pattern TRAIN --val_ratio 0.2 --epochs 700 --lr 0.001 --optimizer RAdam --pos_encoding learnable --num_layers 3 --num_heads 16 --d_model 128 --dim_feedforward 512 --batch_size 64

2023-10-16 01:00:58,017 | INFO : Using device: cuda
2023-10-16 01:00:58,018 | INFO : Loading and preprocessing data ...
119it [00:03, 39.15it/s]
2023-10-16 01:01:01,578 | INFO : 76 samples may be used for training
2023-10-16 01:01:01,578 | INFO : 19 samples will be used for validation
2023-10-16 01:01:01,578 | INFO : 0 samples will be used for testing
2023-10-16 01:01:01,597 | INFO : Creating model ...
/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/transformer.py:282: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer was not TransformerEncoderLayer
  warnings.warn(f"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}")
2023-10-16 01:01:01,639 | INFO : Model:
TSTransformerEncoder(
  (project_inp): Linear(in_features=24, out_features=128, bias=True)
  (pos_enc): LearnablePositionalEncoding(
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (transformer_encoder): TransformerEncoder(
    (layers): ModuleList(
      (0-2): 3 x TransformerBatchNormEncoderLayer(
        (self_attn): MultiheadAttention(
          (out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True)
        )
        (linear1): Linear(in_features=128, out_features=512, bias=True)
        (dropout): Dropout(p=0.1, inplace=False)
        (linear2): Linear(in_features=512, out_features=128, bias=True)
        (norm1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (norm2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (output_layer): Linear(in_features=128, out_features=24, bias=True)
  (dropout1): Dropout(p=0.1, inplace=False)
)
2023-10-16 01:01:01,640 | INFO : Total number of parameters: 619544
2023-10-16 01:01:01,641 | INFO : Trainable parameters: 619544
2023-10-16 01:01:02,995 | INFO : Evaluating on validation set ...
Traceback (most recent call last):
  File "src/main.py", line 307, in <module>
    main(config)
  File "src/main.py", line 235, in main
    aggr_metrics_val, best_metrics, best_value = validate(val_evaluator, tensorboard_writer, config, best_metrics,
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/src/running.py", line 191, in validate
    aggr_metrics, per_batch = val_evaluator.evaluate(epoch, keep_all=True)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/src/running.py", line 346, in evaluate
    predictions = self.model(X.to(self.device), padding_masks)  # (batch_size, padded_length, feat_dim)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/src/models/ts_transformer.py", line 240, in forward
    output = self.transformer_encoder(inp, src_key_padding_mask=~padding_masks)  # (seq_length, batch_size, d_model)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/transformer.py", line 387, in forward
    output = mod(output, src_mask=mask, is_causal=is_causal, src_key_padding_mask=src_key_padding_mask_for_layers)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/dblab/pal/Documents/pyscripts/mvts_transformer/mvtsenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'is_causal'

Following is the data in datasets/Monash_UEA_UCR_Regression_Archive/AppliancesEnergy:

• AppliancesEnergy_TEST.ts
• AppliancesEnergy_TRAIN.ts

These are my package versions:

Package                  Version
------------------------ ------------
absl-py                  2.0.0
asttokens                2.4.0
backcall                 0.2.0
cachetools               5.3.1
certifi                  2023.7.22
charset-normalizer       3.3.0
contourpy                1.1.1
cycler                   0.12.1
decorator                5.1.1
executing                2.0.0
filelock                 3.12.4
fonttools                4.43.1
fsspec                   2023.9.2
google-auth              2.23.3
google-auth-oauthlib     1.0.0
grpcio                   1.59.0
idna                     3.4
importlib-metadata       6.8.0
importlib-resources      6.1.0
ipdb                     0.13.13
ipython                  8.12.3
jedi                     0.19.1
Jinja2                   3.1.2
joblib                   1.3.2
kiwisolver               1.4.5
Markdown                 3.5
MarkupSafe               2.1.3
matplotlib               3.7.3
matplotlib-inline        0.1.6
mpmath                   1.3.0
networkx                 3.1
numpy                    1.24.4
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.18.1
nvidia-nvjitlink-cu12    12.2.140
nvidia-nvtx-cu12         12.1.105
oauthlib                 3.2.2
packaging                23.2
pandas                   2.0.3
parso                    0.8.3
patsy                    0.5.3
pexpect                  4.8.0
pickleshare              0.7.5
Pillow                   10.1.0
pip                      23.0.1
prompt-toolkit           3.0.39
protobuf                 4.24.4
ptyprocess               0.7.0
pure-eval                0.2.2
pyasn1                   0.5.0
pyasn1-modules           0.3.0
Pygments                 2.16.1
pyparsing                3.1.1
python-dateutil          2.8.2
pytz                     2023.3.post1
requests                 2.31.0
requests-oauthlib        1.3.1
rsa                      4.9
scikit-learn             1.3.1
scipy                    1.10.1
setuptools               56.0.0
six                      1.16.0
sktime                   0.4.1
stack-data               0.6.3
statsmodels              0.14.0
sympy                    1.12
tabulate                 0.9.0
tensorboard              2.14.0
tensorboard-data-server  0.7.1
threadpoolctl            3.2.0
tomli                    2.0.1
torch                    2.1.0
tqdm                     4.66.1
traitlets                5.11.2
triton                   2.1.0
typing_extensions        4.8.0
tzdata                   2023.3
urllib3                  2.0.6
wcwidth                  0.2.8
Werkzeug                 3.0.0
wheel                    0.41.2
xlrd                     2.0.1
xlutils                  2.0.0
xlwt                     1.3.0
zipp                     3.17.0

Python version is 3.8.18.
Would appreciate it if anyone could suggest some changes to the code.

[VirtualVsMachine]

I received the same. I deleted the parameter is_causal = is_causual from the function call and it worked.

[aayushukla]

Hi, set is_causal=None as argument to TransformerBatchNormEncoderLayer.forward. This made the code work for me.