an AttributeError occurred suddently during train with cityscapes dataset...

[shanhuhaifeng]

i downloaded the cityscapes dataset for training the model HRNet-OCR. Since data from the directory 'train_extra' is so massive, i skipped the part, and only used 2975 images and labels from the 'train' and 'val'.
the training state is ok at the begining of the procedure, but an error occurred at the Train Epoch 2 Train Iteration1930 as the follow:

.....
2022-01-13 06:47:40,820 INFO [trainer_contrastive.py, 318] 60 images processed

2022-01-13 06:47:40,894 INFO [trainer_contrastive.py, 318] 60 images processed

2022-01-13 06:47:41,245 INFO [trainer_contrastive.py, 318] 60 images processed

2022-01-13 06:47:41,654 INFO [trainer_contrastive.py, 318] 60 images processed

2022-01-13 06:47:45,347 INFO [base.py, 84] Performance 0.0 -> 0.38793964487753546
2022-01-13 06:47:49,701 INFO [trainer_contrastive.py, 394] Test Time 50.082s, (0.795) Loss 0.64488318

2022-01-13 06:47:49,702 INFO [base.py, 33] Result for seg
2022-01-13 06:47:49,702 INFO [base.py, 49] Mean IOU: 0.38793964487753546

2022-01-13 06:47:49,702 INFO [base.py, 50] Pixel ACC: 0.8894303367559715
.......
2022-01-13 06:58:05,453 INFO [trainer_contrastive.py, 283] Train Epoch: 2 Train Iteration: 1930 Time 6.409s / 10iters, (0.641) Forward Time 2.593s / 10iters, (0.259) Backward Time 3.720s / 10iters, (0.372) Loss Time 0.081s / 10iters, (0.008) Data load 0.014s / 10iters, (0.001401)
Learning rate = [0.00956490946871452, 0.00956490946871452] Loss = 0.31870443 (ave = 0.43296322)

Traceback (most recent call last):
File "/home/xxx/paper_code/ContrastiveSeg-main/main_contrastive.py", line 217, in
model.train()
File "/home/xxx/paper_code/ContrastiveSeg-main/segmentor/trainer_contrastive.py", line 420, in train
self.__train()
File "/home/xxx/paper_code/ContrastiveSeg-main/segmentor/trainer_contrastive.py", line 241, in __train
loss = self.pixel_loss(outputs, targets, with_embed=with_embed)
File "/home/xxx/anaconda3/envs/contrastSeg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/home/xxx/paper_code/ContrastiveSeg-main/lib/loss/loss_contrast.py", line 229, in forward
loss_contrast = self.contrast_criterion(embedding, target, predict)
File "/home/xxx/anaconda3/envs/contrastSeg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/home/xxx/paper_code/ContrastiveSeg-main/lib/loss/loss_contrast.py", line 146, in forward
loss = self.contrastive(feats, labels)
File "/home/xxx/paper_code/ContrastiveSeg-main/lib/loss/loss_contrast.py", line 92, in contrastive
anchor_num, n_view = feats.shape[0], feats.shape[1]
AttributeError: 'NoneType' object has no attribute 'shape'

the config details about my experiment is:
(contrastSeg) xxx@ubuntu:~/paper_code/ContrastiveSeg-main$ bash scripts/cityscapes/hrnet/run_h_48_d_4_ocr_contrast.sh train 4
Logging to /home/xxx/paper_code/ContrastiveSeg-main//logs/Cityscapes/hrnet_w48_ocr_contrast_lr1x_ocr_contrast_40k.log
World size: 4
2022-01-13 06:35:32,252 INFO [distributed.py, 49] ['main_contrastive.py', '--configs', 'configs/cityscapes/H_48_D_4.json', '--drop_last', 'y', '--phase', 'train', '--gathered', 'n', '--loss_balance', 'y', '--log_to_file', 'n', '--backbone', 'hrnet48', '--model_name', 'hrnet_w48_ocr_contrast', '--data_dir', '/home/xxx/paper_code/ContrastiveSeg-main/dataset//Cityscapes', '--loss_type', 'contrast_auxce_loss', '--gpu', '0', '1', '2', '3', '--max_iters', '40000', '--max_epoch', '500', '--checkpoints_root', '/home/xxx/paper_code/ContrastiveSeg-main/Checkpoints/hrnet_w48_ocr_contrast/Cityscapes/', '--checkpoints_name', 'hrnet_w48_ocr_contrast_lr1x_ocr_contrast_40k', '--pretrained', '/home/xxx/paper_code/ContrastiveSeg-main/pretrained_model/hrnetv2_w48_imagenet_pretrained.pth', '--train_batch_size', '4', '--distributed', '--base_lr', '0.01']
['--configs', 'configs/cityscapes/H_48_D_4.json', '--drop_last', 'y', '--phase', 'train', '--gathered', 'n', '--loss_balance', 'y', '--log_to_file', 'n', '--backbone', 'hrnet48', '--model_name', 'hrnet_w48_ocr_contrast', '--data_dir', '/home/xxx/paper_code/ContrastiveSeg-main/dataset//Cityscapes', '--loss_type', 'contrast_auxce_loss', '--gpu', '0', '1', '2', '3', '--max_iters', '40000', '--max_epoch', '500', '--checkpoints_root', '/home/xxx/paper_code/ContrastiveSeg-main/Checkpoints/hrnet_w48_ocr_contrast/Cityscapes/', '--checkpoints_name', 'hrnet_w48_ocr_contrast_lr1x_ocr_contrast_40k', '--pretrained', '/home/xxx/paper_code/ContrastiveSeg-main/pretrained_model/hrnetv2_w48_imagenet_pretrained.pth', '--train_batch_size', '4', '--distributed', '--base_lr', '0.01']

i probably knew the problem was with the loading of the dataset, but failed to find the concrete problem. what's more, this error is unstablely occurred at the same iteration step!! what an amazing problem, so, i want to find some help from the author for a details about data format and a simple demo for running the .sh file. Thank you so much and waiting for your reply.

[shanhuhaifeng]

sorry, i find the mistakes made during preprocessing dataset..... Be carefull is pretty critical ! now i success running the code with cityscapes dataset, anyone who has problems can tell me, and i will be glad and try my best to give my experience to you

[Sibyl233]

I met the same question. Can you describe the mistake when preprocessing datasets?

[ramdrop]

@shanhuhaifeng I met the same issue, could you share how to solve it?

[GewelsJI]

sorry, i find the mistakes made during preprocessing dataset..... Be carefull is pretty critical ! now i success running the code with cityscapes dataset, anyone who has problems can tell me, and i will be glad and try my best to give my experience to you

@shanhuhaifeng could you please reveal your solution to cope with that?

[GewelsJI]

@Sibyl233 @ramdrop have you guys solved this problem yet? Could you give me some advice for that?

[JDekun]

@shanhuhaifeng I have the same issue, could you share me how to solve it?

[shanhuhaifeng]

@shanhuhaifeng I have the same issue, could you share me how to solve it?

if you are using the cityscapes dataset as the demo, please make sure that the subdir named 'coarse' should be ignore. and then revised the code that defined for the Cityscapes dataset. if you are using your custom data, i suggest you ceate your own data load class file.