Using two gpus to train a model on ms_coco, validation results is inconsistent

[mynobody]

I trained a model using two gpus on ms_coco (dist_train.sh), the model was later tested using the same val datasets. but the results from training and from testing are different.
Similarly, I trained the model using only one gpu, and tested it in the same way, the results from training and from testing are same.
BTW, the val and test pipeline are same.

Anyone know which one is correct? how to test on model trained with dist_train.sh?