Huge variation on Cora when repeating 5 times with different seed

[huangchenxi304]

Thank you for the wonderful work!

I use default setting for the node-level experiment on Cora.

python run_node_exps.py --model GCN\ --dataset Cora\ --is_iid iid\ --num_workers 5\ --num_mali 1\ --epoch_backdoor 0\ --trigger_size 3\ --trigger_type renyi\ --trigger_position random\ --poisoning_intensity 0.1\ --overlapping_rate 0.0

The result is:

average_overall_performance	average_ASR	average_Flip_ASR	average_transfer_attack_success_rate
0.738776	0.86	0.833333	0.815
0.767347	0.76	0.733333	0.655
0.755102	0.9	0.886364	0.825
0.710204	0.92	0.914894	0.9
0.726531	0.74	0.675	0.685

The variance on average_ASR is really huge.
This variance even increased when I try Cora with GraphSage:

average_overall_performance	average_ASR	average_Flip_ASR	average_transfer_attack_success_rate
0.722449	0.84	0.809524	0.675
0.755102	0.64	0.6	0.585
0.730612	0.84	0.818182	0.67
0.697959	0.94	0.93617	0.695
0.734694	0.72	0.65	0.555

I try to troubleshoot it. First, I set all seed to 996, i.e.:

# in run_node_exps.py
# seeds = rs.randint(1000, size=5)
seeds = [996, 996, 996, 996, 996]

And the result looks good:

average_overall_performance	average_ASR	average_Flip_ASR	average_transfer_attack_success_rate
0.726531	0.84	0.818182	0.665
0.738776	0.82	0.795455	0.645
0.722449	0.82	0.795455	0.655
0.726531	0.82	0.795455	0.64
0.714286	0.82	0.795455	0.655

Then, I independently fix the seed for these three lines in backdoor_node_clf.py:

def main(args, logger):
    # np.random.seed(args.seed)
    # torch.manual_seed(args.seed)
    # torch.cuda.manual_seed(args.seed)
    np.random.seed(996)
    torch.manual_seed(996)
    torch.cuda.manual_seed(996)

However, the result is:

average_overall_performance	average_ASR	average_Flip_ASR	average_transfer_attack_success_rate
0.706122	0.86	0.829268	0.715
0.767347	0.92	0.913043	0.655
0.730612	0.84	0.818182	0.665
0.714286	0.66	0.595238	0.64
0.738776	0.72	0.658537	0.615

Finally, I thought it may be due to the small size of Cora so I try Pubmed:

average_overall_performance	average_ASR	average_Flip_ASR	average_transfer_attack_success_rate
0.851269	0.86802	0.841121	0.786168
0.861929	0.944162	0.932907	0.902919
0.868528	0.939086	0.926829	0.88769
0.861929	0.92132	0.899351	0.874365
0.846701	0.964467	0.958333	0.925127

Is this huge variation on average_ASR normal, just with different seed? What could be the possible reasons? Thank you.