Infra requirements

[karam86]

What are the minimum infra requirements for participating in this challenge in terms of RAM, CPUs, GPU.

Thanks

[weihua916]

Hi! Great question. We have not rigorously measured it, but roughly speaking, RAM 100GB, 4CPUs, 1 GPU should be enough to run our best gnn baseline. Let me know if this is not the case.

[tsotfsk]

Hi @weihua916, I want to know the peak memory of the GNN models. It seems that it is not enough to run them with 128G of RAM.

[weihua916]

Thanks for sharing. I have not measured the peak memory. If anyone has any idea, we'd be appreciated.

[rusty1s]

The actual training of our GNN models do not utilize that much CPU memory. For training, the main CPU memory is consumed by loading adjacency information (around 40 and 80GB for loading GNN and R-GNN edge connectivity).

The main CPU memory bottleneck is in the pre-processing stage, which you may want to further customize. For example, you may want to run the different pre-processing stages separately. You can also customize the creation of author and institution features by specifying lower values for node_chunk_size and dim_chunk_size in our rgnn.py script.

[drchangliu]

Making node_chunk_size and dim_chunk_size smaller does not reduce the size of the generated full_feat.npy file. So it takes the same amount of memory to load "full_feat" and perform the training after that. To reduce main memory requirement during training, what else can we reduce if we are willing to take a hit on other properties? Thanks in advance for any tips.

[rusty1s]

Yes, full_feat.npy will have the same size. In our original script, we make use of numpy.memmap to load the full_feat.npy matrix, which reads the full_feat matrix directly from disk instead of holding in CPU memory (but requires a speedy SSD).

There are other options to train the GNN though that do not require full_feat matrix to be created. For example, you could think of creating mean author and institution papers on the fly during training. Another option is to use random node features instead of pre-computed ones for these node types: https://arxiv.org/abs/2002.03155

[CnBDM-Su]

I ran the rgnn baseline model with the default settings, but it costs more than 2 hour for each epochs, which means I need 200 hours for training 100 epochs. How can you finish the training only in 20 hours. For training loader I use 16 workers and 1 gou for training.

[weihua916]

Hi! Your disk read may not be fast enough. Try solutions here #131.

[CnBDM-Su]

Thanks, I used your scripts to test the speed of reading 200 rows from disk, which costs 0.8 seconds. Is it slow?
And if I don't use memmap, just push all data on the memory. How much memory do I need?

[weihua916]

Yes, 0.8 seconds is slow. I remember I can do 0.02 seconds. you will probably need 200GB for gnn.py, and 500GB for gnn.py---I have not measured it.

[CnBDM-Su]

I checked the IO rate of my SSD is 500MB/s. Could you tell me about your ssd performance?
I want to locate the bottleneck, Thanks

[zbn123]

Hi! Your disk read may not be fast enough. Try solutions here #131.

Hi, I want to ask a question how to speed up np.memmap reading