FileNotFoundError: xxx/massradiusEarthlikeRocky.txt not found.

[stefsmeets]

Hi all, #220 changes something and now the dummy input fails with:
FileNotFoundError: /home/stef/python/proteus/fwl_data/mass_radius/Mass-radius/Zeng2019/massradiusEarthlikeRocky.txt not found.

I seem to be missing some file, but it is unclear to me how I can get it.

[stefsmeets]

Seems like I need to run proteus get reference, but it says that the data already exists:

$ find . -name '*massradius*'
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradius_50percentH2O_1000K_1mbar.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradius_50percentH2O_500K_1mbar.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradius_100percentH2O_500K_1mbar.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradiusFe.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradius_50percentH2O_300K_1mbar.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradius_100percentH2O_300K_1mbar.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradius_100percentH2O_700K_1mbar.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradiushydrogen.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradiusMaxCollisionStripping.txt
./fwl_data/mass_radius/Mass-radius/Zeng2019/massradiusmgsio3.txt

[stefsmeets]

I removed the mass_radius directory and ran proteus get reference again. It works again now.

Seems like the download command only checks for the presence of the directory. This seems like an oversight, especially because it fails silently and it isn't documented.

[nichollsh]

Is this true for all of the download commands then? Or just this one?

[stefsmeets]

I don't know, I didn't look at the other ones.

[stefsmeets]

Oof, there is a lot of room for refactoring the proteus.utils.data module. All download functions check for the existance of the directory, and skips downloading if the directory exists. If any of the files inside the directory are updated, then these will not be downloaded.

However, checking for every file is also extremely slow. And I believe OSF does not have something like a file index to check against, you must check file-by-file which is a performance killer.

[nichollsh]

It might be possible to get this information using the API?
e.g. curl -X "GET" https://api.osf.io/v2/nodes/fzwr4/files/osfstorage/

[nichollsh]

Actually, this can be done with the osf library that we are already using: here

bash> osf -p fzwr4 list

[nichollsh]

And one other thing: while working on this, it might be a good time to address #206.

[stefsmeets]

@lsoucasse I don't believe this was fixed. #206 is not a duplicate of this issue.

[lsoucasse]

@lsoucasse I don't believe this was fixed. #206 is not a duplicate of this issue.

Sorry, thought PR #244 fully addressed it.