Integrate RDIR for sector scripts

[yerbol-akhmetov]

Changes proposed in this Pull Request

Good day. Here I propose to integrate RDIR for input and outputs of sector rules. Currently, sector related intermediate data is directly stored in resources/ which his not convenient when simulating multiple scenarios or different countries. This PR aims to resolve it by making sector scripts consistent with power only scripts. I have tested the change for NG and US. What are your thoughts about it @davide-f? @GbotemiB told me that you have discussed that option.

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

Thanks @yerbol-akhmetov :D

This PR touches a topic that was discussed a while back with -sec users and at the time we decided to decouple the definition of RDIR to avoid retriggering the whole workflow for the same country while a subset of parameters change.

To be honest, I'd be in favour of merging this PR, but a comment by @energyLS and/or @hazemakhalek and/or @Eddy-JV is encouraged.

While reviewing, I noticed that the output of prepare_energy_totals is better placed into resurces rather than data.
If Leon and hazem also agree with merging this PR, that could be combined here if interested :)

[hazemakhalek]

@davide-f I agree about the prepare_energy_totals output.

The idea of the PR is good and makes the setup cleaner, but in practice, this will cost us so much unnecessary computation. Shall we chat about this tomorrow?

[yerbol-akhmetov]

@davide-f I agree about the prepare_energy_totals output.

The idea of the PR is good and makes the setup cleaner, but in practice, this will cost us so much unnecessary computation. Shall we chat about this tomorrow?

Thanks, @hazemakhalek. Yes, I would be glad to hear pros and cons.

[davide-f]

By comparing the original -sec and the merge, some changes have been merged but the old feature is no longer available.
I fear that even the current implementation may not be desirable for -sec.

We could think about considering the scenario management that is implemented in -eur

[yerbol-akhmetov]

ome changes have been merged but the old feature is no longer available.
I fear that even the current implementation may n

In PyPSA-Eur, paths are generally provided using resources, where resources = path_provider("resources/", RDIR, shared_resources, exclude_from_shared). shared_resources are false by default, and exclude_from_shared are empty. So files are stored in resources/<run name>/ in that case.

[yerbol-akhmetov]

Hi, @davide-f. I have implemented folder structure that we discussed in PyPSA-Earth meeting. SECDIR constant was added into Snakefile which reads sector directory name from sector_name from config file. So sector data are stored in resources/<name>/sector/<sector_name>. I have tested the approach locally with different sector configurations and sector_name. It ensured that power scripts are not re-run for the second sector scenario. The sector data were stored to appropriate sector folders inside resources/<run_name>/sector/.

[davide-f]

Hi, @davide-f. I have implemented folder structure that we discussed in PyPSA-Earth meeting. SECDIR constant was added into Snakefile which reads sector directory name from sector_name from config file. So sector data are stored in resources/<name>/sector/<sector_name>. I have tested the approach locally with different sector configurations and sector_name. It ensured that power scripts are not re-run for the second sector scenario. The sector data were stored to appropriate sector folders inside resources/<run_name>/sector/.

Great proposal for the naming; that seems a great approach. I fear there was a misunderstanding: the proposal was to use SECDIR instead of RDIR, while keeping the rest the same.
This ensures that if uses what to use the "older" approach for the -sec model, we can simply specify sector_name="" and that preserves the old behavior.
If we want a mixed approach, then we can have name=sector_name

[yerbol-akhmetov]

Hi, @davide-f. I have implemented folder structure that we discussed in PyPSA-Earth meeting. SECDIR constant was added into Snakefile which reads sector directory name from sector_name from config file. So sector data are stored in resources/<name>/sector/<sector_name>. I have tested the approach locally with different sector configurations and sector_name. It ensured that power scripts are not re-run for the second sector scenario. The sector data were stored to appropriate sector folders inside resources/<run_name>/sector/.

Great proposal for the naming; that seems a great approach. I fear there was a misunderstanding: the proposal was to use SECDIR instead of RDIR, while keeping the rest the same. This ensures that if uses what to use the "older" approach for the -sec model, we can simply specify sector_name="" and that preserves the old behavior. If we want a mixed approach, then we can have name=sector_name

Hi, @davide-f, thanks for clear explanation. Now I see what you mean. Your suggestion is to store sector data in resources/SECDIR while power data is stored in resources/RDIR. If the old structure is needed we can use sector_name='', while for separate scenarios we can use any sector_name right?

Now we see two methods. Which approach would you prefer? If we use SECDIR instead of RDIR for sector, the only thing is if you accidentally name sector_name the same for different power based models. Then it will access the same SECDIR folder.

[yerbol-akhmetov]

Hi, @davide-f. I have implemented your suggestion. Now sector data is stored in SECDIR. So if SECDIR='', it will be old way of data management (and it is by default). If you want to have different sector scenario, we just name SECDIR. The output files in results are stored as results/RDIR/SECDIR. So if SECDIR if empty by default, the outputs are stored as in the old way. If you have SECDIR, then data is stored in SECDIR inside RDIR. What do you think? I have tested it locally, it worked as described.

[davide-f]

Great work @yerbol-akhmetov :D
Merging