Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Partial writing #428

Open
steven-murray opened this issue Jul 17, 2023 · 0 comments
Open

Feature Request: Partial writing #428

steven-murray opened this issue Jul 17, 2023 · 0 comments
Assignees

Comments

@steven-murray
Copy link
Contributor

Currently, the whole simulation is written to a single file at the end of simulation. This has three potential drawbacks:

  1. Sometimes you want to split your simulation into multiple files for easier handling after the simulation (eg. splitting on time)
  2. Sometimes you want to split the simulation into different chunks which run independently, but might want to write the results into one file (eg. vis cpu is most easily run by sending each frequency to its own SLURM task. It would be nice if the results were all written back into one file though).
  3. Sometimes you might want to "checkpoint" the simulation for large sims.

It would be good to think about what an extension to the "filing" configuration might look like that can enable this. Also, keep in mind that there should be some conceptual separation between the configuration and the implementation -- hera_sim for example uses the pyuvsim configuration definition, but not its implementation of the simulation itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant