Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per-model preprocessors #807

Open
jfrost-mo opened this issue Aug 23, 2024 · 0 comments
Open

Per-model preprocessors #807

jfrost-mo opened this issue Aug 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@jfrost-mo
Copy link
Member

jfrost-mo commented Aug 23, 2024

What problem does your feature request solve?

Sometimes, in cases like #675, it is useful to apply a transformation to an entire set of model data before any recipes are applied, such as cutting out a region, or fixing something. CSET does not currently have the capability to do that, as modifying a recipe won't allow it to be done per-model.

Describe the solution you'd like

Add a pre-processing step to the workflow that can optionally run a user defined recipe to prepare the data, per model.

As part of the data time cycling work in #750, I've added the rose config side of a preprocessor system, where transformations can be applied to an entire set of model data before use.

We need to consider how we will define these preprocessors (more recipes?), and how configurable they should be. Do we want YAML recipes, or arbitrary python scripts?

Describe alternatives you've considered

Bundling this into recipes makes too many separate recipes.

Build logic into the recipes, so it can be done conditionally. While powerful this would vastly increase the complexity of the recipes, making them less reliable and harder to reason about.

@jfrost-mo jfrost-mo added the enhancement New feature or request label Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant