Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to choose covariates when extracting to TF record #7

Open
dtpc opened this issue Jun 24, 2019 · 1 comment
Open

Add option to choose covariates when extracting to TF record #7

dtpc opened this issue Jun 24, 2019 · 1 comment

Comments

@dtpc
Copy link
Collaborator

dtpc commented Jun 24, 2019

Currently we import a set of TIFs into HDF5 format then extract ALL those covariates (with an optional halfwidth parameter) to create a tf records containing the train/test and query X data.

In order to exclude a particular TIF, or to try different combinations of TIFs, we need to import them to separate HDF5 files, resulting in a lot of data duplication.

The proposal is to add an option to the extract commands to list the covariates by name which we want to extract to a new tf record. The default would be to extract all covarates. It would likely also require a command to query the names of all covariates within an existing HDF5 file.

It could potentially work as --include/--exclude flags to allow for filtering out covariates.

@dtpc
Copy link
Collaborator Author

dtpc commented Jun 27, 2019

Ok, so you can ignore any covariates within the model config. So it's just a trade-off between extracting a new tfrecord vs the overhead of reading in unused covariates during training. While I think this would still be useful, I guess its not that important as there is a way to achieve it already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant