Task definition matters in terms of memory usage

[rvandewater]

As suggested by @Oufattole, this kind of config seems to use excessive amounts of memory (more than 400 GB on MIMIC-IV in my case)

using regex mitigates this:

https://github.com/Oufattole/meds-torch/blob/main/MIMICIV_INDUCTIVE_EXPERIMENTS/configs/tasks/mortality/in_icu/first_24h.yaml

I'm not sure what the reason could be, but perhaps it's something to attend users to. Tested with es-aces 0.5.1 and command: aces-cli --multirun hydra/launcher=joblib data=sharded data.standard=meds data.root="$MIMICIV_MEDS_DIR/data" "data.shard=$(expand_shards $MIMICIV_MEDS_DIR/data)" cohort_dir="$cohort_dir" cohort_name="$TASK_NAME"

[justin13601]

Yep this is a limitation with the creation of the predicates as memory peaks during this process (also brought up in #89, which was closed after #90 was merged with the ability to match using regex).

Basically, each time a predicate is defined in a configuration file, a column is created which corresponds to that predicate. So, in the first example, defining the different types of admission predicates ultimately creates 10 columns, whereas the regex option will only create 1 column that matches everything. And when your dataframe is relatively large, adding these columns really blow up the memory required.

I'll add a note about this in the documentation as well as the repo README - thanks!