Controlling memory usage by tapp

[tylerhoecker]

I am struggling to control memory usage while using tapp. I specify memfrac = 0, memmax = 5 and provide a temporary filename to the function being applied within tapp, but many 10s of GBs of RAM are still being used. This is an issue when I try to run the code in parallel using future_walk with callr, triggering a std::bad_alloc error. I can't figure out if I am misunderstanding how to use memmax (or related parameters) or if there is a bug where tapp is not respecting memfrac and memmax .

Here is a snippet of the problematic code:

terra::tapp(
            spatrast,
            "months",
            fun = \(x) terra::mean(x),
            filename = output_file,
            overwrite = TRUE,
            wopt = list(
                "gdal" = c("COMPRESS=LZW", "BIGTIFF=YES")
 )

The general problem being solved here is calculating the mean across several thousands bands that represent hourly climate data.

I can work on generating a reprex if the issue is not clear. Thanks in advance!

[tylerhoecker]

It appears that using steps=nrow(x)/10 as @rhijmans suggests in this issue prevents the rampant memory use. This is described as the older work-around, making me wonder if this new feature didn't make it to tapp.

[rhijmans]

I cannot do much without more information. In the closed issue you refer to you can see the type of thing that you can report (terraOptions and mem_info) . Adding verbose=TRUE to the wopt list can also help. You are also not reporting your OS/versions.