Finalize recipe (+ test new rclone copy stage)

[jbusecke]

This seems like a pretty simple recipe, but running it locally on the hub with

mamba create -n runner0102 python=3.11 -y
conda activate runner0102
pip install pangeo-forge-runner==0.10.2 --no-cache-dir

pangeo-forge-runner bake \
  --repo=./ \
  --Bake.recipe_id=chirps-global-daily\
  -f configs/config_local_hub.py

Blows out the 128GB of memory! The files are extremely compressed, but should still fit into memory (1GB on disk, ~20GB in memory x2 files). This smells like the same issue we (@norlandrhagen) had with fsspec balloning the memory for older recipes and virtualizarr generation? What is going on here...

[jbusecke]

Oh I did not set the target_chunks

[jbusecke]

Ok nice, this is working locally.

Running a test on dataflow now.

[jbusecke]

Ok that works out fine. @ks905383 can you check out the test dataset?

You can run this on the leap hub

import xarray as xr
xr.open_dataset("gs://leap-scratch/data-library/feedstocks/chirps_feedstock/chirps-global-daily.zarr", engine='zarr', chunks={})

[ks905383]

Nice, seems to work for 1981-1982 (and I see you've staged the recipe for the whole timeframe)

[jbusecke]

@ks905383 awesome. I would love to see what sort of processing you apply to the data, so we can prototype that. If you think that should rather be applied afterwards we can get this finalized.

[jbusecke]

I have rebased this on leap-stc/leap-data-management-utils#60 and will see if the copy stage works (moving it to m2lines-testing like in https://github.com/leap-stc/test-transfer) before merging and processing the entire thing.

• Redirect to proper OSN bucket
• Update secrets in manager to fit bucket

[jbusecke]

Ayyyy this worked apparently. Need to run for dinner, but will test output tomorrow! Exciting.

[jbusecke]

Was able to confirm that the transfer works. Lets change the target bucket/creds and then ingest the entire recipe over at leap-stc/leap-data-management-utils#60

[jbusecke]

There is still a bit of a smell here since I have to define the target bucket+path in the recipe and the creds are hardcoded here. @norlandrhagen do you have opinions on how to reconcile this?

[jbusecke]

This should be all set, but it seems the original server is down at the moment.

cc @ks905383

[norlandrhagen]

For this bit?

| CopyRclone(target=catalog_store_urls["chirps-global-daily"].replace("https://nyu1.osn.mghpcc.org/","")) #FIXME`

Maybe we can separate the prefix from the bucket/path with some leap_data_management_utils help?

[jbusecke]

@norlandrhagen 👍 i think we could possibly do this as a clean up sprint where we reduce the amount of manuall entered (and highly interdependent) naming entries for the user?

Tracking this in leap-stc/LEAP_template_feedstock#61

[jbusecke]

What the hell is this error?

gcsfs.retry.HttpError: The object leap-scratch/data-library/feedstocks/output/chirps_feedstock/chirps-global-daily-11557918430-1/chirps-global-daily.zarr/latitude/1 exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429., 429

[jbusecke]

I guess we were hammering the storage at a rate that is not allowed? See https://cloud.google.com/storage/docs/gcs429

I thought this was the cloud! Hahaha.

[jbusecke]

This is pretty odd. We also specify max-workers as 50 and it seemed to have scaled beyond that?

[jbusecke]

Ughhh it might be similar to this where each write tries to create an empty 'directory'? But that seems counterintuitive to my understanding of object storage...

[jbusecke]

And I guess max-workers is useless when using dataflow prime.

[norlandrhagen]

I also hit a max quota error the other week, so there may be some google console settings to dial.

[jbusecke]

Ill try one more time with even less workers and larger chunks (hopefully resulting in less writes?), but I wonder if we could address this more generally by creating an 'empty directory structure' before writing out the chunks. This seems to be what people suggest here and here

[jbusecke]

ughhhh its scaling past the limit I set again... is this a bug in runner?

[jbusecke]

Ill have to look into this more next week. Will have to turn to some other things now.