Disruptions and update to new cdsapi

[veenstrajelmer]

Disruptions expected in spring 2024. Also, "Users of the CDS Toolbox should however take note that the CDS Toolbox will be discontinued and will not be migrated to the new CDS infrastructure. New tool packages will be made available (post new CDS launch)"

https://forum.ecmwf.int/t/a-new-cds-soon-to-be-launched-expect-some-disruptions/1607

Update 9-5-2024: new beta package (?) will be available from end of May 2024

Beta readiness: https://forum.ecmwf.int/t/informative-page-in-readiness-for-new-cds-beta/2985

Migration: https://confluence.ecmwf.int/display/CKB/Please+read%3A+CDS+and+ADS+migrating+to+new+infrastructure%3A+Common+Data+Store+%28CDS%29+Engine

Beta now live: https://forum.ecmwf.int/t/the-new-climate-data-store-beta-cds-beta-is-now-live/3315

Decent instructions: https://forum.ecmwf.int/t/step-by-step-instructions-on-how-to-download-data-using-new-climate-data-store-beta-cds-beta/3743

Retrieving data now also shows warnings: As per our announcements on the Forum, this instance of CDS will soon be decommissioned. Please update your cdsapi package to a version >=0.7.0, create an account on CDS-Beta and update your .cdsapirc file. We strongly recommend users to check our Guidelines at https://confluence.ecmwf.int/x/uINmFw The current legacy system will be kept for a while, but we will reduce resources gradually until full decommissioning in September 2024.

Todo:

• possible to login via toolbox instead of updating ~/.cdsapirc manually? >> probably no priority
• test with new ~.cdsapirc file from https://cds-beta.climate.copernicus.eu/how-to-api (first login)
• new url now contains "beta", will this be removed again in the near future? >> will keep working backwards compatible
• update new default cds url in dfm_tools code
• add check+warning for old url in ~.cdsapirc, to point users to new account+cdsapirc
• consider support when only supplying CDSAPI_KEY envvar, for EDITO
• prevent getting cdsapi-url envvar twice (default url is duplicated, can also make that central instead)
• test_download_era5 now downloads dataset with valid_time instead of time variable, we could rename this but this also causes issues: Error in renameDimension Unidata/netcdf4-python#1357 >> now done as part of preprocess_ERA5() function
• document the new workflow (see draft below)

How to work with new CDS (draft):

• https://confluence.ecmwf.int/display/CKB/Please+read%3A+CDS+and+ADS+migrating+to+new+infrastructure%3A+Common+Data+Store+%28CDS%29+Engine
• create an ECMWF account
• login at https://cds-beta.climate.copernicus.eu/profile
• activate your profile by filling in details and accepting terms and conditions
• go to https://cds-beta.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download, scroll all the way down to the "terms of use" and accept them, otherwise you will get a RuntimeError as documented in CDS beta raises unclear error about accepting licenses ecmwf/cdsapi#108
• create a ~/.cdsapirc file manually or run the dfm_tools modelbuilder or download_ERA5 function and you will be prompted for your new apikey without worrying about a file