Improve support for DAG updates

[LucaCinquini]

Currently, the OGC API allows to register a DAG that already exists in a Github repo (which is periodically synced to the SPS DAG catalog). But if the DAG is updated in the Github repo, the updates are not automatically propagated to SPS: the DAG needs to be unregistered and then registered again to be updated.

We need to design and implement a better process for automatically propagating DAG updates to the SPS Airflow deployment.

[rtapella]

Are you sure? Do you mean that you need to build some custom CD during development… ? Can Airflow have multiple versions of the same DAG deployed (e.g. for comparison testing)?

Also this is just for the DAG, not for the App Package/algorithm part, correct?

[LucaCinquini]

So right now Airflow does not support DAG versioning, you would need to include the version in the ID, such as: "my_dag_id_1", "my_dag_id_2" etc.
And yes this ticket is just about DAGs and Airflow, not about Application Packages. If different versions of an application package are available at different URLs, they can both be executed through the same "cwl_dag" DAG.