You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are using EFS as a way to share data between Tasks in Airflow DAGs, because EFS access points can be easily attached as Volumes in ECS tasks without modifying any existing Docker containers.
However, keeping data in EFS volumes is expensive and it is required to delete data once we have completed processing a DAG. It is required to add an additional Airflow tasks to PDS use case DAGs to delete data from EFS at the end.
Make sure to delete relevant data from S3 staging bucket first as mentioned in #94, to avoid copying data from S3 staging bucket to EFS again.
💡 Description
We are using EFS as a way to share data between Tasks in Airflow DAGs, because EFS access points can be easily attached as Volumes in ECS tasks without modifying any existing Docker containers.
However, keeping data in EFS volumes is expensive and it is required to delete data once we have completed processing a DAG. It is required to add an additional Airflow tasks to PDS use case DAGs to delete data from EFS at the end.
Make sure to delete relevant data from S3 staging bucket first as mentioned in #94, to avoid copying data from S3 staging bucket to EFS again.
⚔️ Parent Epic / Related Tickets
Parent: #93
The text was updated successfully, but these errors were encountered: