Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove data from EFS at the end of PDS basic use case #95

Closed
ramesh-maddegoda opened this issue Mar 13, 2024 · 1 comment
Closed

Remove data from EFS at the end of PDS basic use case #95

ramesh-maddegoda opened this issue Mar 13, 2024 · 1 comment

Comments

@ramesh-maddegoda
Copy link
Contributor

ramesh-maddegoda commented Mar 13, 2024

💡 Description

We are using EFS as a way to share data between Tasks in Airflow DAGs, because EFS access points can be easily attached as Volumes in ECS tasks without modifying any existing Docker containers.

However, keeping data in EFS volumes is expensive and it is required to delete data once we have completed processing a DAG. It is required to add an additional Airflow tasks to PDS use case DAGs to delete data from EFS at the end.

Make sure to delete relevant data from S3 staging bucket first as mentioned in #94, to avoid copying data from S3 staging bucket to EFS again.

⚔️ Parent Epic / Related Tickets

Parent: #93

@ramesh-maddegoda
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 🏁 Done
Development

No branches or pull requests

2 participants