A daily letter composed of jokes, news, a deep quote and some good song suggestions.
Motivation behind this project was to understand how the ETL pipelines work and learn how frameworks such as airflow and argo are used
airflow, mongodb, postgres, docker
Our airflow DAG looks something like this
first we fetch the information from various open apis
we later combine this info together, and save the json data inside mongodb for safekeeping
and we generate the pdf using that same data, and send it to the user at the scheduled time
docker-compose updocker-compose up will kickstart all of the services. once all the services are up and running, visit http://localhost:8080 to open the airflow console
there you can trigger the DAG cereal-letter
pdfs get stored in the /temp folder
to enable emails, you will need to configure your sendgrid.com account.
after setting it up, add the email and the api key in the .env file
then the email operator should succeed
Multiple users should be able to subscribe to the letter, and all receive letters customised with their own preferences