Improve deployment #143

henry410213028 · 2024-07-07T08:52:13Z

Types of Changes

Bugfix
Refactoring
Documentation Update

Description

The main purpose of this PR is to improve our Airflow setup process, fix some errors, and make it easier for new contributors to use.

Below is the list of changes:

Airflow Configuration
- Change AIRFLOW_HOME from "/usr/local/airflow" to "/opt/airflow" because we have switched to the official Airflow Python 3.8 Docker image.
- Disable the example DAGs.
DAG files
- Fix AIRFLOW_HOME related file path issues in some DAG tasks.
Python dependencies
- Use pip to manage dependencies, not use poetry.
- Update requirements.txt and add Airflow constraints file.
- Remove poetry configuration.
Update Dockerfile
- Install dependencies via pip instead of poetry.
- Use entrypoint.sh to allow adding initialization tasks before starting Airflow services.
- Remove unused Dockerfile.test.
Use docker-compose to deploy Airflow services
- Update Makefile and add related aliases to make it easier to use.
- Use an external Docker volume to store Airflow data to prevent data loss when restarting the container.
Update documentation
- Remove outdated information in README.md.
- Move some information to CONTRIBUTING.md, MAINTENANCE.md, and DEPLOYMENT.md to make it easier to understand how to contribute, maintain, and deploy.
Remove unused Node.js configuration files.

Steps to Test This Pull Request

Follow the steps for setting up the dev/test environment in README.md.

Lee-W · 2024-07-21T07:01:33Z

requirements.txt

+# Dev dependencies
+bandit
+black
+flake8
+isort
+mypy
+pytest
+pytest-cov
+safety


Should we separate it as requirement-dev.txt for local development testing?

Lee-W · 2024-07-21T07:02:48Z

requirements.txt

+pygsheets
+requests
+searchconsole
+StrEnum


Is StrEnum required? Could we use the trick here? https://docs.python.org/3.8/library/enum.html#others

I try to remove it.

Lee-W · 2024-07-21T07:03:31Z

pyproject.toml

+# azure-datalake-store, pip._vendor.packaging.requirements.InvalidRequirement
+azure-datalake-store = ">=0.0.45"
+azure-mgmt-datalake-store = ">=0.5.0"
+# https://stackoverflow.com/questions/68687548/apache-airflow-airflow-initdb-throws-modulenotfounderror-no-module-named-wtf
+marshmallow = "2.21.0"
+wtforms = "2.3.3"
+# https://stackoverflow.com/questions/66774109/install-airflow-importerror-no-module-named-clsregistry
+sqlalchemy = "1.3.23"
+flask-sqlalchemy = "2.4.4"

 [tool.poetry.dev-dependencies]
 safety = "^1.9.0"


I thought we were removing poetry. am I miss anything?

Yes, poetry configuration should be removed to avoid any confusion.

@david30907d I need your help to fix the Python CI install dependencies step, as I don’t have permission to make changes.

@david30907d I need your help to fix the Python CI install dependencies step, as I don’t have permission to make changes.

why is that, you're already admin of this repo 😂

Lee-W · 2024-07-21T07:05:16Z

constraints-3.8.txt

@@ -0,0 +1,349 @@
+# for apache-airflow==1.10.13


I would like to confirm where it comes from. Airflow Official

Lee-W · 2024-07-21T07:09:16Z

.env.template

@@ -1,4 +1,4 @@
-# get the complete env from other volunteers, please
+AIRFLOW_HOME=/opt/airflow
 BIGQUERY_PROJECT=pycontw-225217


nitpick: I feel this might be something we should make it configurable. 🤔

Is AIRFLOW_HOME redundant?

no, I mean BIGQUERY_PROJECT

developer may use their own GCP project for testing

Yep, that's why I think making it configurable might be a good idea. WDYT?

Lee-W · 2024-07-21T07:11:24Z

README.md

+- [Python 3.8+](https://www.python.org/downloads/release/python-3811/)
+- [Docker](https://docs.docker.com/get-docker/)
+- [Git](https://git-scm.com/book/zh-tw/v2/%E9%96%8B%E5%A7%8B-Git-%E5%AE%89%E8%A3%9D%E6%95%99%E5%AD%B8)
+- [Poetry](https://python-poetry.org/docs/#installation) (Optional, only for creating virtual environments during development)


Should we add instruction on how to virtualenv instead? or use tools like pip-tools, Pipenv. AFAIK, poetry does not use pip to resolve dep. I would suggest we either use poetry for all env or not use it at all

ok, I use venv module instead.

Lee-W · 2024-07-21T07:13:12Z

dags/ods/survey_cake/dags/questionnaire_2_bigquery.py

@@ -27,7 +29,8 @@
 )
 with dag:
    if bool(os.getenv("AIRFLOW_TEST_MODE")):
-        FILENAMES: Dict[str, Dict] = {"fixtures/data_questionnaire.csv": {}}
+        filepath = Path(AIRFLOW_HOME) / "dags/fixtures/data_questionnaire.csv"
+        FILENAMES: Dict[str, Dict] = {str(filepath): {}}


nitpick: we could use from __future__ import annotations and replace it as the following

Suggested change

FILENAMES: Dict[str, Dict] = {str(filepath): {}}

FILENAMES: dict[str, dict] = {str(filepath): {}}

thanks your suggestion.

Lee-W · 2024-07-21T07:14:22Z

docs/CONTRIBUTING.md

+
+### Commit Message
+
+It is recommended to use [Commitizen](https://commitizen-tools.github.io/commitizen/).


Lee-W · 2024-07-21T07:15:01Z

docs/DEPLOYMENT.md

+        * ETL: `/home/zhangtaiwei/pycon-etl`
+        * Metabase is located here: `/mnt/disks/data-team-additional-disk/pycontw-infra-scripts/data_team/metabase_server`
+
+2. Pull the latest codebase to this server: `sudo git pull`


sudo looks dangerous. Just want to confirm whether it's required

This is a issue, project path should be move to another path.

henry410213028 · 2024-08-17T13:53:24Z

upgrade airflow to v1.10.15, due to v1.10.13 has been yanked.

…rom pip

…t guide

…ts file

henry410213028 requested a review from david30907d July 20, 2024 13:18

Lee-W self-requested a review July 21, 2024 02:37

Lee-W reviewed Jul 21, 2024

View reviewed changes

henry410213028 force-pushed the improve-deployment branch from 6e17cfc to 011209e Compare August 18, 2024 13:55

Henry Lee added 25 commits September 1, 2024 20:50

build(docker): move dockerfile and add compose file

5f17e82

refactor(airflow): change airflow home file path

274ceec

fix(dags): broken attachment file path

63be6a6

build(airflow): set airflow home env variable

11c0fc8

fix(airflow): not load example dags

430ee50

build(docker): configurable airflow and python version

0ba4446

build(pip): solve requirements from poetry and install dependencies f…

664a56f

…rom pip

build(docker): file path

514fc95

build(pip): update deps

20ed3e8

fix(pip): solve some deps runtime error

6e3d0ef

build(docker): add entrypoint script

9743a70

build(docker): update compose files

3b2d034

fix(airflow): update env template

b208a82

build(makefile): add deployment alias

dd52d9a

docs(project): update README, add contribute, maintain, and deploymen…

5837994

…t guide

build(npm): remove un-used config files

2a67908

fix(project): misc

8ede6cb

docs(project): add more contributing steps

5166cde

build(ci): restore docker image ci config change

04b1242

build(docker): restore and change test dockerfile

eddf1bc

fix(docker): create fernet key if not exists

7a5c76d

build(project): use pip and constraint file to manage deps

c563ae4

fix(docker): not upgrade pip

b283f66

docs(project): update README

309058f

fix(docker): add env variable check on entry script

a161d16

Henry Lee added 14 commits September 1, 2024 20:50

fix(ods): questionnaire test error

5f58673

docs(project): misc

fc16627

build(docker): create fernet key if not exists

8b0e911

docs(project): misc

34874b1

refactor(dags): tpying

1ed4c1b

build(python): remove poetry and move dev deps to another requiremens…

185dfdf

…ts file

docs(project): add venv setup guide

d3d72bf

docs(project): change production project path

45cdf8f

ci(python): install dependencies with pip

de60e7d

fix(project): upgrade airflow to v1.10.15

7b0bfeb

style(project): reformat

6783283

style(project): recover pyproject.toml style rule

3dfdb09

style(project): reformat

e5f4375

ci(docker): use gcp artifact-registry

b047fa1

henry410213028 force-pushed the improve-deployment branch from 4932299 to b047fa1 Compare September 1, 2024 13:02

docs(project): update

16e015a

henry410213028 merged commit 778ad30 into master Sep 1, 2024
2 checks passed

henry410213028 deleted the improve-deployment branch September 1, 2024 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve deployment #143

Improve deployment #143

henry410213028 commented Jul 7, 2024 •

edited

Loading

Lee-W Jul 21, 2024

Lee-W Jul 21, 2024

henry410213028 Aug 17, 2024

Lee-W Jul 21, 2024

henry410213028 Aug 17, 2024 •

edited

Loading

henry410213028 Aug 17, 2024

david30907d Aug 17, 2024

Lee-W Jul 21, 2024

Lee-W Jul 21, 2024

henry410213028 Aug 17, 2024

Lee-W Aug 17, 2024

henry410213028 Aug 18, 2024

Lee-W Aug 19, 2024

Lee-W Jul 21, 2024

henry410213028 Aug 17, 2024

Lee-W Jul 21, 2024

henry410213028 Aug 17, 2024

Lee-W Jul 21, 2024

Lee-W Jul 21, 2024

henry410213028 Aug 17, 2024

Lee-W Aug 17, 2024

henry410213028 commented Aug 17, 2024

	FILENAMES: Dict[str, Dict] = {str(filepath): {}}
	FILENAMES: dict[str, dict] = {str(filepath): {}}


		### Commit Message

		It is recommended to use [Commitizen](https://commitizen-tools.github.io/commitizen/).

Improve deployment #143

Improve deployment #143

Conversation

henry410213028 commented Jul 7, 2024 • edited Loading

Types of Changes

Description

Steps to Test This Pull Request

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henry410213028 Aug 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henry410213028 commented Aug 17, 2024

henry410213028 commented Jul 7, 2024 •

edited

Loading

henry410213028 Aug 17, 2024 •

edited

Loading