Merge pull request #400 from datacoves/DCV-1334-move-from-git-secret-…

…to-1-password-secrets-on-dbt-coves

feat(DCV-1334 move tests secrets from git-secret to 1password

.github/workflows/main_ci.yml

@@ -39,11 +39,34 @@ jobs:
       - name: Install Dependencies using Poetry
         run: poetry install --with test
 
-      - name: Decrypt secrets
-        run: |
-          sudo apt install git-secret
-          gpg --import <(echo "$GPG_PRIVATE_KEY")
-          git secret reveal -f
+      - name: Load secrets
+        uses: 1password/load-secrets-action@v1
+        with:
+          # Export loaded secrets as environment variables
+          export-env: true
+        env:
+          OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.ONE_PASSWORD_SERVICE_ACCOUNT }}
+          # Redshift
+          PROFILE_DBT_COVES_REDSHIFT: op://Engineering - dbt-coves/dbt-coves-tests/PROFILE_DBT_COVES_REDSHIFT
+          HOST_REDSHIFT: op://Engineering - dbt-coves/dbt-coves-tests/HOST_REDSHIFT
+          USER_REDSHIFT: op://Engineering - dbt-coves/dbt-coves-tests/USER_REDSHIFT
+          PASSWORD_REDSHIFT: op://Engineering - dbt-coves/dbt-coves-tests/PASSWORD_REDSHIFT
+          DATABASE_REDSHIFT: op://Engineering - dbt-coves/dbt-coves-tests/DATABASE_REDSHIFT
+          SCHEMA_REDSHIFT: op://Engineering - dbt-coves/dbt-coves-tests/SCHEMA_REDSHIFT
+          # Snowflake
+          PROFILE_DBT_COVES_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/PROFILE_DBT_COVES_SNOWFLAKE
+          USER_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/USER_SNOWFLAKE
+          PASSWORD_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/PASSWORD_SNOWFLAKE
+          ACCOUNT_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/ACCOUNT_SNOWFLAKE
+          WAREHOUSE_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/WAREHOUSE_SNOWFLAKE
+          ROLE_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/ROLE_SNOWFLAKE
+          DATABASE_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/DATABASE_SNOWFLAKE
+          SCHEMA_SNOWFLAKE: op://Engineering - dbt-coves/dbt-coves-tests/SCHEMA_SNOWFLAKE
+          # Bigquery
+          PROFILE_DBT_COVES_BIGQUERY: op://Engineering - dbt-coves/dbt-coves-tests/PROFILE_DBT_COVES_BIGQUERY
+          SERVICE_ACCOUNT_GCP: op://Engineering - dbt-coves/dbt-coves-tests/SERVICE_ACCOUNT_GCP
+          PROJECT_BIGQUERY: op://Engineering - dbt-coves/dbt-coves-tests/PROJECT_BIGQUERY
+          DATASET_BIGQUERY: op://Engineering - dbt-coves/dbt-coves-tests/DATASET_BIGQUERY
 
       - name: Create profiles
         run: |

CONTRIBUTING.md

@@ -139,31 +139,33 @@ The whole package is managed using [Poetry](https://python-poetry.org/). It's re
 
 If you're comfortable writing tests for your features, we use the [`pytest`](https://docs.pytest.org/en/stable/) framework. It'll be installed automatically when you set up your poetry environment. If you don't know how to write tests that's fine, we'll work it out with you during the review process 💪🏻
 
-Have a look into the `tests/` folder for how the tests are written and if you want to trigger tests locally you can do so from the root of the repo with
+For our current tests, we must have certain environment variables defined in a `.env` file inside `/tests`:
 
-```bash
-pytests tests/
 ```
-
-### How to deploy a new dbt-coves version
-
-1. Replace version number everywhere
-2. Run
-
-```
-./publish.sh
+PROFILE_DBT_COVES_REDSHIFT
+HOST_REDSHIFT
+USER_REDSHIFT
+PASSWORD_REDSHIFT
+DATABASE_REDSHIFT
+SCHEMA_REDSHIFT
+PROFILE_DBT_COVES_SNOWFLAKE
+USER_SNOWFLAKE
+PASSWORD_SNOWFLAKE
+ACCOUNT_SNOWFLAKE
+WAREHOUSE_SNOWFLAKE
+ROLE_SNOWFLAKE
+DATABASE_SNOWFLAKE
+SCHEMA_SNOWFLAKE
+PROFILE_DBT_COVES_BIGQUERY
+PROJECT_BIGQUERY
+DATASET_BIGQUERY
+SERVICE_ACCOUNT_GCP
 ```
 
-### Release new dbt-coves docker image to public repo
+We recommend using 1Password. In it, an entry named `dbt-coves-tests` with the keys above, installing [1Password CLI](https://developer.1password.com/docs/cli/) and running `tests/generate_onepsw_env_file.py`
 
-Url: https://hub.docker.com/r/datacoves/dbt-coves
+Once you have the `.env` file ready, just run:
 
-```
-./docker/build_and_push.sh <dbt version> <dbt-coves version>
-```
-
-i.e.
-
-```
-./docker/build_and_push.sh 0.21.0 0.21.0a14
+```bash
+pytests tests/
 ```

tests/generate_onepsw_env_file.py

@@ -0,0 +1,17 @@
+import json
+import os
+import subprocess
+from pathlib import Path
+
+BASEDIR = os.path.abspath(os.path.dirname(__file__))
+
+subprocess.run(["op", "signin"])
+onepassword_entry = subprocess.check_output(
+    ["op", "item", "get", "dbt-coves-tests", "--format", "json"]
+)
+onepassword_entry = json.loads(onepassword_entry)
+dbt_coves_fields = onepassword_entry.get("fields")
+with open(os.path.join(BASEDIR, ".env"), "w+") as f:
+    for field in dbt_coves_fields:
+        if "value" in field:
+            f.write("{}={}\n".format(field["label"], field["value"]))

tox.ini

@@ -6,7 +6,9 @@ require_poetry = true
 
 setenv =
     COVERAGE_FILE = .coverage.{envname}
+
 [testenv:dbt110]
+passenv = *
 deps=
     dbt-core==1.1.0
     dbt-postgres==1.1.0
@@ -24,6 +26,7 @@ commands =
     coverage report -m
 
 [testenv:150]
+passenv = *
 deps=
     dbt-core==1.5.0
     dbt-postgres==1.5.0