Deployment

Deployment of the DQA Tool

• Deployment of the DQA Tool
  • Docker
  • Kubernetes
    • Background
    • How to use
    • Thanks

Docker

Kubernetes

Background

The manifest ./docker/dqastats-workflow.yaml uses Argo Workflows to shedule the dockerized version of DQAstats to run a data quality (DQ) analysis on a regular basis.

How to use

1. Install KinD (Kubernetes in Docker).

2. Create a local cluster for testing:

   kind create cluster

3. Install Argo Workflows:

   ## Add the HELM repo for Argo:
   helm repo add bitnami https://charts.bitnami.com/bitnami
   
   ## Install Argo Workflow with own presets:
   helm install argo-wf bitnami/argo-workflows \
       --set server.serviceAccount.name=argo-wf-san

4. Follow the instructions in the console to obtain the Bearer token, these might be similar to the following:

   ## Note: If you changed the name `arg-wf` of the deployment
   ## in the `helm install ...` command above,
   ## you also need to change it here:
   SECRET=$(kubectl get sa argo-wf-san -o=jsonpath='{.secrets[0].name}')
   ARGO_TOKEN="Bearer $(kubectl get secret $SECRET -o=jsonpath='{.data.token}' | base64 --decode)"
   echo "$ARGO_TOKEN"

5. Change the manifest ./docker/dqastats-workflow.yaml to your needs or keep the current one for demo purpose.

6. Send the secret and the workflow to the cluster:

   kubectl apply -f ./docker/dqastats-secret.yaml
   kubectl apply -f ./docker/dqastats-workflow.yaml

Thanks

🎉 Big thanks to @christian.gulden / @chgl for all Kubernetes Support! The draft of this "How to ..." section is borrowed from him, originally from here: https://gitlab.miracum.org/miracum/charts/-/blob/master/README.md.