Merge stable into master for release 1.3.0 (#378)

* Upgrade from helm2 to helm3 by updating the base image. (#370) * Add links for client & ImageJ Plugin to architecture diagram (#377) * Update version to 1.3.0 for pending release.
vanvalenlab · Sep 18, 2020 · 87c414d · 87c414d
1 parent 2e3e3b6
commit 87c414d
Show file tree

Hide file tree

Showing 21 changed files with 50 additions and 47 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -1,6 +1,6 @@
-FROM cloudposse/build-harness:0.31.1 as build-harness
+FROM cloudposse/build-harness:0.39.0 as build-harness
 
-FROM cloudposse/geodesic:0.123.1
+FROM cloudposse/geodesic:0.135.0
 
 RUN apk add --update dialog libqrencode
 

diff --git a/README.md b/README.md
@@ -49,7 +49,7 @@ Tracked Image
 Start a terminal shell and install the DeepCell Kiosk wrapper script:
 
 ```bash
-docker run -e DOCKER_TAG=1.2.1 vanvalenlab/kiosk-console:1.2.1 | sudo bash
+docker run -e DOCKER_TAG=1.3.0 vanvalenlab/kiosk-console:1.3.0 | sudo bash
 ```
 
 To start the kiosk, just run `kiosk-console` from the terminal shell.
@@ -60,20 +60,25 @@ Check out our [docs](https://deepcell-kiosk.readthedocs.io/en/master/GETTING_STA
 
 ![Kiosk Architecture](https://raw.githubusercontent.com/vanvalenlab/kiosk-console/master/docs/images/Kiosk_Architecture.png)
 
-- [Frontend](https://github.com/vanvalenlab/kiosk-frontend): API for creating and managing jobs, and a React-based web interface for the DeepCell Kiosk.
-
 - [Consumer](https://github.com/vanvalenlab/kiosk-redis-consumer): Retrieves items from the Job Queue and handles the processing pipeline for that item. Each consumer only works on one item at a time.
 
 - [Model Server](https://github.com/vanvalenlab/kiosk-tf-serving): Serves models over a gRPC API, allowing consumers to send data and get back predictions.
 
 - [GPU Autoscaler](https://github.com/vanvalenlab/kiosk-autoscaler): Automatically and efficiently scales Kubernetes GPU resources.
 
-- _Not pictured above_:
+- [Frontend](https://github.com/vanvalenlab/kiosk-frontend): API for creating and managing jobs, and a React-based web interface for the DeepCell Kiosk.
 
-  * [Bucket Monitor](https://github.com/vanvalenlab/kiosk-bucket-monitor): Purges the bucket of uploaded and processed files that are older than `AGE_THRESHOLD`, 3 days by default.
+- _Additional Data Entry Tools_:
+
+  - [ImageJ Plugin](https://github.com/vanvalenlab/kiosk-imageJ-plugin): An ImageJ 1.x plugin for processing images with an existing cluster.
+
+  - [Command-line Interface](https://github.com/vanvalenlab/kiosk-client): A python-based CLI for submitting and managing DeepCell Kiosk jobs.
+
+- _Not pictured above_:
 
-  * [Janitor](https://github.com/vanvalenlab/kiosk-redis-janitor): Monitors in-progress items and makes sure no jobs get left un-finished.
+  - [Bucket Monitor](https://github.com/vanvalenlab/kiosk-bucket-monitor): Purges the bucket of uploaded and processed files that are older than `AGE_THRESHOLD`, 3 days by default.
 
+  - [Janitor](https://github.com/vanvalenlab/kiosk-redis-janitor): Monitors in-progress items and makes sure no jobs get left un-finished.
 
 ## Contribute
 

diff --git a/conf/Makefile b/conf/Makefile
@@ -23,7 +23,6 @@ destroy: \
 # Unit tests
 test/unit:
 	@echo "Linting helm charts"
-	@helm init --client-only
 	@ls -1 -d /conf/charts/* | xargs helm lint
 	@echo "Linting helmfiles"
 	@helmfile -f /conf/ELK_helmfiles/ -q build 1>/dev/null

diff --git a/conf/charts/autoscaler/templates/clusterrole.yaml b/conf/charts/autoscaler/templates/clusterrole.yaml
@@ -1,5 +1,5 @@
 {{- if .Values.rbac.create -}}
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/autoscaler/templates/clusterrolebinding.yaml b/conf/charts/autoscaler/templates/clusterrolebinding.yaml
@@ -1,5 +1,5 @@
 {{- if .Values.rbac.create -}}
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/autoscaler/templates/deployment.yaml b/conf/charts/autoscaler/templates/deployment.yaml
@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/benchmarking/templates/deployment.yaml b/conf/charts/benchmarking/templates/deployment.yaml
@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/bucket-monitor/templates/deployment.yaml b/conf/charts/bucket-monitor/templates/deployment.yaml
@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/frontend/templates/ingress.yaml b/conf/charts/frontend/templates/ingress.yaml
@@ -1,7 +1,7 @@
 {{- if .Values.ingress.enabled -}}
 {{- $fullName := include "fullname" . -}}
 {{- $ingressPath := .Values.ingress.path -}}
-apiVersion: extensions/v1beta1
+apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   name: {{ $fullName }}

diff --git a/conf/charts/redis-janitor/templates/clusterrole.yaml b/conf/charts/redis-janitor/templates/clusterrole.yaml
@@ -1,5 +1,5 @@
 {{- if .Values.rbac.create -}}
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/redis-janitor/templates/clusterrolebinding.yaml b/conf/charts/redis-janitor/templates/clusterrolebinding.yaml
@@ -1,5 +1,5 @@
 {{- if .Values.rbac.create -}}
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/redis-janitor/templates/deployment.yaml b/conf/charts/redis-janitor/templates/deployment.yaml
@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: {{ template "fullname" . }}

diff --git a/conf/charts/tensorboard/templates/ingress.yaml b/conf/charts/tensorboard/templates/ingress.yaml
@@ -1,7 +1,7 @@
 {{- if .Values.ingress.enabled -}}
 {{- $fullName := include "fullname" . -}}
 {{- $ingressPath := .Values.ingress.path -}}
-apiVersion: extensions/v1beta1
+apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   name: {{ $fullName }}

diff --git a/conf/helmfile.d/0000.ingress.yaml b/conf/helmfile.d/0000.ingress.yaml
@@ -1,7 +1,7 @@
 repositories:
 # Stable repo of official helm charts
-- name: "stable"
-  url: "https://kubernetes-charts.storage.googleapis.com"
+- name: stable
+  url: https://kubernetes-charts.storage.googleapis.com
 
 releases:
 
@@ -13,29 +13,29 @@ releases:
 # References:
 #   - https://github.com/cloudposse/charts/blob/master/incubator/nginx-ingress/values.yaml
 #
-- name: "ingress"
-  namespace: "kube-system"
+- name: ingress
+  namespace: kube-system
   labels:
-    chart: "nginx-ingress"
-    repo: "stable"
-    component: "ingress"
-    namespace: "kube-system"
-    vendor: "kubernetes"
+    chart: nginx-ingress
+    repo: stable
+    component: ingress
+    namespace: kube-system
+    vendor: kubernetes
     default: "false"
-  chart: "stable/nginx-ingress"
-  version: "0.24.0"
+  chart: stable/nginx-ingress
+  version: 1.40.2
   wait: true
   values:
     - controller:
         ### Optional: NGINX_INGRESS_CONTROLLER_REPLICA_COUNT; e.g. 2
         replicaCount: {{ env "NGINX_INGRESS_CONTROLLER_REPLICA_COUNT" | default "4" }}
         resources:
           limits:
-            cpu: "200m"
-            memory: "3Gi"
+            cpu: 200m
+            memory: 3Gi
           requests:
-            cpu: "50m"
-            memory: "128Mi"
+            cpu: 50m
+            memory: 128Mi
         stats:
           ### Optional: NGINX_INGRESS_CONTROLLER_STATS_ENABLED;
           enabled: {{ env "NGINX_INGRESS_CONTROLLER_STATS_ENABLED" | default "true" }}
@@ -53,11 +53,11 @@ releases:
         replicaCount: {{ env "NGINX_INGRESS_DEFAULT_BACKEND_REPLICA_COUNT" | default "2" }}
         resources:
           limits:
-            cpu: "200m"
-            memory: "256Mi"
+            cpu: 200m
+            memory: 256Mi
           requests:
-            cpu: "50m"
-            memory: "128Mi"
+            cpu: 50m
+            memory: 128Mi
         service:
           annotations:
             ### Required: NGINX_INGRESS_HOSTNAME; e.g. ingress.us-west-2.cloudposse.co

diff --git a/conf/tasks/Makefile.gke b/conf/tasks/Makefile.gke
@@ -257,7 +257,6 @@ gke/create/resources: \
 	gke/create/node-pools \
 	gke/create/elk-node-pools \
 	gke/create/bucket \
-	gke/deploy/helm \
 	gke/deploy/nvidia
 
 ## Create Cluster

diff --git a/conf/tasks/Makefile.helmfile b/conf/tasks/Makefile.helmfile
@@ -22,7 +22,7 @@ helmfile/create/elk:
 helmfile/destroy/all:
 	@echo " "
 	@echo "Destroying all Kubernetes resources using helmfiles..."
-	@helm ls --all --short | xargs helm delete --purge
+	@helm ls --all --short | xargs helm delete
 	@echo "Helmfile destruction finished."
 	@echo " "
 	@echo " "
diff --git a/conf/tasks/Makefile.kubectl b/conf/tasks/Makefile.kubectl
@@ -41,7 +41,7 @@ kubectl/display/ip:
 ## Tear down prometheus-operator and it's CRDs
 kubectl/destroy/prometheus/operator:
 	@echo "Deleting prometheus-operator..."
-	-@helm delete prometheus-operator --purge
+	-@helm delete prometheus-operator
 	-@kubectl delete crd prometheuses.monitoring.coreos.com
 	-@kubectl delete crd prometheusrules.monitoring.coreos.com
 	-@kubectl delete crd servicemonitors.monitoring.coreos.com

diff --git a/docs/source/CUSTOM-JOB.rst b/docs/source/CUSTOM-JOB.rst
@@ -316,9 +316,9 @@ You will need to sync your helmfile in order to update your frontend website to
 
 .. code-block:: bash
 
-    helm delete --purge frontend; helmfile -l name=frontend sync
-    helm delete --purge redis-janitor; helmfile -l name=redis-janitor sync
-    helm delete --purge autoscaler; helmfile -l name=autoscaler sync
+    helm delete frontend; helmfile -l name=frontend sync
+    helm delete redis-janitor; helmfile -l name=redis-janitor sync
+    helm delete autoscaler; helmfile -l name=autoscaler sync
 
 In a few minutes the Kiosk will be ready to process the new job type.
 

diff --git a/docs/source/GETTING_STARTED.rst b/docs/source/GETTING_STARTED.rst
@@ -113,7 +113,7 @@ You are now ready to start the Kiosk!
 
 .. code-block:: bash
 
-    docker run -e DOCKER_TAG=1.2.1 vanvalenlab/kiosk-console:1.2.1 | sudo bash
+    docker run -e DOCKER_TAG=1.3.0 vanvalenlab/kiosk-console:1.3.0 | sudo bash
 
 .. note:: This command and the one that follows may need to be preceded by `sudo` depending on your permission settings. This will require you to enter your password.
 

diff --git a/docs/source/TROUBLESHOOTING.rst b/docs/source/TROUBLESHOOTING.rst
@@ -73,7 +73,7 @@ A consumer should always either successfully consume a job or fail and provide a
 
 * Redeploy the cluster with the more powerful nodes than the default ``n1-standard-1``.
 
-* Increase the memory/cpu resource request in the helmfile of the consumer. (Remember to follow this by issuing the following command ``helm delete consumer-name --purge; helmfile -l name=consumer-name sync``)
+* Increase the memory/cpu resource request in the helmfile of the consumer. (Remember to follow this by issuing the following command ``helm delete consumer-name; helmfile -l name=consumer-name sync``)
 
 A prediction job may also never finish if the ``tf-serving`` pod never comes up. If you see that the ``tf-serving`` pod is not in status ``Running`` or has been restarting, there is likely a memory/resource issue with the model server itself. If this is the case, please read below.
 

diff --git a/scripts/gke-helmfile-deployment.sh b/scripts/gke-helmfile-deployment.sh
@@ -10,7 +10,7 @@ for filename in /conf/helmfile.d/*.yaml; do
       helmfile --selector name=${name} sync
       [[ $? -eq 0 ]] && break
       echo "Something went wrong while deploying ${name}. Retrying in 30 seconds."
-      helm delete ${name} --purge
+      helm delete ${name}
       sleep 30
     done
   done