Skip to content

Commit

Permalink
Merge stable into master for release 1.2.0 (#343)
Browse files Browse the repository at this point in the history
* Feature/cicd (#268)

* Set all custom charts image.pullPolicy to IfNotPresent (#258)

* setting TRANSLATE_COLON_NOTATION=false by default (#289)

* Update Getting Started  (#287)

* Update PULL_REQUEST.md for grammar (#292)

* Use gomplate to template patches/hpa.yaml. (#293)

* default account has 100 firewalls, not 200. (#297)

* Update all documentation and links to reference kiosk-console instead of kiosk (#295)

* Use yq and helmfile build to dynamically deploy helm charts based on release name. (#300)

* Upgrade the openvpn chart to latest 4.2.1. (#301)

* Change CLUSTER in Makefile to kiosk-console to fix binary name issue. (#302)

* update raw.gif and tracked.gif with new nearly perfect gif (#303)

* Update default values for tf-serving (#306)

* Update Redis to the latest helm chart before they migrate to bitnami (#307)

* Update autoscaler to 0.4.1 (#308)

* Update redis-janitor to 0.3.1 (#309)

* Update frontend to 0.4.1. (#310)

* Update OpenVPN command for version 4.2.1 (#313)

* Upgrade consumers to 0.5.1 and update models to DeepWatershed. (#311)

* Set no-appendfsync-on-rewrite=yes to prevent Redis latency issues during AOF fsync (#316)

* Install yq in install_script.sh (#319)

* Use 4 random digits for cluster names. (#318)

* update to latest version of the frontend (#322)

* Change default consumer machine type to n1-standard-2 (#323)

* Upgrade benchmarking to 0.2.4 and fix for Deep Watershed models (#324)

* Use GRAFANA_PASSWORD env var to override the default grafana password. (#325)

* Update Getting Started docs with new user feedback (#321)

* Add basic unit tests (#326)

* Use the docker container to run integration tests. (#327)

* Warn users if bucket's region and cluster's region do not match (#329)

* Bump benchmarking to latest 0.2.5 release (#331)

* Add Logo Banner and Update README (#332)

* Add new menu option for default settings with 4 GPUs (#333)

* Update HPA target to 2 keys per zip consumer pod. (#334)

* Bump consumers to version 0.5.2 (#336)

* Update consumer and benchmarking versions (#337)

* Bump redis-janitor to 0.3.2 to fix empty key bug. (#339)

* bump benchmarking to 0.3.1 to fix No route to host bug. (#341)

* Allow users to select which zone(s) to deploy the cluster (#340)

* Pin KUBERNETES_VERSION to 1.14. (#346)

* Fix bug indexing into last array element of valid_zones. (#348)

* Fix logs to indicate finality and be less redundant. (#351)

* If KUBERNETES_VERSION is 1.14, warn user of potential future version removal (#352)

Co-authored-by: dylanbannon <[email protected]>
Co-authored-by: MekWarrior <[email protected]>
  • Loading branch information
3 people authored May 22, 2020
1 parent 059281a commit 38873e3
Show file tree
Hide file tree
Showing 41 changed files with 637 additions and 2,144 deletions.
144 changes: 53 additions & 91 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,114 +6,76 @@ git:

language: shell

services:
- docker

stages:
- unit_tests
- integration_tests
- deploy

jobs:
allow_failures:
- script: make test/integration/gke/deploy/elk
- name: "Integration test - ELK Enabled"
fast_finish: true
include:
- stage: unit_tests
name: "Unit Tests"
install:
- docker build -t kiosk .
- docker run -d -it --env CLOUD_PROVIDER=gke --entrypoint=/bin/bash --name kiosk kiosk
script:
- docker exec kiosk make test/unit
- docker kill kiosk && docker rm kiosk

- stage: integration_tests
name: "Integration test - no ELK"
if: ( (type = pull_request AND branch = master) OR (commit_message =~ /\[build-integration-tests\]/) )
env:
- CHARTS_PATH=/home/travis/build/vanvalenlab/kiosk/conf/charts
- CLOUD_PROVIDER=gke
- CLOUDSDK_BUCKET=deepcell-output-benchmarking
- CLOUDSDK_COMPUTE_REGION=us-west1
- CLOUDSDK_CONFIG=/home/travis/build/vanvalenlab/kiosk/.config/gcloud/
- CLOUDSDK_CORE_PROJECT=deepcell-209717
- CLOUDSDK_CORE_VERBOSITY=debug
- CONF_PATH_PREFIX=/home/travis/build/vanvalenlab/kiosk
- CONSUMER_MACHINE_TYPE=n1-highmem-2
- GCP_PREDICTION_GPU_TYPE=nvidia-tesla-t4
- GCP_TRAINING_GPU_TYPE=nvidia-tesla-v100
- GKE_MACHINE_TYPE=n1-standard-1
- GCP_SERVICE_ACCOUNT=continuous-integration-test@deepcell-209717.iam.gserviceaccount.com
- GPU_MACHINE_TYPE=n1-highmem-2
- GPU_MAX_DIVIDED_BY_FOUR=1
- GPU_MAX_DIVIDED_BY_THREE=1
- GPU_MAX_DIVIDED_BY_TWO=2
- GPU_MAX_TIMES_FIFTY=200
- GPU_MAX_TIMES_FIVE=20
- GPU_MAX_TIMES_FOUR=16
- GPU_MAX_TIMES_FOURTY=160
- GPU_MAX_TIMES_ONE_HUNDRED=400
- GPU_MAX_TIMES_ONE_HUNDRED_FIFTY=600
- GPU_MAX_TIMES_SEVENTY_FIVE=300
- GPU_MAX_TIMES_TEN=40
- GPU_MAX_TIMES_THIRTY=120
- GPU_MAX_TIMES_THREE=12
- GPU_MAX_TIMES_TWENTY=80
- GPU_MAX_TIMES_TWO=8
- GPU_MAX_TIMES_TWO_HUNDRED=800
- GPU_NODE_MIN_SIZE=0
- GPU_NODE_MAX_SIZE=4
- GPU_PER_NODE=1
- KUBERNETES_VERSION=latest
- NODE_MIN_SIZE=1
- NODE_MAX_SIZE=60
- REGION_ZONES_WITH_GPUS=us-west1-a,us-west1-b
name: "Integration Tests - ELK Disabled"
if: ((type = pull_request AND branch = master) OR (commit_message =~ /\[build-integration-tests\]/))
before_install:
- openssl aes-256-cbc -K $encrypted_84460f9bb44c_key -iv $encrypted_84460f9bb44c_iv -in .travis/gcloud_key_base64.enc -out gcloud_key_base64 -d
- cat gcloud_key_base64 | base64 --decode > gcloud_key.json
- export PATH=/home/travis/build/vanvalenlab/kiosk:$PATH
install: ./.travis/install_script.sh
script: make test/integration/gke/deploy
install:
- docker build -t kiosk .
- >
docker run -d -it
--volume $(readlink -f gcloud_key.json):/tmp/keys/gcloud_key.json:ro
--env CLOUD_PROVIDER=gke
--env GCP_SERVICE_ACCOUNT=$GCP_SERVICE_ACCOUNT
--env CLOUDSDK_CORE_PROJECT=$CLOUDSDK_CORE_PROJECT
--env CLOUDSDK_BUCKET=$CLOUDSDK_BUCKET
--env CLOUDSDK_COMPUTE_REGION=us-west1
--env REGION_ZONES_WITH_GPUS=us-west1-a,us-west1-b
--env GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/gcloud_key.json
--entrypoint=/bin/bash
--name kiosk kiosk
script:
- docker exec kiosk make test/integration/gke/deploy
- docker kill kiosk && docker rm kiosk

- stage: integration_tests
name: "Integration test - with ELK"
if: ( (type = pull_request AND branch = master) OR (commit_message =~ /\[build-integration-tests\]/) ) AND (commit_message =~ /\[test-elk\]/)
env:
- CHARTS_PATH=/home/travis/build/vanvalenlab/kiosk/conf/charts
- CLOUD_PROVIDER=gke
- CLOUDSDK_BUCKET=deepcell-output-benchmarking
- CLOUDSDK_COMPUTE_REGION=us-west1
- CLOUDSDK_CONFIG=/home/travis/build/vanvalenlab/kiosk/.config/gcloud/
- CLOUDSDK_CORE_PROJECT=deepcell-209717
- CLOUDSDK_CORE_VERBOSITY=debug
- CONF_PATH_PREFIX=/home/travis/build/vanvalenlab/kiosk
- CONSUMER_MACHINE_TYPE=n1-highmem-2
- GCP_PREDICTION_GPU_TYPE=nvidia-tesla-t4
- GCP_TRAINING_GPU_TYPE=nvidia-tesla-v100
- GKE_MACHINE_TYPE=n1-standard-1
- GCP_SERVICE_ACCOUNT=continuous-integration-test@deepcell-209717.iam.gserviceaccount.com
- GPU_MACHINE_TYPE=n1-highmem-2
- GPU_MAX_DIVIDED_BY_FOUR=1
- GPU_MAX_DIVIDED_BY_THREE=1
- GPU_MAX_DIVIDED_BY_TWO=2
- GPU_MAX_TIMES_FIFTY=200
- GPU_MAX_TIMES_FIVE=20
- GPU_MAX_TIMES_FOUR=16
- GPU_MAX_TIMES_FOURTY=160
- GPU_MAX_TIMES_ONE_HUNDRED=400
- GPU_MAX_TIMES_ONE_HUNDRED_FIFTY=600
- GPU_MAX_TIMES_SEVENTY_FIVE=300
- GPU_MAX_TIMES_TEN=40
- GPU_MAX_TIMES_THIRTY=120
- GPU_MAX_TIMES_THREE=12
- GPU_MAX_TIMES_TWENTY=80
- GPU_MAX_TIMES_TWO=8
- GPU_MAX_TIMES_TWO_HUNDRED=800
- GPU_NODE_MIN_SIZE=0
- GPU_NODE_MAX_SIZE=4
- GPU_PER_NODE=1
- KUBERNETES_VERSION=latest
- NODE_MIN_SIZE=1
- NODE_MAX_SIZE=60
- REGION_ZONES_WITH_GPUS=us-west1-a,us-west1-b
name: "Integration test - ELK Enabled"
if: ((type = pull_request AND branch = master) OR (commit_message =~ /\[build-integration-tests\]/)) AND (commit_message =~ /\[test-elk\]/)
before_install:
- openssl aes-256-cbc -K $encrypted_84460f9bb44c_key -iv $encrypted_84460f9bb44c_iv -in .travis/gcloud_key_base64.enc -out gcloud_key_base64 -d
- cat gcloud_key_base64 | base64 --decode > gcloud_key.json
- export PATH=/home/travis/build/vanvalenlab/kiosk:/home/travis/build/vanvalenlab/kiosk/linux-amd64:$PATH
install: ./.travis/install_script.sh
script: make test/integration/gke/deploy/elk
- stage: unit_tests
name: "Unit Tests"
if: type = push
script: make test/unit
install:
- docker build -t kiosk .
- >
docker run -d -it
--volume $(readlink -f gcloud_key.json):/tmp/keys/gcloud_key.json:ro
--env CLOUD_PROVIDER=gke
--env GCP_SERVICE_ACCOUNT=$GCP_SERVICE_ACCOUNT
--env CLOUDSDK_CORE_PROJECT=$CLOUDSDK_CORE_PROJECT
--env CLOUDSDK_BUCKET=$CLOUDSDK_BUCKET
--env CLOUDSDK_COMPUTE_REGION=us-west1
--env REGION_ZONES_WITH_GPUS=us-west1-a,us-west1-b
--env GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/gcloud_key.json
--entrypoint=/bin/bash
--name kiosk kiosk
script:
- docker exec kiosk make test/integration/gke/deploy/elk
- docker kill kiosk && docker rm kiosk

- stage: deploy
name: "Docker Deployment"
if: tag IS present
Expand Down
33 changes: 0 additions & 33 deletions .travis/install_script.sh

This file was deleted.

8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ If you have improvements to the Kiosk (or its dependencies, ie: redis-consumer,
[how-to](https://help.github.com/articles/using-pull-requests/).

If you want to contribute, start working through the codebase. Navigate to the
[Github "issues" tab](https://github.com/vanvalenlab/kiosk/issues) and start
[Github "issues" tab](https://github.com/vanvalenlab/kiosk-console/issues) and start
looking through interesting issues. If you are not sure of where to start, look for one of the smaller/easier issues here i.e.
[issues with the "good first issue" label](https://github.com/vanvalenlab/kiosk/labels/good%20first%20issue)
[issues with the "good first issue" label](https://github.com/vanvalenlab/kiosk-console/labels/good%20first%20issue)
and then take a look at the
[issues with the "contributions welcome" label](https://github.com/vanvalenlab/kiosk/labels/stat%3Acontributions%20welcome).
[issues with the "contributions welcome" label](https://github.com/vanvalenlab/kiosk-console/labels/stat%3Acontributions%20welcome).
These are issues that we believe are well suited for outside contributions. If you decide to start on an issue, leave a comment so that other people know that you're working on it. If you want to help out, but not alone, use the issue comment thread to coordinate.

### Contribution guidelines and standards

Before sending your pull request for
[review](https://github.com/vanvalenlab/kiosk/pulls),
[review](https://github.com/vanvalenlab/kiosk-console/pulls),
make sure your changes are consistent with the guidelines and follow the
DeepCell coding style.

Expand Down
9 changes: 3 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ FROM cloudposse/geodesic:0.123.1

RUN apk add --update dialog libqrencode

ENV DOCKER_IMAGE="vanvalenlab/kiosk"
ENV DOCKER_IMAGE="vanvalenlab/kiosk-console"
ENV DOCKER_TAG="latest"

# Geodesic banner
Expand Down Expand Up @@ -39,7 +39,7 @@ ENV GPU_NODE_MIN_SIZE="0"
ENV GPU_PER_NODE="1"

# gcloud config
ENV KUBERNETES_VERSION="latest"
ENV KUBERNETES_VERSION="1.14"
ENV CLOUDSDK_CORE_PROJECT=""
ENV CLOUDSDK_CONTAINER_CLUSTER=""
ENV CLOUDSDK_BUCKET=""
Expand All @@ -49,15 +49,12 @@ ENV GCP_PREDICTION_GPU_TYPE="nvidia-tesla-t4"
ENV GCP_TRAINING_GPU_TYPE="nvidia-tesla-v100"
ENV GKE_MACHINE_TYPE="n1-standard-1"
ENV GPU_MACHINE_TYPE="n1-highmem-2"
ENV CONSUMER_MACHINE_TYPE="n1-highmem-2"
ENV CONSUMER_MACHINE_TYPE="n1-standard-2"

# Deployment config
ENV CLOUD_PROVIDER=""
ENV ELK_DEPLOYMENT_TOGGLE=""

# Kiosk filesystem config
ENV CONF_PATH_PREFIX=""

# Filesystem entry for tfstate
RUN s3 fstab '${KOPS_STATE_STORE}' '/' '/s3'

Expand Down
29 changes: 2 additions & 27 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
export CLUSTER ?= kiosk
export CLUSTER ?= kiosk-console
export DOCKER_ORG ?= vanvalenlab
export DOCKER_IMAGE ?= $(DOCKER_ORG)/$(CLUSTER)
export DOCKER_TAG ?= latest
export DOCKER_IMAGE_NAME ?= $(DOCKER_IMAGE):$(DOCKER_TAG)
export DOCKER_BUILD_FLAGS =
export DOCKER_BUILD_FLAGS =
export README_DEPS ?= docs/targets.md
export INSTALL_PATH ?= /usr/local/bin

Expand Down Expand Up @@ -33,28 +33,3 @@ install:
## Start the geodesic shell by calling wrapper script
run:
$(CLUSTER)

## Target for testing cluster deployment
test/integration/gke/deploy: export CLOUDSDK_CONTAINER_CLUSTER = deepcell-test-$(shell bash -c 'echo $$RANDOM')
test/integration/gke/deploy:
# check environment variables
printenv
# check that necessary binaries are installed
helmfile --version
gomplate --version
kubectl version --client
helm version -c
gcloud version
# execute make targets
# make init
cd ./conf && make test/create
cd ./conf && make test/destroy
# celebrate
echo "TESTED"

test/integration/gke/deploy/elk: export ELK_DEPLOYMENT_TOGGLE = ON
test/integration/gke/deploy/elk: \
test/integration/gke/deploy

test/unit:
echo "No unit tests at this time."
39 changes: 21 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,25 @@
# DeepCell Kiosk: A Scalable and User-Friendly Environment for Biological Image Analysis
# ![DeepCell Kiosk Banner](https://raw.githubusercontent.com/vanvalenlab/kiosk-console/master/docs/images/DeepCell_Kiosk_Banner.png)

[![Build Status](https://travis-ci.com/vanvalenlab/kiosk.svg?branch=master)](https://travis-ci.com/vanvalenlab/kiosk)
[![Read the Docs](https://img.shields.io/readthedocs/kiosk?logo=Read%20the%20Docs)](https://deepcell-kiosk.readthedocs.io/en/master)
[![Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/vanvalenlab/kiosk/blob/master/LICENSE)
[![Build Status](https://travis-ci.com/vanvalenlab/kiosk-console.svg?branch=master)](https://travis-ci.com/vanvalenlab/kiosk-console)
[![Read the Docs](https://img.shields.io/readthedocs/kiosk-console?logo=Read%20the%20Docs)](https://deepcell-kiosk.readthedocs.io/en/master)
[![Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/vanvalenlab/kiosk-console/blob/master/LICENSE)

The DeepCell Kiosk is the entry point for users to spin up an end-to-end DeepCell environment in the cloud using [Kubernetes](https://kubernetes.io/). It is designed to allow researchers to easily deploy and scale a deep learning platform for biological image analysis. Once launched, users can drag-and-drop images to be processed in parallel using publicly available, or custom-built, TensorFlow models. To train custom models, please refer to [DeepCell-TF](https://github.com/vanvalenlab/deepcell-tf), which was designed to facilitate model development and export these models for use with the DeepCell Kiosk.
The `kiosk-console` is the entry point for users to spin up a DeepCell Kiosk, a cloud-native implementation of the DeepCell ecosystem.

The scalability of the DeepCell Kiosk software is enabled by [cloud computing](https://en.wikipedia.org/wiki/Cloud_computing). At present, the Kiosk is only compatible with [Google Cloud](https://cloud.google.com/).
The DeepCell Kiosk is designed to allow researchers to easily deploy and scale a deep learning platform for biological image analysis. Once launched, users can drag-and-drop images to be processed in parallel using publicly available, or custom-built, TensorFlow models. To train custom models, please refer to [`deepcell-tf`](https://github.com/vanvalenlab/deepcell-tf), which was designed to facilitate model development and is capable of exporting these models for use with the DeepCell Kiosk.

A running example of the DeepCell Kiosk is live at [DeepCell.org](https://deepcell.org).
The scalability of the DeepCell Kiosk software is enabled by [Kubernetes](https://kubernetes.io/). At present, the Kiosk is only compatible with [Google Cloud](https://cloud.google.com/).

An example of the DeepCell Kiosk is live at [DeepCell.org](https://deepcell.org).

## Features

- Cloud-based deployment of deep-learning models
- Scalable platform that minimizes cost and inference time
- Drag and drop interface for running predictions

### Example
### Examples

<table width="700" border="1" cellpadding="5">

<tr>
Expand All @@ -31,11 +34,11 @@ Tracked Image

<tr>
<td align="center" valign="center">
<img src="https://raw.githubusercontent.com/vanvalenlab/kiosk/master/docs/images/raw.gif" alt="Raw Image" />
<img src="https://raw.githubusercontent.com/vanvalenlab/deepcell-tf/master/docs/images/raw.gif" alt="Raw Image" />
</td>

<td align="center" valign="center">
<img src="https://raw.githubusercontent.com/vanvalenlab/kiosk/master/docs/images/tracked.gif" alt="Tracked Image" />
<img src="https://raw.githubusercontent.com/vanvalenlab/deepcell-tf/master/docs/images/tracked.gif" alt="Tracked Image" />
</td>
</tr>

Expand All @@ -46,16 +49,16 @@ Tracked Image
Start a terminal shell and install the DeepCell Kiosk wrapper script:

```bash
docker run -e DOCKER_TAG=1.1.0 vanvalenlab/kiosk:1.1.0 | sudo bash
docker run -e DOCKER_TAG=1.2.0 vanvalenlab/kiosk-console:1.2.0 | sudo bash
```

To start the kiosk, just run `kiosk` from the terminal shell.
To start the kiosk, just run `kiosk-console` from the terminal shell.

Check out our [docs](https://deepcell-kiosk.readthedocs.io/en/master/GETTING_STARTED.html) for more information on how to start your own kiosk.

## Software Architecture

![Kiosk Architecture](https://raw.githubusercontent.com/vanvalenlab/kiosk/master/docs/images/Kiosk_Architecture.png)
![Kiosk Architecture](https://raw.githubusercontent.com/vanvalenlab/kiosk-console/master/docs/images/Kiosk_Architecture.png)

- [Frontend](https://github.com/vanvalenlab/kiosk-frontend): API for creating and managing jobs, and a React-based web interface for the DeepCell Kiosk.

Expand All @@ -74,17 +77,17 @@ Check out our [docs](https://deepcell-kiosk.readthedocs.io/en/master/GETTING_STA

## Contribute

We welcome contributions to the kiosk. If you are interested, please refer to our [Developer Documentation](https://deepcell-kiosk.readthedocs.io/en/master/DEVELOPER.html), [Code of Conduct](https://github.com/vanvalenlab/kiosk/blob/master/CODE_OF_CONDUCT.md) and [Contributing Guidelines](https://github.com/vanvalenlab/kiosk/blob/master/CONTRIBUTING.md).
We welcome contributions to the kiosk. If you are interested, please refer to our [Developer Documentation](https://deepcell-kiosk.readthedocs.io/en/master/DEVELOPER.html), [Code of Conduct](https://github.com/vanvalenlab/kiosk-console/blob/master/CODE_OF_CONDUCT.md) and [Contributing Guidelines](https://github.com/vanvalenlab/kiosk-console/blob/master/CONTRIBUTING.md).

## Support

Issues are managed through [Github](https://github.com/vanvalenlab/kiosk/issues).
Issues are managed through [Github](https://github.com/vanvalenlab/kiosk-console/issues).
Documentation is hosted on [Read the Docs](https://deepcell-kiosk.readthedocs.io/en/master).
A [FAQ](http://www.deepcell.org.faq) page is also available.
A [FAQ](http://www.deepcell.org/faq) page is also available.

## License

This software is license under a modified [APACHE2](https://opensource.org/licenses/Apache-2.0). See [LICENSE](https://github.com/vanvalenlab/kiosk/blob/master/LICENSE) for full details.
This software is license under a modified Apache-2.0 license. See [LICENSE](https://github.com/vanvalenlab/kiosk-console/blob/master/LICENSE) for full details.

## Trademarks

Expand All @@ -98,5 +101,5 @@ This kiosk was developed with [Cloud Posse, LLC](https://cloudposse.com). They c

## Copyright

Copyright © 2018-2020 [The Van Valen Lab](http://www.vanvalen.caltech.edu/) at the California Institute of Technology (Caltech), with support from the Paul Allen Family Foundation, Google, & National Institutes of Health (NIH) under Grant U24CA224309-01.
Copyright © 2018-2020 [The Van Valen Lab](http://www.vanvalen.caltech.edu/) at the California Institute of Technology (Caltech), with support from the Shurl and Kay Curci Foundation, the Paul Allen Family Foundation, Google, & National Institutes of Health (NIH) under Grant U24CA224309-01.
All rights reserved.
Loading

0 comments on commit 38873e3

Please sign in to comment.