Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redesign Build Deployment Process (External) #125

Merged
merged 73 commits into from
Aug 14, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
6aed57a
Create image_build_push.yml
MukuFlash03 Mar 18, 2024
8b0317f
Added docker build and push for dashboard and notebook images + Updat…
Mar 22, 2024
7d3cd86
Merge pull request #2 from MukuFlash03/image-push
MukuFlash03 Mar 22, 2024
355db20
Changed sed to jq + Renamed docker image in image_push
Mar 26, 2024
3c17d18
Merge pull request #3 from MukuFlash03/image-push
MukuFlash03 Mar 26, 2024
e402a61
Added TODO to change image push branch
Mar 26, 2024
ffcdbc8
Merge pull request #4 from MukuFlash03/image-push
MukuFlash03 Mar 26, 2024
d6a6809
Removed printing Docker username
Mar 26, 2024
102899a
Merge pull request #5 from MukuFlash03/image-push
MukuFlash03 Mar 26, 2024
50a7b63
Added cert.sh + Modified Dockerfiles
Mar 28, 2024
db85f75
Merge pull request #6 from MukuFlash03/image-push
MukuFlash03 Mar 28, 2024
2dab7a1
Removing ENV variables from Dockerfile
Apr 3, 2024
43f3d5a
Merge pull request #7 from MukuFlash03/image-push
MukuFlash03 Apr 3, 2024
89ed59a
Removed sed / jq usage from start scripts
Apr 12, 2024
37c7599
Merge pull request #8 from MukuFlash03/image-push
MukuFlash03 Apr 12, 2024
e962d13
Changing base image to build from redesign server image
Apr 12, 2024
8d2464d
Bumped up base server image tag
Apr 16, 2024
d49a275
Merge branch 'image-push-merge' into image-push
MukuFlash03 Apr 16, 2024
efaf216
Merge pull request #9 from MukuFlash03/image-push
MukuFlash03 Apr 16, 2024
baac678
Bump up base server image tag
Apr 16, 2024
629b3a1
Merge pull request #10 from MukuFlash03/image-push
MukuFlash03 Apr 16, 2024
fee9437
Artifact download test - 1
Apr 26, 2024
9622279
Bumped up server image tag
Apr 30, 2024
6e3f2be
Artifact + Matrix - 1
May 2, 2024
1745634
Artifact + Matrix - 2
May 2, 2024
21ca992
Artifact + Matrix - 3
May 2, 2024
58093d3
Artifact + Matrix - 4
May 2, 2024
368900b
Artifact + Matrix - 5
May 3, 2024
bf3e9f7
Updated docker image tag in .env to the latest timestamp:
actions-user May 3, 2024
6eebf87
Updated docker image tag in .env to the latest timestamp: 2024-05-03-…
actions-user May 3, 2024
e760598
Added TODOs in github actions workflow YAML file
May 3, 2024
9444e60
Artifact + Matrix - 6
May 3, 2024
40beb80
Updated docker image tag in .env file to the latest timestamp
actions-user May 3, 2024
29ebf49
Updated docker image tag in .env file to the latest timestamp
actions-user May 3, 2024
1d0a937
Merge branch 'main' into image-push
nataliejschultz May 3, 2024
308eca0
Updated docker image tag in .env file to the latest timestamp
actions-user May 6, 2024
35e09cb
Merge branch 'image-push' into tags-combo-approach
nataliejschultz May 8, 2024
d919a34
Updated docker image tag in .env file to the latest timestamp
actions-user May 8, 2024
11cdafb
Cleanup image_build_push.yml
nataliejschultz May 8, 2024
1dda106
Polishing image_build_push.yml
nataliejschultz May 8, 2024
66bd0a6
Polishing fetch_runID.py
nataliejschultz May 8, 2024
426ae50
Merge branch 'image-push' into tags-combo-approach
nataliejschultz May 8, 2024
3b298fc
Merge pull request #11 from MukuFlash03/tags-combo-approach
nataliejschultz May 8, 2024
4335f39
Cert copy
nataliejschultz May 8, 2024
b330f96
Delete viz_scripts/docker/cert.sh
nataliejschultz May 8, 2024
9b1dca5
Merge branch 'main' into image-push
nataliejschultz May 8, 2024
ab0e157
Update docker-compose.yml
nataliejschultz May 21, 2024
7c78c18
Replacing docker build in image_build_push.yml with docker compose
nataliejschultz May 23, 2024
27c2f42
Delete .env
nataliejschultz May 23, 2024
c8b6dca
Remove whitespace changes to docker-compose.dev.yml
nataliejschultz May 23, 2024
40bf355
Removing redundant pip install
nataliejschultz May 23, 2024
803e9c2
Updating tag name
nataliejschultz May 23, 2024
7ee8e61
Adding artifact upload
nataliejschultz May 24, 2024
af3c2d3
Removing DB_HOST fallback in start_notebook.sh
nataliejschultz May 24, 2024
91d5251
DOCKER_IMAGE_TAG --> SERVER_IMAGE_TAG rename
nataliejschultz May 24, 2024
d3bf89f
Testing workflow
nataliejschultz May 26, 2024
c58b5dd
Testing workflow (again)
nataliejschultz May 26, 2024
14775b5
Push test + adding rename step
nataliejschultz May 26, 2024
f595b0d
dash to dot
nataliejschultz May 26, 2024
2f17e5d
change FROM context
nataliejschultz May 26, 2024
25dcba1
Reverting changes made to test
nataliejschultz May 26, 2024
82cf4e9
Switching to build prod instead of dev
nataliejschultz May 28, 2024
5d7f6d8
Modified tag variable names to be more relevant
Aug 13, 2024
cedf7b1
Updated username to clarify that env file is being updated
Aug 13, 2024
6849f84
Re-added .env file
Aug 14, 2024
f325eb3
Changing images to use docker-compose DEV file
Aug 14, 2024
eb4672c
Tag and Push frontend dashboard image only on Push event
Aug 14, 2024
e87058d
Hardcoded WEB_SERVER_HOST to 0.0.0.0
Aug 14, 2024
7483f2a
Changing to use non-dev version
Aug 14, 2024
2434648
Removing artifact method
Aug 14, 2024
29353b0
Merge branch 'main' into image-push
MukuFlash03 Aug 14, 2024
b98cf53
Added suffix to frontend image
Aug 14, 2024
8153155
Merge branch 'image-push' of https://github.com/MukuFlash03/em-public…
Aug 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/image_build_push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# This is a basic workflow to help you get started with Actions

name: docker-image-push-public-dash

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
push:
# Mukul:
# I've added a local test branch on my system and using it for testing image push.
# So, for testing purposes, need to checkout a branch "image-push-merge"
# TODO: Need to change to build off master or main once it looks good.
branches: [ image-push-merge ]
shankari marked this conversation as resolved.
Show resolved Hide resolved


# Env variable
env:
DOCKER_USER: ${{secrets.DOCKER_USER}}
DOCKER_PASSWORD: ${{secrets.DOCKER_PASSWORD}}

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
- name: docker login
run: | # log into docker hub account
docker login -u $DOCKER_USER -p $DOCKER_PASSWORD

- name: Get current date # get the date of the build
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%d--%M-%S')"

#Runs a single command using the runners shell
- name: Run a one-line script
run: echo running in repo ${GITHUB_REPOSITORY#*/} branch ${GITHUB_REF##*/} on ${{ steps.date.outputs.date }}

# Runs a set of commands using the runners shell
- name: build docker image
run: |
docker build -t $DOCKER_USER/${GITHUB_REPOSITORY#*/}:${GITHUB_REF##*/}_${{ steps.date.outputs.date }} ./frontend
docker build -t $DOCKER_USER/${GITHUB_REPOSITORY#*/}_notebook:${GITHUB_REF##*/}_${{ steps.date.outputs.date }} ./viz_scripts
docker images

- name: push docker image
run: |
docker push $DOCKER_USER/${GITHUB_REPOSITORY#*/}:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}
docker push $DOCKER_USER/${GITHUB_REPOSITORY#*/}_notebook:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}
4 changes: 3 additions & 1 deletion viz_scripts/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# python 3
FROM shankari/e-mission-server:master_2024-02-10--19-38
# Please change once all PR changes are final, so it reads from shankari/e-mission-server
# FROM shankari/e-mission-server:master_2024-02-10--19-38
FROM mukuflash03/e-mission-server:image-push-merge_2024-04-16--49-36
shankari marked this conversation as resolved.
Show resolved Hide resolved

VOLUME /plots

Expand Down
4 changes: 3 additions & 1 deletion viz_scripts/docker/Dockerfile.dev
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# python 3
FROM shankari/e-mission-server:master_2024-02-10--19-38
# FROM shankari/e-mission-server:master_2024-02-10--19-38
# Please change once all PR changes are final, so it reads from shankari/e-mission-server
shankari marked this conversation as resolved.
Show resolved Hide resolved
FROM mukuflash03/e-mission-server:image-push-merge_2024-04-16--49-36

VOLUME /plots

Expand Down
6 changes: 6 additions & 0 deletions viz_scripts/docker/cert.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this script launched from?

Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
if [ -z ${PROD_STAGE} ] ; then
echo “Not in staging / production environment, continuing build...”
shankari marked this conversation as resolved.
Show resolved Hide resolved
elif [ ${PROD_STAGE} = "TRUE" ] ; then
wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem -O /etc/ssl/certs/rds-combined-ca-bundle.pem
echo "In staging / production environment, added AWS certificates"
fi
5 changes: 2 additions & 3 deletions viz_scripts/docker/start_notebook.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that you should also remove the WEB_SERVER_HOST variable in this file since it is unused.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file does use this variable in setting the PYTHONPATH.

if [ -z ${CRON_MODE} ] ; then
    echo "Running notebook in docker, change host:port to localhost:47962 in the URL below"
    PYTHONPATH=/usr/src/app jupyter notebook --no-browser --ip=${WEB_SERVER_HOST} --allow-root
else

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
echo "DB host = "${DB_HOST}
if [ -z ${DB_HOST} ] ; then
local_host=`hostname -i`
sed "s-localhost-${local_host}_" conf/storage/db.conf.sample > conf/storage/db.conf
else
sed "s-localhost-${DB_HOST}-" conf/storage/db.conf.sample > conf/storage/db.conf
export DB_HOST=$local_host
echo "Setting db host environment variable to localhost"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, future fix: this can be removed and simplified (similar to e-mission/e-mission-server#961 (comment))

fi

### configure the saved-notebooks directory for persistent notebooks
Expand Down