Redesign Build Deployment Process (External)

[MukuFlash03]

This PR serves as the implementation for this issue for redesigning our build and deployment processes across e-mission-server, join, admin-dash, public-dash primarily.

Collaborating on this along with @nataliejschultz .

All four redesign PRs for the external repos:
E-mission-server: e-mission/e-mission-server#961
Join: e-mission/nrel-openpath-join-page#29
Admin-dash: e-mission/op-admin-dashboard#113
Public-dash: #125

---

A rough plan can be:

1st phase

• Consolidating Differences
  • Minimize the differences in internal and external versions, cleanup unnecessary files, so on.
• Image_Build_Push.yml:
  • CI / CD via GitHub actions to build and push images to Dockerhub for all four repos based on the current process for e-mission-server.
  • Image build push only builds for that specific repo when merge is made to a specific branch.
  • Additionally, for e-mission-server, image-build-push will now also need to trigger the GitHub action in join, admin-dash, public-dash.

2nd phase:

• Multi-tier uniform repo structure in nrelopenpath
  • Need to change the internal repos to pull from the externally built images for each repo without duplicating the entire external code repository internally.

[MukuFlash03]

Checking in the first initial set of changes for consolidating the differences in the external and internal versions.
These are based on the differences identified here

Current changes involve:

1. Sed changed to jq
   Changed it in this script viz_scripts/docker/start_noteboook.sh to match the way it is being done internally.
   TODO: Might not even need to do this, once we decide how to handle environment variables, and might get rid of using the respective conf files at all.

---

These changes are still pending:
viz_scripts/docker/start_noteboook.sh: Python notebooks run directly internally instead of cron tab
viz_scripts/Dockerfile: AWS certificates.

[nataliejschultz]

I wanted to comment on the addition of cert.sh to the repo. This is a file that is better to be kept internally only, and will be added to the internal container build by using the prod/jenkins variable. However, we can keep it external for testing.

For example, we can add our prod/jenkins variable to our GitHub action, and see what it looks like when the different portions of code are run according to our script. This way we can perfect it before modifying the internal repos.

[nataliejschultz]

More reminders and comment for why we need the script.

[MukuFlash03]

Docker Volume usage

When testing initially, I was unable to see generated plots in the frontend dashboard page.
This is when I learnt about how volumes defined in docker-compose actually work.

So, I was testing with the internal repos,

• Created folders in internal nrelopenpath multi-tier.

  • Created public_dash directory and moved frontend and viz_scripts inside it.
  • Created .gitignored common plots directory in public-dash.

• Added -v flags to both frontend and viz_scripts containers docker run commands.

• This creates two-way connection - any changes in either of the directories before and after colon in -v flag will be updated when changes occur in the other.

  • Thus, viz_scripts should update and store plots in its internal docker plots/
  • Due to volume connection, this should update and also store plots in external local or host plots/
  • Again, frontend is also connected to the external volume, so it should also have access to plots/ in its internal docker public/plots/

---

Great! It’s working. Plots visible.

• Plots do have the label - “Unable to generate plots”, though which come up as empty tables with 0 (zero) values.

  • Hopefully it’s there for internal repo main branch as well.
    • Yes, same message of missing plots. Perhaps unable to read from db as no data loaded. [Confirmed, this was the reason.]

• Run “docker run” command inside nrelopenpath/public_dash/ directory, else plots/ created outside public_dash in root nrelopenpath folder.

  • No issues as such, but let’s keep all public-dash stuff inside its own directory.

[shankari]

@MukuFlash03 one more minor change. But changing the repo name later will be complicated, so let's try to get this right the first time.

[shankari]

Let's make sure that both the notebook image and the webserver image are labeled. Otherwise just seeing a "dashboard" image will cause confusion.

Suggested change

	docker image tag em-pub-dash/frontend:latest $DOCKER_USER/${GITHUB_REPOSITORY#*/}:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}
	docker image tag em-pub-dash/frontend:latest $DOCKER_USER/${GITHUB_REPOSITORY#*/}_frontend:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}

[shankari]

Suggested change

	docker push $DOCKER_USER/${GITHUB_REPOSITORY#*/}:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}
	docker push $DOCKER_USER/${GITHUB_REPOSITORY#*/}_frontend:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}

[shankari]

Squash-merging again (similar to e-mission/e-mission-server#961 (comment)) since otherwise we have 73 commits for 8 files.

[shankari]

@MukuFlash03 This push generated a build which failed: https://github.com/e-mission/em-public-dashboard/actions/runs/10393542152
Seems to be related to the recent changes around not building the frontend for workflow_dispatch.
Did you test those changes?

[MukuFlash03]

This push generated a build which failed: https://github.com/e-mission/em-public-dashboard/actions/runs/10393542152

We are using the non-dev version of the docker compose file to build the image.
This renames the image with the suffix :

dashboard:
    image: em-pub-dash-prod/frontend

The fix should be to simply just change the name for both frontend and notebook images to add the prod suffix.

[MukuFlash03]

@nataliejschultz had added the commit but it was using the incorrect file name.

So I changed the file name in this commit. But I ended up changing the image name as well in this commit, which is wrong.

[shankari]

@MukuFlash03 Note that, with these changes, the frontend and notebook images will have different tags. While you are changing this, you may want to change the final step to upload an artifact with two tags (or two artifacts, one for each image). @nataliejschultz will then have to change the internal script to handle both tags

[MukuFlash03]

@MukuFlash03 Note that, with these changes, the frontend and notebook images will have different tags. While you are changing this, you may want to change the final step to upload an artifact with two tags (or two artifacts, one for each image). @nataliejschultz will then have to change the internal script to handle both tags

We do not need changes for this since the tag is the timestamp generated:

    - name: push docker images
      run: |
        if [ "${{ github.event_name }}" == "push" ]; then
          docker push $DOCKER_USER/${GITHUB_REPOSITORY#*/}_frontend:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}
        fi
        docker push $DOCKER_USER/${GITHUB_REPOSITORY#*/}_notebook:${GITHUB_REF##*/}_${{ steps.date.outputs.date }}
   
    - name: Create a text file
      run: |
          echo ${{ steps.date.outputs.date }} > public_dash_tag_file.txt
          echo "Created tag text file"

This is the same for all images in the workflow run. So internally, the image name without the timestamp differs for frontend and notebook images which is hardcoded anyways as it is static. Only the timestamp tag changes based on the workflow run.

[shankari]

This is the same for all images in the workflow run. So internally, the image name without the timestamp differs for frontend and notebook images which is hardcoded anyways as it is static. Only the timestamp tag changes based on the workflow run.

@MukuFlash03 For the workflow_dispatch runs, it is not.
Consider the following:

• push: both images are tagged with 2024-08-14
• new server build on 2024-08-16, viz_script is tagged with 2024-08-16, frontend is still at 2024-08-14

[MukuFlash03]

I see.
In that case, @nataliejschultz can handle this in the internal script by checking for the type of trigger event.
I did inspect the response from the GitHub REST API (URL for my branch workflows)

The event type is available:

      "run_number": 49,
      "event": "push",
      "status": "completed",
      "conclusion": "success",
      "workflow_id": 90180351,
....
....
....
      "run_number": 31,
      "event": "workflow_dispatch",
      "status": "completed",
      "conclusion": "success",
      "workflow_id": 90180351,
``

[shankari]

@MukuFlash03 Just because something is possible, doesn't mean that it is the elegant approach.

If you see the script that @nataliejschultz I think it would be a lot easier to just download two archive files. Note that she has a common function to download the archive file which is invoked once for each tag. having to looking at the type for only this repo would make that script a lot more ugly.

[MukuFlash03]

I tried implementing the logic for handling two different tags for frontend and notebook but it's not correct.
I'm trying to think it through but we don't maintain the separate tags for frontend / notebook in the workflow currently.

---

For push events, we can have two artifacts. They'll have the same timestamps as tags.
For workflow, for the notebook image we'll use the latest timestamp for the 1st artifact.

But what should be passed as the 2nd artifact for frontend image?
We'll need to retrieve the last stored tag for frontend from somewhere?

[MukuFlash03]

I tested the end-to-end flow with workflow dispatch from the server repo in these runs:

• server, public-dash, admin-dash.

Well, two tags are generated in the public-dash workflow run, but the tags are wrong.

Notebook tag is correct since it is fetched from current timestamp in either case.

But frontend tag is storing the server image tag.
It should store last tagged frontend image.

---

So, as mentioned in this comment, I'm thinking how to retrieve the last tag for frontend image. This will need to be stored / maintained somewhere as well.

[shankari]

I can think of tons of options - here are two off the top of my head

• you can just save it in a different .env file which is overridden on push but not on workflow_dispatch
• Or you can remove the optimization and rebuild both images every time, arguing that it makes the downstream flow easier and that the frontend image size is small (you have to check and actually make that argument)

[MukuFlash03]

Frontend image is small ~ 50 MB [Source: My dockerhub account]
But I've gone ahead and implemented it using a different .env file.

[MukuFlash03]

1. Push event run

This workflow run was triggered by a PUSH event.
The tags were updated correctly and two artifacts produced (FRONTEND, NOTEBOOK).

---

2. Workflow Dispatch run [server, public-dash, admin-dash]

Next, I triggered a workflow run from e-mission-server via Workflow Dispatch event.

This was able to read in the current FRONTEND image tag from the .env.repoTags file and upload the existing value as an artifact.
NOTEBOOK tag on the other hand is updated normally as it should irrespective of event type.