NERT-132: Database backup

[KjartanE]

Links to Jira Tickets

• https://apps.nrs.gov.bc.ca/int/jira/browse/NERT-132

Description of Changes

In the openshift bcGov infrastructure we are responsible to backup the Database.
Investigate how that is done by other apps and implement.

https://developer.gov.bc.ca/docs/default/component/platform-developer-docs/docs/database-and-api-management/database-backup-best-practices/#backing-up-your-data

https://github.com/BCDevOps/backup-container

Testing Notes

• {List any relevant testing considerations, necessary pre-reqs, and areas of the app to focus on. Specifically, include anything that will help the reviewers of this PR verify the code is functioning as expected.}

[github-actions]

Openshift URLs for the PR Deployment:

• App Route: https://restoration-tracker-app-158-d83219-dev.apps.silver.devops.gov.bc.ca/
• Api Route: https://restoration-tracker-api-158-d83219-dev.apps.silver.devops.gov.bc.ca/
• Pods: https://console.apps.silver.devops.gov.bc.ca/k8s/ns/d83219-dev/pods?name=158

[KjartanE]

Starting backup server in cron mode ...

Settings:
- Run mode: cron

- Backup strategy: rolling
- Current backup type: daily
- Backups to retain:
  - Daily: 6
  - Weekly: 4
  - Monthly: 1
- Current backup folder: /backups/daily/2024-08-16/
- Time Zone: PDT -0700

- Schedule:
  - 0 1 * * * default ./backup.sh -s
  - 0 4 * * * default ./backup.sh -s -v all

- Container Type: postgres
- Databases (filtered by container type):
  - postgres=restoration-tracker-db-postgresql-dev-deploy:5432/restoration-tracker

- FTP server: �[33mnot configured�[0m
- S3 endpoint: nrs.objectstore.gov.bc.ca
- Webhook Endpoint: �[33mnot configured�[0m
- Environment Friendly Name: NERT DB Backups
- Environment Name (Id): dev

�[34mStarting go-crond as a background task ...
�[0m
time="2024-08-16T12:19:57-07:00" level=info msg="starting go-crond version 23.2.0 (868c01d; go1.19.6) "
time="2024-08-16T12:19:57-07:00" level=info msg="{\"ShowVersion\":false,\"ShowOnlyVersion\":false,\"ShowHelp\":false,\"Cron\":{\"DefaultUser\":\"1009800000\",\"IncludeCronD\":null,\"Auto\":false,\"RunParts\":null,\"RunParts1m\":null,\"RunParts15m\":null,\"RunPartsHourly\":null,\"RunPartsDaily\":null,\"RunPartsWeekly\":null,\"RunPartsMonthly\":null,\"AllowUnprivileged\":true,\"WorkDir\":\"/\",\"EnableUserSwitching\":false},\"Log\":{\"Verbose\":true,\"Json\":false},\"Server\":{\"Bind\":\"\",\"ReadTimeout\":5000000000,\"WriteTimeout\":10000000000,\"Metrics\":false},\"Args\":{\"Crontabs\":[\"backup.conf\"]}}"
time="2024-08-16T12:19:57-07:00" level=warning msg="go-crond is NOT running as root, disabling user switching"
time="2024-08-16T12:19:57-07:00" level=info msg="cronjob added" command="./backup.sh -s" crontab=/backup.conf shell=sh spec="0 1 * * *" user=default
time="2024-08-16T12:19:57-07:00" level=info msg="cronjob added" command="./backup.sh -s -v all" crontab=/backup.conf shell=sh spec="0 4 * * *" user=default
time="2024-08-16T12:19:57-07:00" level=info msg="start runner with 2 jobs\n"

Current backup container startup logs

[KjartanE]

NOTE: Verification cronjob is failing.
Related issue: BCDevOps/backup-container#82
Waiting for linked PR: BCDevOps/backup-container#131

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 44.13%. Comparing base (2998f38) to head (1a09139).
Report is 2 commits behind head on dev.

Additional details and impacted files

@@           Coverage Diff           @@
##              dev     #158   +/-   ##
=======================================
  Coverage   44.13%   44.13%           
=======================================
  Files         289      289           
  Lines        9033     9033           
  Branches     1639     1639           
=======================================
  Hits         3987     3987           
  Misses       4859     4859           
  Partials      187      187           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[KjartanE]

[KjartanE]

Initial backup to s3 functioning YAML ENV variables slightly incorrect placement

[KjartanE]

Initial backup to s3 functioning YAML ENV variables slightly incorrect placement

[KjartanE]

Update S3 backup location to /backups/${environment}/

[KjartanE]

Verifying backup ...

Settings:
- Database: postgres=restoration-tracker-db-postgresql-dev-deploy:5432/restoration_tracker
- Backup file: /backups/daily/2024-08-26/restoration-tracker-db-postgresql-dev-deploy-restoration_tracker_2024-08-26_14-45-09.sql.gz

Already clean ...

waiting for server to start....................................................................................................................
The server failed to start within 0h:0m:0s:0ms:0µs:0ns.


The server started in 0h:0m:0s:0ms:0µs:0ns.


waiting for server to shut down.... done
server stopped
Cleaning up ...

[!!ERROR!!] - Backup verification failed: /backups/daily/2024-08-26/restoration-tracker-db-postgresql-dev-deploy-restoration_tracker_2024-08-26_14-45-09.sql.gz

Elapsed time: 0h:2m:2s:393ms:188µs:902ns - Status Code: 1
Missing PagerDuty service key

Verification process failing currently.
Unsure issue or cause, currently investigating

[oscar-ip]

Awesome!! good work!
as discuss e2e test restoring the database and documenting the process in Jira would be needed

[KjartanE]

sh-5.2$ ./backup.sh -r postgres=restoration-tracker-db-postgresql-dev-deploy:5432/restoration_tracker 

Restoring database ...

Settings:
- Database: postgres=restoration-tracker-db-postgresql-dev-deploy:5432/restoration_tracker
- Backup file: /backups/daily/2024-08-26/restoration-tracker-db-postgresql-dev-deploy-restoration_tracker_2024-08-26_15-19-29.sql.gz

Would you like to continue?  Press Ctrl-C to exit, or any other key to continue ...

Admin password (postgres=restoration-tracker-db-postgresql-dev-deploy:5432/restoration_tracker):

Restoring '/backups/daily/2024-08-26/restoration-tracker-db-postgresql-dev-deploy-restoration_tracker_2024-08-26_15-19-29.sql.gz' to 'restoration-tracker-db-postgresql-dev-deploy:5432/restoration_tracker' ...

DROP DATABASE "restoration_tracker";
DROP DATABASE

CREATE DATABASE "restoration_tracker";
CREATE DATABASE

DROP SCHEMA IF EXISTS metric_helpers CASCADE;
NOTICE:  schema "metric_helpers" does not exist, skipping
DROP SCHEMA
DROP SCHEMA IF EXISTS user_management CASCADE;
NOTICE:  schema "user_management" does not exist, skipping
DROP SCHEMA

GRANT ALL ON DATABASE "restoration_tracker" TO "postgres";
GRANT

SET
SET
SET
ERROR:  role "postgres" already exists

Restore failed.

Restore failed due to know issue: BCDevOps/backup-container#132