CI/CD for AWS deployments

[pwalsh]

Related to #69 and #77

Description

Write a CI job that demonstrates the AWS deployment target for CCO.

• [need credentials for a real sandbox account on AWS for these tests]
• use terraform default configuration to set up environment required by CCO on AWS:
  • Create RDS instance
  • Create EKS cluster
  • Create IAM roles if necessary
• ckan-cloud-operator cluster initialize should run cleanly
  • no extra step or configuration should be required out of terraform + cco cluster init
• deploy a ckan instance with all required services
• write tests that show CKAN is responding as expected for read and and writes
  • Same ones as the ones running in CI/CD for minikube deployments #77
• teardown the environment on AWS

Acceptance criteria

• Deployment of a CKAN instance into the cluster is green
• CCO and its AWS dependencies are deployed into the GSA sandbox via CircleCI

[adborden]

If this ticket is for GSA, we already have the credentials for CI and the base environment setup. Work should be done in https://github.com/gsa/datagov-infrastructure-live and https://github.com/gsa/datagov-infrastructure-modules unless there is reason not to.

[pwalsh]

@adborden ok.

@akariv cc

[adborden]

Why is the minikube story #77 a dependency?

[pwalsh]

@adborden by ensuring we are super clear in code and documentation about explicit and implicit dependencies on this cloud platform or that, we are paving a clear path for deterministic deployments on all cloud providers (two GSA related - AWS and GCP), and perhaps more importantly we are making it easier to develop and debug without having to standup sandbox environments on these cloud providers.

With the 4 different CI/CD tasks we have defined in here at #77 #78 #79 #80 we are also making it much easier for both the core developers and future contributors to isolate issues as being general (eg reproducible on the minikube environment) or target platform specific (AWS, GCP, Azure).

This work is not all 100% for GSA, we also have some team members working on this via other projects (so, some work is in parallel).

[adborden]

That is great, and I am all for accomplishing these, but in terms of priorities right now, the focus should be on GSA and making CCO work for GSA's infrastructure under GSA's processes. This story seems too big, with too many dependencies. Can we either a) re-scope this story to be more narrow and not include the other dependencies or b) create a new issue that we can prioritize next sprint that works toward GSA's goals?

[akariv]

@aluminiumgeek, please take over this issue - I updated the task list in the description with some more details.

Make sure you build upon @zelima 's work in #77.

[kimwdavidson]

@estebanruseler and @akariv to rework this ticket to be more specific to GSA needs

[rufuspollock]

Discussed with @adborden and did clarify that all of this is needed. Just want to be really clear that this is so we have a GSA-working CI script.

[adborden]

I added this AC:

• CCO and its AWS dependencies are deployed into the GSA sandbox via CircleCI

[adborden]

I don't think the AC was met yet, are we deploying into the GSA cluster as part of CI/CD?

[kimwdavidson]

@estebanruseler to determine status of this ticket

[pwalsh]

@akariv @zelima is this in a state for me to do acceptance testing yet (aside from data.gov specific requirements)?

[zelima]

Not sure about Circle CI but it's green on travis

[pwalsh]

ok. so @akariv just waits for you to confirm on CCI for datagov

[akariv]

Yes, CI is travis-ci and it's deploying to AWS (Datopian Sandbox) on each run.

[adborden]

AC for CCO and its AWS dependencies are deployed into the GSA sandbox via CircleCI isn't met, but we're going to call this done and move on.