Mcdaniel meta data (#49)

* add module to gather environment state data of current user

* add cookiecutter meta tags

* force recreation of meta resource on each apply

* persist current version to a file that Terraform can read and add to its state

* add cookiecutter meta tags

* add cookiecutter meta tags

* add cookiecutter meta tags

* de-abstract tags

* testing

* change the source of the taint. switch template_file to local_file

* refactor all elements into a single null_resourc that is tainted by the last commit

* add local_file resources now depent on null_resource 'environment'

* lint

* add meta tags

* documentation

* prototype a kubernetes stack module

* testing

* add tags to iam policy

* revert to nutmeg.3 and set TUTOR_OPENEDX_COMMON_VERSION in build workflow

* revert to nutmeg.2 with tutor 14.2.4

* add more settings defaults

* add more template tags

* add meta, module and resource tags

* add meta, module and resource tags

* terraform fmt -recursive

* change cookiecutter meta secrets names

* persist config dump to k8s secrets

* ci_openedx_actions_tutor_print_dump=v1.0.2

* ci_openedx_actions_tutor_print_dump=v1.0.3

* refactor provider declarations into separate modules

* rename module

* rename module

* documentation

* documentation

* add eduNEXT proxy service for scorm backends

* DRY the aws s3 bucket names

* ci_openedx_actions_tutor_print_dump=v1.0.4

* rename module from s3_openedx_storage to s3

* remove duplicate variable declaration

* add release notes

* lint

CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [1.1.0] (2023-3-17)
+
+Lots of new functionality added related to gathering and storing meta data about the exact environment that was used to create AWS resources via Terraform. Also added functionality to gather and persist as much information as possible about build and deploy CI work flows.
+
+- add module to gather environment state data of current user
+- add cookiecutter meta tags for AWS resources
+- revert to installing nutmeg.2 by default
+- gather and persist CI build and deploy meta data in new k8s secrets
+- add scorm proxy service to backend file storage based on eduNEXT prototype
+
 ## [1.0.26] (2023-3-8)
 
 - bug fix: settings_merge.yml PREVIEW_LMS_BASE
@@ -272,7 +282,7 @@ General production release
 - resolved deprecation warnings in all modules
 - restructured terraform folders
 - fixed a bug that was causing multiple SSL/TLS certificates to be created in both us-east-1 as well as the environment region
-- added the text 'openedx_devops' to the descriptions of all security groups, IAM roles, and IAM policies resources that are explicitly created by this repository
+- added the text 'cookiecutter' to the descriptions of all security groups, IAM roles, and IAM policies resources that are explicitly created by this repository
 
 
 ## [0.0.3] - 2022-03-20

README.rst

@@ -141,7 +141,7 @@ Create a Github repo and push it there:
   git add .
   git commit -m "first commit"
   git branch -M main
-  git remote add origin https://github.com/lpm0073/openedx_devops.git
+  git remote add origin https://github.com/youraccount/{{ cookiecutter.github_repo_name }}.git
   git push -u origin main
 
 Now take a look at your repo. Don't forget to carefully look at the generated README. Awesome, right?

cookiecutter.json

@@ -16,6 +16,8 @@
     "global_aws_route53_hosted_zone_id": "Z1234567ABCDE1U23DEF",
     "global_aws_region": "us-east-1",
     "global_account_id": "123456789012",
+    "global_google_analytics_account": "SET-ME-PLEASE",
+    "global_language_code": "en",
     "stack_add_bastion": ["Y", "N"],
     "stack_add_bastion_openedx_dev_environment": ["N", "Y"],
     "stack_add_k8s_dashboard": ["Y", "N"],
@@ -26,7 +28,7 @@
     "stack_add_remote_mysql":  ["Y", "N"],
     "stack_add_remote_mongodb": ["Y", "N"],
     "stack_add_remote_redis":  ["Y", "N"],
-    "ci_build_tutor_version": "15.2.0",
+    "ci_build_tutor_version": "14.2.4",
     "ci_build_kubectl_version": "1.25/stable",
     "kubernetes_cluster_version": "1.25",
     "ci_build_theme_repository": "edx-theme-example",
@@ -38,7 +40,7 @@
     "ci_build_xblock_org": "openedx",
     "ci_build_xblock_repository": "edx-ora2",
     "ci_build_xblock_ref": "master",
-    "ci_deploy_open_edx_version": "olive.1",
+    "ci_deploy_open_edx_version": "nutmeg.2",
     "ci_deploy_install_backup_plugin": ["N", "Y"],
     "ci_deploy_install_credentials_server": ["N", "Y"],
     "ci_deploy_install_discovery_service": ["Y", "N"],
@@ -66,7 +68,7 @@
     "ci_openedx_actions_tutor_k8s_configure_mongodb_version": "v1.0.1",
     "ci_openedx_actions_tutor_k8s_configure_redis_version": "v1.0.0",
     "ci_openedx_actions_tutor_k8s_configure_smtp_version": "v1.0.0",
-    "ci_openedx_actions_tutor_print_dump": "v1.0.0",
+    "ci_openedx_actions_tutor_print_dump": "v1.0.4",
     "ci_openedx_actions_tutor_plugin_build_backup_version": "v0.1.7",
     "ci_openedx_actions_tutor_plugin_build_credentials_version": "v1.0.0",
     "ci_openedx_actions_tutor_plugin_build_license_manager_version": "v0.0.2",
@@ -117,26 +119,26 @@
     "redis_port": 6379,
     "redis_family": "redis6.x",
     "terraform_required_version": "~> 1.3",
-    "terraform_aws_modules_acm": "~> 4.3",
-    "terraform_aws_modules_cloudfront": "~> 3.1",
-    "terraform_aws_modules_eks": "~> 19.4",
+    "terraform_aws_modules_acm": "4.3",
+    "terraform_aws_modules_cloudfront": "3.1",
+    "terraform_aws_modules_eks": "19.4",
     "terraform_aws_modules_iam": "~> 5.9",
-    "terraform_aws_modules_iam_assumable_role_with_oidc": "~> 5.10",
-    "terraform_aws_modules_rds": "~> 5.2",
-    "terraform_aws_modules_s3": "~> 3.6",
-    "terraform_aws_modules_sg": "~> 4.16",
-    "terraform_aws_modules_vpc": "~> 3.18",
-    "terraform_helm_cert_manager": "~> 1.11",
-    "terraform_helm_ingress_nginx_controller": "~> 4.4",
-    "terraform_helm_vertical_pod_autoscaler": "~> 6.0",
-    "terraform_helm_karpenter": "~> 0.16",
-    "terraform_helm_dashboard": "~> 6.0",
-    "terraform_helm_kubeapps": "~> 12.2",
-    "terraform_helm_kubecost": "~> 1.100",
-    "terraform_helm_metrics_server": "~> 3.8",
+    "terraform_aws_modules_iam_assumable_role_with_oidc": "5.10",
+    "terraform_aws_modules_rds": "5.2",
+    "terraform_aws_modules_s3": "3.6",
+    "terraform_aws_modules_sg": "4.16",
+    "terraform_aws_modules_vpc": "3.18",
+    "terraform_helm_cert_manager": "1.11",
+    "terraform_helm_ingress_nginx_controller": "4.4",
+    "terraform_helm_vertical_pod_autoscaler": "6.0",
+    "terraform_helm_karpenter": "0.16",
+    "terraform_helm_dashboard": "6.0",
+    "terraform_helm_kubeapps": "12.2",
+    "terraform_helm_kubecost": "1.100",
+    "terraform_helm_metrics_server": "3.8",
     "terraform_helm_prometheus": "39.6.0",
     "terraform_provider_kubernetes_version": "~> 2.16",
-    "terraform_provider_hashicorp_aws_version": "~> 4.48",
+    "terraform_provider_hashicorp_aws_version": "4.48",
     "terraform_provider_hashicorp_local_version": "~> 2.2",
     "terraform_provider_hashicorp_random_version": "~> 3.4",
     "terraform_provider_hashicorp_kubectl_version": "~> 1.14",

{{cookiecutter.github_repo_name}}/.github/workflows/build-openedx.yml

@@ -39,6 +39,11 @@ jobs:
           aws-region: {% raw %}${{ env.AWS_REGION }}{% endraw %}
           tutor-version: "{{ cookiecutter.ci_build_tutor_version }}"
 
+      - name: Load additional environment specific settings
+        shell: bash
+        run: |-
+          echo "TUTOR_OPENEDX_COMMON_VERSION=open-release/{{ cookiecutter.ci_deploy_open_edx_version }}" >> $GITHUB_ENV
+
       #------------------------------------------------------------------------
       # Add a custom theme here.
       #------------------------------------------------------------------------
@@ -88,6 +93,9 @@ jobs:
 
       - name: Dump tutor config
         uses: openedx-actions/tutor-print-dump@{{ cookiecutter.ci_openedx_actions_tutor_print_dump }}
+        with:
+          namespace: {% raw %}${{ env.NAMESPACE }}{% endraw %}
+          action: build
 
       #------------------------------------------------------------------------
       # Build and upload the Docker container

{{cookiecutter.github_repo_name}}/.github/workflows/deploy-{{cookiecutter.environment_name}}.yml

@@ -130,7 +130,7 @@ jobs:
           echo "TUTOR_LMS_HOST=$LMS_HOSTNAME" >> $GITHUB_ENV
           echo "TUTOR_CMS_HOST=$CMS_HOSTNAME" >> $GITHUB_ENV
           echo "TUTOR_DOCKER_IMAGE_OPENEDX=${AWS_ECR_REPOSITORY_OPENEDX}" >> $GITHUB_ENV
-          echo "OPENEDX_COMMON_VERSION=open-release/{{ cookiecutter.ci_deploy_open_edx_version }}" >> $GITHUB_ENV
+          echo "TUTOR_OPENEDX_COMMON_VERSION=open-release/{{ cookiecutter.ci_deploy_open_edx_version }}" >> $GITHUB_ENV
 
       # ---------------------------------------------------------------------------------
       # Configure optional tutor plugins
@@ -319,6 +319,9 @@ jobs:
 
       - name: Dump tutor config
         uses: openedx-actions/tutor-print-dump@{{ cookiecutter.ci_openedx_actions_tutor_print_dump }}
+        with:
+          namespace: {% raw %}${{ env.NAMESPACE }}{% endraw %}
+          action: deploy
 
       # -----------------------------------------------------------------------
       # Deploy

{{cookiecutter.github_repo_name}}/VERSION

@@ -0,0 +1 @@
+v1.0.27

{{cookiecutter.github_repo_name}}/ci/tutor-deploy/environments/{{cookiecutter.environment_name}}/settings_merge.yml

@@ -1,73 +1,117 @@
 ---
+ACTIVATION_EMAIL_SUPPORT_LINK: https://{{ cookiecutter.global_root_domain }}/support/
+AUTH_PASSWORD_VALIDATORS:
+-   NAME: django.contrib.auth.password_validation.UserAttributeSimilarityValidator
+-   NAME: common.djangoapps.util.password_policy_validators.MinimumLengthValidator
+    OPTIONS:
+        min_length: 8
+-   NAME: common.djangoapps.util.password_policy_validators.MaximumLengthValidator
+    OPTIONS:
+        max_length: 75
+AWS_SES_REGION_ENDPOINT: email.{{ cookiecutter.global_aws_region }}.amazonaws.com
+AWS_SES_REGION_NAME: "{{ cookiecutter.global_aws_region }}"
 CORS_ORIGIN_ALLOW_ALL: true
 CORS_ORIGIN_WHITELIST:
 - https://{{ cookiecutter.global_root_domain }}
 - https://{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
 - https://{{ cookiecutter.environment_studio_subdomain }}.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
 - https://apps.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
+COURSE_ABOUT_VISIBILITY_PERMISSION: see_about_page
+COURSE_CATALOG_VISIBILITY_PERMISSION: see_in_catalog
+CREDIT_HELP_LINK_URL: https://{{ cookiecutter.global_root_domain }}/support/
 CROSS_DOMAIN_CSRF_COOKIE_DOMAIN: "{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}"
 CROSS_DOMAIN_CSRF_COOKIE_NAME: native-csrf-cookie
 CSRF_COOKIE_SECURE: true
-CSRF_TRUSTED_ORIGINS: []
+CSRF_TRUSTED_ORIGINS:
+- https://apps.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
 DCS_SESSION_COOKIE_SAMESITE: lax
 DCS_SESSION_COOKIE_SAMESITE_FORCE_ALL: true
-ACTIVATION_EMAIL_SUPPORT_LINK: https://{{ cookiecutter.global_root_domain }}/support/
-AWS_SES_REGION_ENDPOINT: email.{{ cookiecutter.global_aws_region }}.amazonaws.com
-AWS_SES_REGION_NAME: "{{ cookiecutter.global_aws_region }}"
-CREDIT_HELP_LINK_URL: https://{{ cookiecutter.global_root_domain }}/support/
 DEFAULT_MOBILE_AVAILABLE: false
 DEFAULT_EMAIL_LOGO_URL: https://cdn.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
 ENTERPRISE_SUPPORT_URL: https://{{ cookiecutter.global_root_domain }}/support/
 ENTERPRISE_TAGLINE: "{{ cookiecutter.global_platform_name }}"
 FACEBOOK_API_VERSION: v12.0
 FEATURES:
-  ENABLE_CHANGE_USER_PASSWORD_ADMIN: true
-  CERTIFICATES_HTML_VIEW: true
-  PREVIEW_LMS_BASE: "preview.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain  }}"
-  ENABLE_COURSEWARE_INDEX: true
-  ENABLE_CSMH_EXTENDED: false
-  ENABLE_LEARNER_RECORDS: true
-  ENABLE_LIBRARY_INDEX: true
-  MILESTONES_APP: true
-  ENABLE_PREREQUISITE_COURSES: true
-  ENABLE_DASHBOARD_SEARCH: false
+  ALLOW_ALL_ADVANCED_COMPONENTS: true
+  ALLOW_HIDING_DISCUSSION_TAB: true
   AUTH_USE_OPENID_PROVIDER: false
+  AUTH_USE_OPENID: false
   AUTOMATIC_AUTH_FOR_TESTING: false
+  CERTIFICATES_ENABLED: true
+  CERTIFICATES_HTML_VIEW: true
+  CUSTOM_CERTIFICATE_TEMPLATES_ENABLED: true
   CUSTOM_COURSES_EDX: false
+  ENABLE_ACCOUNT_DELETION: true
   ENABLE_BULK_ENROLLMENT_VIEW: true
+  ENABLE_CHANGE_USER_PASSWORD_ADMIN: true
   ENABLE_COMBINED_LOGIN_REGISTRATION: true
   ENABLE_CORS_HEADERS: true
   ENABLE_COUNTRY_ACCESS: false
+  ENABLE_COURSEWARE_INDEX: true
+  ENABLE_COURSEWARE_MICROFRONTEND: false
   ENABLE_CREDIT_API: false
   ENABLE_CREDIT_ELIGIBILITY: false
   ENABLE_CROSS_DOMAIN_CSRF_COOKIE: true
-  ENABLE_DISCUSSION_HOME_PANEL: false
-  ENABLE_DISCUSSION_SERVICE: false
+  ENABLE_CSMH_EXTENDED: false
+  ENABLE_DASHBOARD_SEARCH: true
+  ENABLE_DISCUSSION_EMAIL_DIGEST: true
+  ENABLE_DISCUSSION_HOME_PANEL: true
+  ENABLE_DISCUSSION_SERVICE: true
+  ENABLE_DJANGO_ADMIN_SITE: true
   ENABLE_EDXNOTES: true
   ENABLE_ENROLLMENT_RESET: true
   ENABLE_EXPORT_GIT: false
   ENABLE_GRADE_DOWNLOADS: true
   ENABLE_INSTRUCTOR_ANALYTICS: true
+  ENABLE_INSTRUCTOR_EMAIL: true
+  ENABLE_LEARNER_RECORDS: true
+  ENABLE_LIBRARY_INDEX: true
   ENABLE_LTI_PROVIDER: false
   ENABLE_MKTG_SITE: false
   ENABLE_MOBILE_REST_API: true
   ENABLE_OAUTH2_PROVIDER: true
+  ENABLE_PEARSON_HACK_TEST: false
+  ENABLE_PREREQUISITE_COURSES: true
   ENABLE_PUBLISHER: false
   ENABLE_READING_FROM_MULTIPLE_HISTORY_TABLES: false
   ENABLE_SPECIAL_EXAMS: false
   ENABLE_SYSADMIN_DASHBOARD: true
   ENABLE_THIRD_PARTY_AUTH: true
   ENABLE_VIDEO_UPLOAD_PIPELINE: false
+  ENABLE_XBLOCK_VIEW_ENDPOINT: true
+  MILESTONES_APP: true
+  ORGANIZATIONS_APP: true
+  PREVENT_CONCURRENT_LOGINS: true
+  PREVIEW_LMS_BASE: preview.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain  }}
   SHOW_FOOTER_LANGUAGE_SELECTOR: false
   SHOW_HEADER_LANGUAGE_SELECTOR: false
+GOOGLE_ANALYTICS_ACCOUNT: {{ cookiecutter.global_google_analytics_account }}
+HEARTBEAT_EXTENDED_CHECKS:
+- openedx.core.djangoapps.heartbeat.default_checks.check_celery
+- openedx.core.djangoapps.django_comment_common.comment_client.utils.check_forum_heartbeat
 ID_VERIFICATION_SUPPORT_LINK: https://{{ cookiecutter.global_root_domain }}/support/
-LANGUAGE_CODE: en
+LANGUAGE_CODE: {{ cookiecutter.global_language_code }}
 LANGUAGE_COOKIE: openedx-language-preference
+LOGIN_REDIRECT_WHITELIST:
+- https://{{ cookiecutter.environment_studio_subdomain }}.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
+- https://apps.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}
 LOGO_URL_PNG: "https://cdn.{{ cookiecutter.environment_subdomain }}.{{ cookiecutter.global_root_domain }}"
+MKTG_URL_OVERRIDES:
+  ABOUT: '#'
+  BLOG: '#'
+  DONATE: '#'
+  PRIVACY: '#'
+  TOS: '#'
+PARENTAL_CONSENT_AGE_LIMIT: 13
 PLATFORM_DESCRIPTION: "{{ cookiecutter.global_platform_description }}"
 PLATFORM_FACEBOOK_ACCOUNT: http://www.facebook.com/
 PLATFORM_NAME: "{{ cookiecutter.global_platform_name }}"
 PLATFORM_TWITTER_ACCOUNT: ''
+PROFILE_IMAGE_SIZES_MAP:
+    full: 500
+    large: 120
+    medium: 50
+    small: 30
 REGISTRATION_EXTRA_FIELDS:
   city: hidden
   confirm_email: hidden
@@ -93,7 +137,13 @@ SUPPORT_SITE_LINK: https://{{ cookiecutter.global_root_domain }}/support/
 TIME_ZONE: America/New_York
 THIRD_PARTY_AUTH_BACKENDS:
 - social_core.backends.google.GoogleOAuth2
+- social_core.backends.linkedin.LinkedinOAuth2
 - social_core.backends.facebook.FacebookOAuth2
+- social_core.backends.azuread.AzureADOAuth2
+- common.djangoapps.third_party_auth.appleid.AppleIdAuth
+- common.djangoapps.third_party_auth.identityserver3.IdentityServer3
+- common.djangoapps.third_party_auth.saml.SAMLAuthBackend
+- common.djangoapps.third_party_auth.lti.LTIAuthBackend
 WIKI_ENABLED: false
 API_ACCESS_FROM_EMAIL: api-requests@{{ cookiecutter.global_root_domain }}
 API_ACCESS_MANAGER_EMAIL: api-access@{{ cookiecutter.global_root_domain }}

{{cookiecutter.github_repo_name}}/doc/DATA_BACKUP.md

@@ -35,7 +35,7 @@ MongoDB script source: [openedx-backup-mongodb.sh](../terraform/stacks/modules/e
 
 Terraform creates a dedicated AWS S3 bucket, {{ cookiecutter.environment_name }}-{{ cookiecutter.global_platform_name }}-{{ cookiecutter.global_platform_region }}-backup.s3.amazonaws.com, for archiving backups. This bucket does not provide public access. Note that it is preconfigured with a lifecycle policy to retain large files (greater than 1Gb) for 30 days.
 
-See Terraform source code: [openedx_backups.tf](../terraform/environments/modules/s3_openedx_storage/openedx_backups.tf)
+See Terraform source code: [openedx_backups.tf](../terraform/environments/modules/s3/openedx_backups.tf)
 
 ## Local storage
 

{{cookiecutter.github_repo_name}}/terraform/common/cookiecutter_meta/README.md

@@ -0,0 +1,30 @@
+# Cookiecutter Meta
+
+Collects and persists meta data about the current user's environment. Data collected is made visible to Terraform by persisting each data element its own .state file in ./output. These files in turn are exposed within Terraform using "data" declarations.
+
+Cookiecutter Meta is referenced by all modules contained in [environments](../../environments/) and [stacks](../../stacks/) and is ultimated formatted into AWS resource tag elements that are persisted into every AWS resource created by the Terraform scripts contained in this repository.
+
+## Meta Data
+
+Collects the following about your operating environment:
+
+- AWS Command-line interface version number
+- The current git branch of this repository
+- The most recent git commit date from this repository
+- The sha of the most recent git commit from this repository
+- The AWS IAM ARN which contains the key-secret in use for the awscli
+- Kubectl current version
+- The name and version of your computer's operating system
+- Terraform current version
+- Timestamp of the last time this module was executed
+- Cookiecutter version
+
+## Usage
+
+Run this module separately and as needed.
+
+```bash
+  terraform init    # prepare this module to run by downloading all referenced Terraform modules and providers
+  terraform plan    # echo a work plan to the console
+  terraform apply   # run this module
+```