Deploying Cumulus 18.5.1 Notes + Manual changes

[wphyojpl]

• Most changes are done in unity-uds-terraform and cumulus-deploy terraform which are pushed to github.
• One of them is upgrading to Java11 for cnm_to_cma_task and cnm_response_task lambdas.
• It is done by upgrading the version.
• unity-uds-terraform is looking for SNS which doesn't exist in a fresh install.
• It is fixed by passing SNS arn straight away instead of SNS name for unity-uds to look for the resource to get the arn.

There are 3 changes which are done directly in cumulus terraform

• Error 1:

│ Error: creating IAM Role (nnn-nnn-nnnnnnn-dla-migration): AccessDenied: User: arn:aws:sts::xxxxxxxxxxxxx:assumed-role/mcp-tenantOperator/user_x is not authorized to perform: 
iam:CreateRole on resource: arn:aws:iam::xxxxxxxxxxxxx:role/nnn-nnn-nnnnnnn-dla-migration because no identity-based policy allows the iam:CreateRole action
│       status code: 403, request id: 11a856d2-d757-459d-8ca7-ef701a9c0e3f
│ 
│   with module.cumulus.module.dla_migration_lambda.aws_iam_role.dla_migration_role,
│   on .terraform/modules/cumulus/lambdas/dla-migration/iam.tf line 27, in resource "aws_iam_role" "dla_migration_role":
│   27: resource "aws_iam_role" "dla_migration_role" {

permissions_boundary has to be hardcoded to "arn:aws:iam::xxxxxxxxxxxxx:policy/mcp-tenantOperator-AMI-APIG" to fix that
in /cumulus-tf/.terraform/modules/cumulus/lambdas/dla-migration/iam.tf

resource "aws_iam_role" "dla_migration_role" {
  name                 = "${var.prefix}-dla-migration"
  assume_role_policy   = data.aws_iam_policy_document.dla_migration_assume_role_policy.json
  permissions_boundary = var.permissions_boundary_arn
  tags                 = var.tags
}

• Error 2:

│ Error: invoking Lambda Function (nnn-nnn-nnnnnnn-CustomBootstrap): returned error: "{"errorType":"ResponseError","errorMessage":"Response Error","trace":["ResponseError: Response Error","    
at IncomingMessage.<anonymous> (/var/task/es-client/node_modules/@elastic/elasticsearch/lib/Transport.js:289:1)","    
at IncomingMessage.emit (node:events:531:35)","    at endReadableNT (node:internal/streams/readable:1696:12)","    
at processTicksAndRejections (node:internal/process/task_queues:82:21)"]}"
│ 
│   with module.cumulus.module.archive.data.aws_lambda_invocation.custom_bootstrap[0],
│   on .terraform/modules/cumulus/tf-modules/archive/bootstrap.tf line 29, in data "aws_lambda_invocation" "custom_bootstrap":
│   29: data "aws_lambda_invocation" "custom_bootstrap" {

Cloudwatch logs:

2024-11-19T17:13:56.114Z 8c904e0c-bfc9-4cb0-a0ec-8c602f11b09c ERROR Invoke Error {"errorType":"ResponseError","errorMessage":"Response Error","name":"ResponseError","meta":
  {"body":{"Message":"User: arn:aws:sts::xxxxxxxxxxxxx:assumed-role/nnn-nnn-nnnnnnn-lambda-processing/nnn-nnn-nnnnnnn-CustomBootstrap is not authorized to perform: 
    es:ESHttpPut because no identity-based policy allows the es:ESHttpPut action"},"statusCode":403,"headers":{"date":"Tue, 19 Nov 2024 17:13:55 GMT","content-type":"application/json","content-length":"231","connection":"keep-alive","x-amzn-requestid":"2ba60944-533a-4401-82c0-880b8c92e0c7","access-control-allow-origin":"*"},"warnings":null,"meta":{"context":null,"request":{"params":{"method":"PUT","path":"/_cluster/settings","body":"{\"persistent\":{\"action.auto_create_index\":false}}","querystring":"","headers":{"User-Agent":"elasticsearch-js/5.6.22 (linux 5.10.227-239.884.amzn2.x86_64-x64; Node.js v20.18.0)","Content-Type":"application/json","Content-Length":"49"},"timeout":50000},"options":{"warnings":null},"id":1},"name":"elasticsearch-js","connection":
  {"url":"https://vpc-nnn-nnn-nnnnnnn-es-vpc-zzzzzzzzzzzzzzzzzzzzzzz.us-west-2.es.amazonaws.com/",
    "id":"https://vpc-nnn-nnn-nnnnnnn-es-vpc-zzzzzzzzzzzzzzzzzzzzzzz.us-west-2.es.amazonaws.com/","headers":null,"deadCount":0,"resurrectTimeout":0,"_openRequests":0,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}},"stack":["ResponseError: Response Error"," at IncomingMessage.<anonymous> (/var/task/es-client/node_modules/@elastic/elasticsearch/lib/Transport.js:289:1)"," at IncomingMessage.emit (node:events:531:35)"," at endReadableNT (node:internal/streams/readable:1696:12)"," at processTicksAndRejections (node:internal/process/task_queues:82:21)"]}

Adding the following in /cumulus-tf/.terraform/modules/cumulus/tf-modules/cumulus/iam.tf, but it did not help.

  actions =[
    "es:ESHttp*"
  ]
  resources = ["*"]
}

The following access policy is also added to Cumulus Opensearch

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxxx:domain/uds-dev-cumulus-es-vpc/*"
    }
  ]
}