not authorized to perform: execute-api:Invoke error in EDC environment after update to AWS::APIGateway::RestAPI resource

[asjohnston-asf]

$ curl https://hyp3-test-api.asf.alaska.edu
{"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-west-2:********2543:vd2gh6uqw3/api/GET/"}

Our REST API includes an IAM policy granting resources in the same VPC access to invoke the API, implemented at https://github.com/ASFHyP3/hyp3/blob/develop/apps/api/api-cf.yml.j2#L70 . It turns out this policy is insufficient for granting access to however Earthdata Cloud exposes our API to the internet.

Earthdata Cloud quickly applies an alternate policy whenever updates are made to the API, that grants a wider range of permissions to a wider range of VPCs and CIDR blocks. This policy is sufficient to expose the API to the internet.

Such policy changes only take effect when the REST API is deployed. We automatically create a new deployment each time we deploy the CloudFormation stack. Earthdata Cloud does not re-deploy the API when they remediate the policy.

Usually this doesn't matter, because we almost never change the AWS:APIGateway::RestAPI resource, so the EDC policy stays live. If we every update the RestAPI, though, the policy reverts to the insufficient ASF policy and is immediately published. The policy is quickly reverted, but only after the deployment. So users invoking the API get this error until the API is manually re-deployed with the EDC policy.

Waiting a few minutes and re-running the Github action has been sufficient to get the API working again.

Long term, we could edit the policy in our cloudformation template to grant the additional permissions needed to expose it to the internet.