Permission errors when accessing output files in S3

[asjohnston-asf]

Jira: https://asfdaac.atlassian.net/browse/TOOL-2040

Completed HyP3 jobs include an AWS S3 Bucket and Key for their output files, but attempting to access those files using the AWS CLI or the boto3 python library results in AccessDenied or Forbidden errors. For example:

>>> import hyp3_sdk
>>> hyp3 = hyp3_sdk.hyp3()
>>> job = hyp3.get_job_by_id('62db0f98-7c7e-42bf-ad2e-30840abad4ed')
>>> job.files[0]['s3']
{'bucket': 'hyp3-edc-prod-contentbucket-1fv14ed36ifj6', 'key': '62db0f98-7c7e-42bf-ad2e-30840abad4ed/S1AA_20150504T120217_20150621T120220_VVP048_INT80_G_ueF_ABE0.zip'}

>>> import boto3
>>> s3 = boto3.client('s3')
>>> s3.download_file(job.files[0]['s3']['bucket'], job.files[0]['s3']['key'], 'S1AA_20150504T120217_20150621T120220_VVP048_INT80_G_ueF_ABE0.zip')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/boto3/s3/inject.py", line 190, in download_file
    return transfer.download_file(
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/boto3/s3/transfer.py", line 326, in download_file
    future.result()
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Accessing these files directly in S3 was possible until Jun 2022, when the https://hyp3-api.asf.alaska.edu/ API was migrated to a NASA-managed AWS platform where security controls prevent direct public access to files in S3. The bucket and key are still included in the description of each output file, but attempts to use boto3 or the AWS CLI now fail with permissions errors.

[jhkennedy]

To add some context, for https://hyp3-api.asf.alaska.edu, we hope to enable S3 access again and we're looking into options which is why* we haven't removed the s3 info for the files. We'll either need a policy change on the NASA side or to stand up an application to grant S3 credentials.

That said, it's been 10 months already, and it's unlikely to change in the foreseeable future.

---

*We also have "custom" hyp3 deployments outside of NASA's managed AWS environment that do have fully public buckets and can utilize the s3 keys.

[markbroich]

Hi ASF team, I hope you are well. I was wondering if you got to addressing this issue pls? Thank you for considering my questions. Greetings from hot and humid NC. Regards, Mark

[markbroich]

pls also respond to [email protected] thank you, Mark

[markbroich]

Hi ASF team, I am now using the urls for download to s3. Am running parallel lambdas and it works but I think copying keys from s3 to s3 would be faster as I am in US-west2. Pls let me know when you get allowing copying of S3 keys again so I can modify my code. Tnx, Mark

[jhkennedy]

@markbroich getting direct s3 access is a significant effort and we do not currently have a timeline for doing so. We'll update this issue if and when we provide it.