Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add authentication option and some support to path-like s3 uri-s for region specification #32

Open
wants to merge 1 commit into
base: OpenDAL
Choose a base branch
from

Conversation

kgabor
Copy link
Contributor

@kgabor kgabor commented May 22, 2024

Hi Stephan,

please find a draft solution for having both the authentication and the region passing through N5Factory.

  • I added a cmd line option to explicitly request the authentication from the backend (at the moment s3 only). This is necessary for the output bucket writing, because the credentials are based on the assumed role in the EMR-serverless session.

  • Unless the path-like s3 uri (https://s3-us-west-2.amazonaws.com/bucket/...) is used, N5Factory falls back to us-east-1 which does not work for us. Updated ParsedBucket to tolerate path-like uri-s. Maybe we don't really want to use path-like URI-s in the long term but using N5Factory.s3Region to set region obtained from the DefaultAwsRegionProviderChain. See related issue here.

  • Also note that while this branch works with n5-aws-s3 4.1.1, it fails with key error in 4.1.3 :

mpicbg.spim.data.SpimDataIOException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: B6CKA6GRDWMBAXWE; S3 Extended Request ID: KUW8//nsXQhv7NIQlZ1pCe801MBlpYnnrwsCtV01VPgy6tCjR+PryJUtmzxJ/3xAA114oWUI4m4=; Proxy: null), S3 Extended Request ID: KUW8//nsXQhv7NIQlZ1pCe801MBlpYnnrwsCtV01VPgy6tCjR+PryJUtmzxJ/3xAA114oWUI4m4=
	at net.preibisch.bigstitcher.spark.util.Spark.getSparkJobSpimData2(Spark.java:298)
	at net.preibisch.bigstitcher.spark.abstractcmdline.AbstractBasic.loadSpimData2(AbstractBasic.java:30)
	at net.preibisch.bigstitcher.spark.SparkAffineFusion.call(SparkAffineFusion.java:130)

@kgabor
Copy link
Contributor Author

kgabor commented May 22, 2024

p.s. Please discard my changes to TestCloudFunctions.java.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant