Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NullPointerException in populateRequestMetadata #3156

Closed
1 of 2 tasks
ICDI0906 opened this issue Oct 31, 2024 · 2 comments
Closed
1 of 2 tasks

NullPointerException in populateRequestMetadata #3156

ICDI0906 opened this issue Oct 31, 2024 · 2 comments
Labels
bug This issue is a bug. closed-for-staleness p3 This is a minor priority issue response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days.

Comments

@ICDI0906
Copy link

ICDI0906 commented Oct 31, 2024 •
edited by debora-ito
Loading

Upcoming End-of-Support

  • I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

i use spark engine to write s3 gateway. when each partition size is large, about 200M, it report
2024-10-31T19:22:34,559 WARN RenameOperation: While completing all active copies

org.apache.hadoop.fs.s3a.AWSClientIOException: copyFile(pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt, pressure_export_20/_temporary/0/task_202410311920521760879093253987703_0006_m_000008/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt) on pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt: com.amazonaws.AmazonClientException: Unable to complete transfer: null: Unable to complete transfer: null
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3764) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.access$1100(S3AFileSystem.java:225) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1691) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) ~[hadoop-aws-3.3.1.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
	at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196) ~[hadoop-client-api-3.3.1-ldfs4.5.9.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: null
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CopyOutcome.waitForCopy(CopyOutcome.java:72) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$19(S3AFileSystem.java:3780) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115) ~[hadoop-aws-3.3.1.jar:?]
	... 15 more
Caused by: java.lang.NullPointerException
	at com.amazonaws.services.s3.AmazonS3Client.populateRequestMetadata(AmazonS3Client.java:4490) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3800) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.initiateMultipartUpload(CopyCallable.java:250) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInParts(CopyCallable.java:168) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:130) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:132) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:43) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	... 3 more
2024-10-31T19:22:34,568 ERROR SparkHadoopMapRedUtil: Error committing the output of task: attempt_202410311920521760879093253987703_0006_m_000008_32
org.apache.hadoop.fs.s3a.AWSClientIOException: copyFile(pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt, pressure_export_20/_temporary/0/task_202410311920521760879093253987703_0006_m_000008/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt) on pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt: com.amazonaws.AmazonClientException: Unable to complete transfer: null: Unable to complete transfer: null
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3764) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.access$1100(S3AFileSystem.java:225) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1691) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) ~[hadoop-aws-3.3.1.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
	at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196) ~[hadoop-client-api-3.3.1-ldfs4.5.9.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: null
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CopyOutcome.waitForCopy(CopyOutcome.java:72) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$19(S3AFileSystem.java:3780) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115) ~[hadoop-aws-3.3.1.jar:?]
	... 15 more
Caused by: java.lang.NullPointerException
	at com.amazonaws.services.s3.AmazonS3Client.populateRequestMetadata(AmazonS3Client.java:4490) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3800) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.initiateMultipartUpload(CopyCallable.java:250) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInParts(CopyCallable.java:168) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:130) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:132) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:43) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	... 3 more
2024-10-31T19:22:34,626 ERROR Utils: Aborting task
org.apache.hadoop.fs.s3a.AWSClientIOException: copyFile(pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt, pressure_export_20/_temporary/0/task_202410311920521760879093253987703_0006_m_000008/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt) on pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt: com.amazonaws.AmazonClientException: Unable to complete transfer: null: Unable to complete transfer: null
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3764) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.access$1100(S3AFileSystem.java:225) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1691) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) ~[hadoop-aws-3.3.1.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
	at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196) ~[hadoop-client-api-3.3.1-ldfs4.5.9.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: null
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CopyOutcome.waitForCopy(CopyOutcome.java:72) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$19(S3AFileSystem.java:3780) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115) ~[hadoop-aws-3.3.1.jar:?]
	... 15 more
Caused by: java.lang.NullPointerException
	at com.amazonaws.services.s3.AmazonS3Client.populateRequestMetadata(AmazonS3Client.java:4490) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3800) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.initiateMultipartUpload(CopyCallable.java:250) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInParts(CopyCallable.java:168) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:130) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:132) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:43) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	... 3 more
2024-10-31T19:22:34,647 WARN FileOutputCommitter: Could not delete s3a://minimax-infra/pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32
2024-10-31T19:22:34,649 ERROR FileFormatWriter: Job job_202410311920521760879093253987703_0006 aborted.
2024-10-31T19:22:34,652 ERROR Executor: Exception in task 8.0 in stage 6.0 (TID 32)
org.apache.spark.SparkException: Task failed while writing rows.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:655) ~[spark-catalyst_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:358) ~[spark-sql_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:266) ~[spark-sql_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1505) ~[spark-core_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.1-ldspark-1.0-SNAPSHOT.jar:3.3.1-ldspark-1.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFile(pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt, pressure_export_20/_temporary/0/task_202410311920521760879093253987703_0006_m_000008/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt) on pressure_export_20/_temporary/0/_temporary/attempt_202410311920521760879093253987703_0006_m_000008_32/part-00008-387db23c-765c-46ca-a53b-3e5e78d5d96d-c000.txt: com.amazonaws.AmazonClientException: Unable to complete transfer: null: Unable to complete transfer: null
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3764) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.access$1100(S3AFileSystem.java:225) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1691) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) ~[hadoop-aws-3.3.1.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
	at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196) ~[hadoop-client-api-3.3.1-ldfs4.5.9.3.jar:?]
	... 3 more
Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: null
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CopyOutcome.waitForCopy(CopyOutcome.java:72) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$19(S3AFileSystem.java:3780) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3764) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem.access$1100(S3AFileSystem.java:225) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1691) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) ~[hadoop-aws-3.3.1.jar:?]
	at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) ~[hadoop-aws-3.3.1.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
	at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196) ~[hadoop-client-api-3.3.1-ldfs4.5.9.3.jar:?]
	... 3 more
Caused by: java.lang.NullPointerException
	at com.amazonaws.services.s3.AmazonS3Client.populateRequestMetadata(AmazonS3Client.java:4490) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3800) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.initiateMultipartUpload(CopyCallable.java:250) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInParts(CopyCallable.java:168) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:130) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:132) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:43) ~[aws-java-sdk-bundle-1.12.777.jar:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

write s3 success

Current Behavior

nullpointException

Reproduction Steps

use spark to write s3 gateway

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

1.12.777

JDK version used

not known

Operating System and version

not known
@ICDI0906 ICDI0906 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 31, 2024
@debora-ito
Copy link
Member

@ICDI0906 the NPE is coming from line 4490:

aws-sdk-java/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java

Lines 4486 to 4490 in df2ce8a

protected static void populateRequestMetadata(Request<?> request, ObjectMetadata metadata) {
Map<String, Object> rawMetadata = metadata.getRawMetadata();
if (rawMetadata != null) {
for (Entry<String, Object> entry : rawMetadata.entrySet()) {
request.addHeader(entry.getKey(), entry.getValue().toString());

How's the request metadata being populated? Can you provide a code sample?
Have you ruled out any issues with hadoop fs?

@debora-ito debora-ito added response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days. p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Nov 1, 2024
@github-actions GitHub Actions
Copy link

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

@github-actions github-actions bot added closing-soon This issue will close in 2 days unless further comments are made. closed-for-staleness and removed closing-soon This issue will close in 2 days unless further comments are made. labels Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. closed-for-staleness p3 This is a minor priority issue response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@debora-ito @ICDI0906