Issue with writing Avro files directly to S3 bucket

[clarence-coppinger]

When using apache Avro library to write Avro files directly to s3 bucket using the AmazonS3#putObject(String bucketName, String key, InputStream input, ObjectMetadata metadata) method.

Upon trying to read the files from the bucket using Avro tools (command line tool to read Avro files) we see this exception:

“org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -25669624”

When we use a similar process to write files to GCS the files are able to be processed after being uploaded without issue. My question here is: Are we using the wrong method to upload these files? When we write the Avro files locally and upload them we can read them afterwards but trying to write them directly to the bucket seems to cause an issue.

[debora-ito]

Hey @clarence-coppinger we recommend you reach out to the Avro team with this issue. We can't help troubleshooting issues with third-party libraries.

[clarence-coppinger]

Just to clarify this isn't an issue with the avro library. I referenced the putObject method from the AWS SDK and was having an issue with avro files that were uploaded through the AWS S3 API.