Backups to Cloudflare R2 (S3 Compatible) storage fail

[andrewheberle]

I'm backing up to S3 compatible storage (Cloudflare R2) from Cloud Native PG (CNPG) which uses barman-cloud-backup.

Backups are failing with an error of:

ERROR: Upload error: An error occurred (InvalidPart) when calling the CompleteMultipartUpload operation: All non-trailing parts must have the same length. (worker 1)

The above error is surfaced from barman-cloud-backup when attempting to complete the weekly full backup.

$ barman-cloud-backup --version
barman-cloud-backup 3.10.1

I've come across what seems like a similar issue when using Cloudflare R2 as a backend for a Docker registry that was resolved by setting a fixed chunk size for multipart uploads, however I don't know enough about the barman codebase to know if this is the same issue.

https://community.cloudflare.com/t/all-non-trailing-parts-must-have-the-same-length/552190/7

And this discussion here:

distribution/distribution#3940 (comment)

[andrewheberle]

Just an update here, I had "snappy" compression enabled for my base backups however once I changed the settings to use no compression the backups have succeeded so far.

Other backups of very small databases seem OK (ie < 100MB) but this one uncompressed is about 600MB

[maxpain]

the same problem on R2

[martinmarques]

Could this be related to this issue: #957

We support S3-compatible object stores, and though providers outside AWS claim 100% compatibility, as you can see in the case of Linode, it's not always fully compatible.

It would be good to check this for anything Cloudflare doesn't yet support.