concurrent multipart support for transfer

[giacomo-alzetta-aiven]

About this change - What it does

Alternative version of #128

Why this way

Simplifies API (no resume) and avoids an extra class.

[codecov-commenter]

Codecov Report

Patch coverage: 81.14% and project coverage change: +1.08 🎉

Comparison is base (5f65d18) 65.25% compared to head (33ed123) 66.33%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
+ Coverage   65.25%   66.33%   +1.08%     
==========================================
  Files          31       31              
  Lines        3606     3844     +238     
==========================================
+ Hits         2353     2550     +197     
- Misses       1253     1294      +41     

Impacted Files	Coverage Δ
rohmu/object_storage/base.py	76.31% <77.27%> (+0.12%)	⬆️
rohmu/object_storage/local.py	86.13% <77.94%> (-3.41%)	⬇️
rohmu/object_storage/s3.py	66.32% <78.68%> (+2.19%)	⬆️
rohmu/util.py	73.68% <85.71%> (+24.84%)	⬆️
rohmu/object_storage/azure.py	36.28% <100.00%> (ø)
rohmu/object_storage/google.py	40.71% <100.00%> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[sjamgade]

would insist on trying to reuse as much as possible.
Also the main problem with this approach is the *transfer classes get bloated as we add more code to them, it would be nice if there some Protocol could be expected from *transfer classes to better support concurrent uploads

[giacomo-alzetta-aiven]

I did some test trying to upload files using botocore and see the heap size. From what I can tell botocore is using a very small buffer, the RAM is basically constant during upload (I have uploaded a chunk of 2GB, and during the whole duration RAM use was constant).

I discovered that botocore has some heuristics in upload_part. I believe it tries to detect if the input stream is something like an http body or a file. The ProgressStream is treated as a file and thus it calls seek to move around and compute the md5 hash and size, so I had to add seek support for that. From the tests I've done it seems to work fine although it is a bit fragile.

Addressed comments.

Addressed problem

[ilpon]

I'm not authorized to merge this.