Supporting Background Uploads

[nicolelehrer]

Hi all, thanks for the great client - it was super easy to get up and running fast.

I saw that TUSKit provides a means for asking iOS for more time to finish its upload task before the app moves to the background, by using the method described in "Executing Finite-Length Tasks" of this Apple doc.

Are there any plans to support true background uploads? i.e. the upload continuing despite the user leaving the app for a while. It seems there are some key limitations with Apple API in attempting to do so:

• we can't ask a background upload to start from a particular offset in a file, so it will always need to start from the file's beginning (meaning you would have to break the file into chunks if you wanted to start reading bytes from the middle of the original file)
• the system daemon controlling the background upload task can retry the upload on our behalf, hitting the tus server with another upload request at offset 0, when in fact the tus server has already received some of this file's data, causing a mismatch that rejects our request

I've seen the discussions on how to work around these limitations here:

• Issue: Allow a PATCH request to restart at an offset of 0 It seems while this would require the least amount of work client side, this proposed modification to the tus protocol was not adopted since there were no other apparent use cases aside from working around Apple's limitations.
• Pull request: Add Creation With Upload extension It seems this approach was favored as a workaround since it solved other use cases. The code implementing this idea looks like it was merged into tusd but the extension definition was not merged into the formal protocol just yet.

Does the above seem to capture the latest efforts in supporting background uploads on iOS using tus?

Are there plans for TUSKit to offer more backgrounding support than is currently provided?

Thank you in advance!

[cjhenck]

IIRC, with the extension TUSKit could be modified to chunk the files, upload each, and concatenate them if all the uploads we're successful.

I had tested this approach before the extension was added and it's extremely clunky. Are you having problems with the current background uploading?

[nicolelehrer]

Thanks for the info, @cjhenck. It did sound potentially clunky, and I mostly wanted to checkin on if there had been any updates since those discussions had last taken place. Btw, no technical problems running the current implementation, just to make sure that's clear too.

[cjhenck]

I'm afraid I can't answer on the progress front unfortunately, since I haven't been using TUSKit recently. Glad the current version is working, though!

[Acconut]

IIRC, with the extension TUSKit could be modified to chunk the files, upload each, and concatenate them if all the uploads we're successful.

As @cjhenck said, this is the recommended approach from the protocol's point of view. However, I am not sure how far this has been implemented in TUSKit, but @MMasterson should know the details.

[MMasterson]

hey @nicolelehrer an update for these types of functions is being worked on currently.

@cjhenck do you have examples of your methods?

[nicolelehrer]

Thanks for the update @MMasterson. Do you mean that you'll add api for file chunking, uploading each chunk using uploadTaskWithRequest:fromFile: via the creation with upload extension, and then making a concatenation request? Do you possibly have an estimate on about when those changes would make it into the project? Thanks!

[cmrocha]

Thank you for tusKit! Any update on true background uploads?

[MMasterson]

@cmrocha no big changes have been made. There are a few suggestions in this thread and others attached to accomplish it.

I can find some time in the next few weeks to add some support. Or if you'd like, we accept PRs :)

[cmrocha]

Thanks so much for getting back to me! If I come up with a decent solution I will absolutely let everyone know.

[cjhenck]

Just seeing this again - these are the problems I remember, but I may be wrong:

1. You don't have control over the retry. So the OS might upload the same chunk an indeterminate number of times. The protocol needs to support this (it may now, with proper extensions). Also be aware of the data use considerations.
2. I don't think the callbacks are guaranteed, so you need to somehow initiate the merge after all the chunks are completed. (Again, maybe part of the protocol now).
3. Your files will take up extra space on the system, and won't be deleted. You can't use /tmp: https://stackoverflow.com/questions/37309947/will-temporary-files-be-saved-by-an-nsurlsessionuploadtask/51529762#51529762

Hopefully those are helpful pointers.

[MMasterson]

better support being added in swift-development check #65

We can't fix the overall iOS issue, but do our best to counteract it