Enable zip file downloading

[yuliujpl]

Backend and frontend needs to be developed to enable zip file downloading.

[nutjob4life]

@dcrichto1 @hoodriverheather @yuliujpl here are some ideas on how this might work

1. User visits the UI and selects either a set of files, a set of datasets, or a collection for ZIP file downloading
2. The UI at this point could put in some limits using knowledge it might have as to the size
3. Assuming the user's request passes muster at this point, the UI makes an API call to the backend passing
   • The files to put into the archive—this would likely be a Solr query
     • The idea is to find all files that match the query
   • The email address to notify once the ZIP archive is ready
   • This API endpoint is protected and accepts the JWT established in the user's session
4. The backend assigns a UUID to this request and returns it to the UI and immediately and begins a background task
   1. The task uses the Solr query and creates a directory tree of matching files
   2. It ZIPs the archive
   3. It generates a URL to the archive
   4. It starts a timer task to delete the file after a certail number of days (7 days?)
   5. It emails the user a message with the URL to the archive
5. The user opens their email in time and begins downloading the file
   • As the archive can be large, the user can retry the download any time (within 7 days?)

At any point in step 4, an error could occur

• The Solr query matches no files; this could be returned synchronously to the UI
• There's not enough disk space to create the archive; this would be communicated via email to the user
• There's enough space, but the generated archive is too large to download (say 1GB?); this would be communicated via email to the user
  • The archive is immediately deleted
• Generating the archive takes too much time (more than a day?); it's aborted and communicated via email to the user
  • The partial archive is immediately deleted

Using the UUID, the UI can ask about the status of the ZIP file assembly:

• Is it in progress? Can we tell how far it's gotten?
• Is it complete? If so, can we get the URL in case the email was lost?
  • Is it complete but it was never downloaded, and so the file was deleted?
• Did one of the error conditions occur?

@dcrichto1 @hoodriverheather @yuliujpl what do you think? What should some of these thresholds (space and time) be?

[nutjob4life]

@yuliujpl @dcrichto1 @hoodriverheather, reminder:

1. Please check the approach above ↑
2. Leave feedback below ↓

No feedback means we're good and I'll start implementing the server side of this

[hoodriverheather]

@nutjob4life This looks like a good plan to me. Want me to add to our staff agenda for tomorrow?

[nutjob4life]

@nutjob4life This looks like a good plan to me. Want me to add to our staff agenda for tomorrow?

@hoodriverheather sure!

[nutjob4life]

@yuliujpl mentioned in Teams chat:

It should probably run on a separate server so it doesn't compete for resources on the labcas-backend server

Probably; but it'll also need access to the /labcas-data disk so it can gather the files and have enough scratch space to create ZIP archives