Sub-directory support

[MeijeSibbel]

This occurs when i create a sub-directory in FileZilla, add some files to it and then try to sync it with goobox:

Checking if local Goobox folder exists... yes
Checking if cloud Goobox bucket exists... yes
Checking Storj Bridge for changes...
Downloading file Climate change Iberia/On the key role of droughts in Mediterranean Europe.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\On the key role of droughts in Mediterranean Europe.pdf: No such file or directory
Downloading file Climate change Iberia/Rainfall in Iberian transnational basins, a drier future.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Rainfall in Iberian transnational basins, a drier future.pdf: No such file or directory
Downloading file Climate change Iberia/Climatology of the Iberia coastal low-level wind.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Climatology of the Iberia coastal low-level wind.pdf: No such file or directory
Deleting local file music... done
Checking Storj Bridge for changes...
Downloading file Climate change Iberia/On the key role of droughts in Mediterranean Europe.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\On the key role of droughts in Mediterranean Europe.pdf: No such file or directory
Downloading file Climate change Iberia/Rainfall in Iberian transnational basins, a drier future.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Rainfall in Iberian transnational basins, a drier future.pdf: No such file or directory
Downloading file Climate change Iberia/Climatology of the Iberia coastal low-level wind.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Climatology of the Iberia coastal low-level wind.pdf: No such file or directory
Checking Storj Bridge for changes...
Downloading file Climate change Iberia/On the key role of droughts in Mediterranean Europe.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\On the key role of droughts in Mediterranean Europe.pdf: No such file or directory
Downloading file Climate change Iberia/Rainfall in Iberian transnational basins, a drier future.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Rainfall in Iberian transnational basins, a drier future.pdf: No such file or directory
Downloading file Climate change Iberia/Climatology of the Iberia coastal low-level wind.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Climatology of the Iberia coastal low-level wind.pdf: No such file or directory
Checking Storj Bridge for changes...
Downloading file Climate change Iberia/On the key role of droughts in Mediterranean Europe.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\On the key role of droughts in Mediterranean Europe.pdf: No such file or directory
Downloading file Climate change Iberia/Rainfall in Iberian transnational basins, a drier future.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Rainfall in Iberian transnational basins, a drier future.pdf: No such file or directory
Downloading file Climate change Iberia/Climatology of the Iberia coastal low-level wind.pdf...
  Unable to open C:\Users\meije\Goobox\Climate change Iberia\Climatology of the Iberia coastal low-level wind.pdf: No such file or directory
Checking Storj Bridge for changes...
Request failed with status code: 0

[MeijeSibbel]

Libstorj output:

C:\Users\meije\Downloads\libstorj-2.0.0-beta-win64\libstorj-2.0.0-beta\bin>storj list-buckets
Unlock passphrase:                                                                                                      ID: 

ID: 8126211066cbe3545737fa40    Decrypted: true         Created: 2017-10-28T15:21:14.388Z       Name: Goobox

C:\Users\meije\Downloads\libstorj-2.0.0-beta-win64\libstorj-2.0.0-beta\bin>storj list-files 8126211066cbe3545737fa40
Unlock passphrase:

ID: 61239352572b390e2f839523    Size: 4024863 bytes     Decrypted: true         Type: application/octet-stream  Created: 2017-10-30T21:43:27.658Z       Name: Climate change Iberia/On the key role of droughts in Mediterranean Europe.pdf
ID: 1c922c17eb247f9ccabef77e    Size: 1794630 bytes     Decrypted: true         Type: application/octet-stream  Created: 2017-10-30T21:43:27.737Z       Name: Climate change Iberia/Rainfall in Iberian transnational basins, a drier future.pdf
ID: 2f21ae4e07401ca1bed0a065    Size: 6046679 bytes     Decrypted: true         Type: application/octet-stream  Created: 2017-10-30T21:43:29.685Z       Name: Climate change Iberia/Climatology of the Iberia coastal low-level wind.pdf

C:\Users\meije\Downloads\libstorj-2.0.0-beta-win64\libstorj-2.0.0-beta\bin>

It looks like the way Libstorj handles folders (with /) is incompatible in goobox.

[kaloyan-raev]

This issue exists because currently there is no sub-directory support in Goobox. Let's use this issue to track the implementation of such support.

I prefer to work on this topic after implementing the two-sync without any sub-directory support. The two-way sync is a complex topic and it would be better first to have a stable implementation of the simpler use caset.

[MeijeSibbel]

I've given a bit of thought to this. So as i see it Storj doesn't support a file structure as of yet. FileZilla just adds the folder and sub-folder names in the name of the file. The issue is what if i take a file from Goobox and drag it into a another sub-folder in my sync folder, how are we going to make this change clear to other devices on the network that run the same sync folder?. Storj doesn't allow renaming of files, so if we would want to make this change visible to other machines running the same sync folder we would have to delete the file and upload it again with the new name which is a huge waste of bandwidth. As i see it the only way to make this directory structure visible across different devices that run the same Goobox sync folder is to not only sync the files across multiple devices, but also sync the Goobox database across multiple devices. So say i change a file on machine (A), i drag a file into a another folder within the sync folder. The database changes as the paths of the files are now different. This triggers a sync event that uploads the database with the new paths to the storj bucket. The clients on all machines closely monitor if this database file is changed or not. If it is, they (e.g. machine B) download the file and check for the differences in file structure and file naming (because some users might just rename the file). The files are then adjusted accordingly to what's in the sync folder.

[kaloyan-raev]

Implementing the sub-directory support like in FileZilla is enough of a workaround. Yes, moving and renaming files and folder would involve wasteful file transfer.

The idea of synchronizing meta information between the devices is interesting, but it is actually a very serious and complex workaround. Perhaps, we may do it if no other option is available.

Until then I recommend to push the discussion here: storj-archived/sips#19

[kaloyan-raev]

BTW, if the rename feature is implemented in the Bridge in a way that the file id does not change, it should be possible to avoid the superfluous file transfer without synchronizing the database between devices. We already keep track of the file id in the sync db. If the file name changes, but the id stays the same, it should be possible for the device to detect this by comparing the current state of the bridge with the one in the sync db. In such case, the sync app will just move and rename file and folders without doing any file transfer.

[MeijeSibbel]

Ok, lets do the filezilla way for now. Is there a bridge feature request issue open for this?., if not please open one.

[MeijeSibbel]

@jkawamoto, would implementig file renaming on the bridge side be solved with a simple PR?

[kaloyan-raev]

Libstorj: storj-archived/libstorj#398
Bridge: storj-archived/bridge#488

[jkawamoto]

I proposed a simple solution but it was rejected because they have their plan. Currently, a file ID is a hash of the bucket name + the file name. To support renaming without changing file IDs, another database design is needed. I think that it is under discussion.

[kaloyan-raev]

Some additional observations on how sub-directory support is implemented in FileZilla.

If a new empty directory is created, a new file is uploaded to Storj with:

• file name: <dir-name>/
• file size: 1 byte
• file content: '/`
• MIME type: text/html

This file entry is only useful for representing empty directories.

If the above file entry is deleted, but there is another file entry with name <dir-name>/<file-name> then the <dir-name> directory will still be visible and a <file-name> file will be displayed inside it.