dpt-rp1-py attempts to upload (randomly selected) X files if there are > 10,428 files in a given folder.

[wart-github]

I'm sorry the title isn't too informative. This is the latest git pull. I believe this is related to issue #110. According to find . -type f | wc -w I have 12111 files in my sync folder and according to find . -type d | wc -w I have 41 directories. However, in one of my subfolders under this sync folder I have 10,428 files.

Two observations.

(1) Even though the total number of files+folders is < 13,000, when I run sync it still reports that it will be falling back to the slower method: "Looking for changes on device... Warning: Fast folder traversal did not work. Falling back to slower, recursive folder traversal."

(2) When I add one more file to the subfolder (i.e., 10,429th file), it will attempt to upload a randomly selected file again. When I add n more files to the subfolder, it will attempt to upload n randomly selected files again.

Note that (2) happens ONLY on that subfolder. Other subfolders (which do not have as many files by far) accept uploads just fine.

So there is something about a single subfolder having too many files that is causing some oddness (1) and (2). I'm happy to diagnose further, but I thought I'd put this out there first.

[janten]

The script generally has a problem listing folders with many objects (files or other folders) inside. This is mitigated by recursing into each subdirectory and listing it separately. My guess is that this does not work in your case because of the single folder with 10,428 files inside – if these files are not organised in any subfolders, there is nothing to recurse into and the slow and fast approaches will actually be the same. Could you test if the problem persists when you move some of the files into a nested subfolder?

[wart-github]

Yes, that seems correct. I created 'subfolder2' under 'subfolder' (the one with 10,428 files, and put 66 files into it. At first, it asked to re-upload one random file, but I realized this was because subfolder2 actually bumped the count in subfolder above 10,429. Once I deleted a file in subfolder (so the total of folders + files DIRECTLY in subfolder = 10,428), everything worked fine.

Once I have a free moment, I'll chop down the big subfoder into multiple subfolders, and report back, but I suspect that it'll all work fine as long as no given folder has > 10,428 files (or files+folders) in it.

[wart-github]

Can confirm that -splitting- my big subfolder into two smaller subfolders with ~5,000 files each solved this odd redundant upload problem.

It still reverts to the 'old' recursion method, even though total files+folders < 13,000 (at this point: 12,496 files and 50 folders; not sure what the threshold was when it kicked over from fast recursion to the slow one, though).