Directory support on mcu side ?

[pidou46]

The directory support on the side of the mcu would help to keep tidy file system on average at the big project.

Perhaps the "synchronization" feature suggested by jdeltoft will suffice if it processes directories.

uPyLoader gives me the most efficient workflow with micropython on esp8266, I've tried so far, please keep good the work going.

[BetaRavener]

Hi.
I agree that directory support on MCU would be nice feature, but it's non-trivial to implement. The problem is that there is no easy way to get full contents of filesystem. For now the files are listed by running os.listdir() and parsing response. Only way would be recursively calling this on every folder. There could be script that would gather whole tree structure and send it to PC, using some protocol.

Another issue is that when I started uPyLoader, there was no support for MCU folders (as far as I can remember). In current design, I can't imagine an easy way how user could specify destination MCU folder for transferred file. Suggestions are welcome, but it seems like lot of changes would be needed.

I'm not saying it can't be done, just that it's lot of work and my time is limited now. If more people would request this feature, it might go up my priorities list. And if someone decided to implement this, I'd be more than happy to review pull request and ultimately add it to new release.

[profra]

+1 for directory structure ...
... and try to be compatible with CP from Adafruit... it would be awesome!!!
... also new SW for MP is uPyCraft from DFRobot, see https://www.gitbook.com/book/dfrobot/upycraft/details ... no experince from my side yet

[BetaRavener]

@profra Could you explain what you meant by "and try to be compatible with CP from Adafruit".

[BetaRavener]

So, after revisiting current design, I have came up with an idea how to make this work. Below is image describing necessary changes.

The "Set as destination" will be enabled only for directories. The concept also includes support for #16. Any selected directories could be copied recursively (might be an option in settings to turn off), though it will probably take a while. Difficult will be whole MCU directory listing, which may require custom script on MCU (simillar to transfer scripts) as plain REPL communication over serial tends to be unreliable for larger quantities of data. Hopefully WiFi connections (specifically WebREPL) are fine for long communication, because such script is not possible there.

[BetaRavener]

After some considerations, I think the proposed concept will be confusing. For example, I forgot that user can choose multiple files (and folders), possibly marking whole sub-trees in filesystem. As a result, it would be different when user selected and copied 2 files (which would appear in destination folder), or selected 2 files and the folder Dir containing them (which would either copy these 2 files, and place them in Dir inside destination folder, or even copy all files from Dir - because it was selected).

I find the above really confusing, so I would like to ask for your ( @pidou46, @profra , @jdeltoft, and everyone else) ideas how the UI should look and behave like. If nothing comes up, I propose using same interface as Total Commander. There wouldn't be trees, just flat folder structure and user could go into (and out of) directories. The transfer would always happen between current folder on Local and Remote side.

I would like your ideas before 25.11., after which I begin implementing the best solution. Thanks :)

[pidou46]

Hi BetaRavener,

I'm perfectly ok with "Total Commander" behaviour provided that it's possible to select the root directory on the local and remote side to be able to sync the entire file system of a given mcu to/from a local directory.

Here is the file structure on both side (not the GUI):

local file system remote (MCU)
my_project_directory .. <-- Root
board1 <-- Root of MCU lib
lib mylib.py
mylib.py main.py
main.py boot.py
boot.py

It would be great to have the MCU unique ID displayed : machine.unique_id(), to avoid errors.

I hope it could help...

[jdeltoft]

Hi BetaRavener/all,
I agree with pidou46. I don't see folder support on the nodemcu all that important since it's a micro controller. I personally have a project with 10 py files that I compile and move over, and don't have any issue keeping track. Directories just seem a bit of overkill and not worth the complexity.

My earlier ask for "syncronization" feature was more around supporting a way to know what files have changed since I've last updated my nodemcu, and having uPyLoader highlight or auto detect those to auto transfer (after compile if applicable). What I find is that I may have made changes to various files since the last update and then it's hard to be 100% sure what I changed so I often times end up uploading all 10. Nobody wants to debug a setup where they just forgot to update the version of a given py file in the nodemcu!!! Timestamps, or md5sum hashes of the files to compare mcu side to local side? I think spending effort on this would be way more valuable than directory support.

[BetaRavener]

I agree that this is lower priority than sync support. However, I can also imagine when dir support becomes handy. For example, working on multiple projects, where name conflicts may occur, it is good to keep each project in its own dir. As these MCUs become ever more powerful (ESP32), it's nice to be able to take advantage of it.

The reason I took on this before sync support is that once you get some 1:1 mapping between folders, it's easy to synchronize them (from UI perspective). Without doing this, I would have to create separate mechanism for specifying what to sync.

Ad @pidou46 I got a bit confused by your file structure, but basically all files (even in subfolders) contained in current folder would be compared to other side.

[pidou46]

Sorry, I haven't check my comment after publishing and I haven't seen that layout have gone.

I'm agree that we don't need synchro of individual file/directory, nevertheless it would be nice to be able to synchronize a whole file structure including directories.

Said differently, no selection but synchro of the complete structure (including directories if possible)

It's (only) my point off view.

[profra]

My opinions have been published in the other issue... see comments #30 (comment) #30 (comment)

[BetaRavener]

Still in (slow) progress, but the GUI is almost ready and listing files on MCU works. I couldn't find an easy way to include parrent entry ([..]) in the list without subclassing a lot of Qt classes, so buttons will have to do :) And yes, I know that it's not prettiest one, so if we have designer volunteering to make changes, I'd be more than happy.

[profra]

I see you are working on the dev-transfer branch but I cannot get the list of files on Remote (esp8266) ... the function 'Init transfer files' is OK but 'Could not list files'... hence I cannot test anything... what is the difference between 'Project Folder' and 'Parrent Folder'?

[BetaRavener]

I haven't yet pushed the script for listing files, because I'm not sure if I want another script sitting on the MCU. The other approach would be defining necessary functions when connecting (when connected, send bunch of code..). But this have disadvantages too - possible errors during transmission and longer connection time. I was planning to decide when all functions are done (list, remove, mktree) and when I see how much code there is.

Without the script, it can't list the files. It is planned to fallback to basic MCU listing in such case, but not yet implemented.

And the buttons - Project Folder takes you to folder you have selected when File->Navigate. Parrent Folder is just one folder up from the current one.

[BetaRavener]

Here is the necessary file if you are interested. You will have to save it as __list.py to MCU. I'm also coding fallback mode right now.

[BetaRavener]

Just to imagine, this is how the MCU looks like with __list.py. Transferring works when only files are selected.

[profra]

So I'm back home and on PC... I discovered after having a look at the source code that I have to name the file '_list_fs.py' instead '__list.py'... only after that I saw what I should see... 😄

One note... As we start reading a book from the beginning, as we start drawing a timeline from left to right, as we write text from left to right, as we look at the tree directory structure... so IMHO is more logical to swap these two folder icons... on the left the Project Folder icon and on the right the Parrent Folder... because our Project Folder is our beginning, our start=top level of our Project.
Let me know if you need anything to test, I'm always available. Good luck!

[BetaRavener]

Sorry, I have been doing some more changes in meantime and the script changed :) The file should still be __list.py, but you'd also need updated contents. You probably hacked around by naming it _list_fs.py, because _list_fs is actually a new name for the function. Anyway, good for you 👍

Sure, swapping those buttons probably makes sense and it's easy to do. Probably in next commit if I'll remember.

[DanielIbaseta]

I'm also having issues not being able to select a folder destination into de MCU, please bring this issue back up. This is one of the best management apps I've seen for generic MCUs and it is a pity that it cant be used properly because of that.

[zindy]

Just an idea and please do let me know if I'm completely off the mark.

I think os.stat(path) could be used to detect whether a path is a folder or a directory:

>>> import os
>>> os.stat("main.py")
(32768, 0, 0, 0, 0, 0, 198, 814, 814, 814)
>>> os.stat("lib")
(16384, 0, 0, 0, 0, 0, 0, 18, 18, 18)

Then in connection.py, replace the self.send_line() string with:

        if success:
            self.send_line("import os; [chr(35+(os.stat(fn)[0] == 32768))+fn for fn in os.listdir()]")
            # Wait for reply
            try:
                ret = self.read_to_next_prompt()
            except TimeoutError:
                success = False

So this prepends a '#' for a directory and a '$' for file. Doesn't matter what these are, because the real filename is always fn[1:], but now we have a way to differentiate files from folders (fn[0] == '#').

And the actual list_files() method could be changed to accept a folder to check, be it "/" or one of the subfolders:

    def list_files(self,mcu_folder="/"):
...
        if success:
            self.send_line("import os; [chr(35+(os.stat(fn)[0] == 32768))+fn for fn in os.listdir(\""+mcu_folder+"\")]")
...

Then "somehow" add files or directory icons for each rwo in the MCU panel widget. Looking at the screenshots in this thread, the code may already be there. And when you click on a row of type "directory", then change the path sent to list_files()... Or something along these lines :-)

Cheers,
Egor

[BetaRavener]

@zindy Hi Egor. This is already done in this branch of uPyLoader. There's already working protocol that can list all files. The idea was to define a function _fs_list when connecting to device and then use it instead of os.list_dir. Actually, it's already working and I could get recursive list of all files but I had to sack it because I didn't have enough time and there were more pressing issues. I think that the branch is missing only some GUI things and would be ready to be used. But I have only so much free time and this is not currently my priority. If anyone would attempt to finish it, I'd be glad to give more info if needed but I doubt it since there were not many code contributors in the past.

[zindy]

@BetaRavener Hi Ivan, thank you for coming back to me! Don't worry, I perfectly understand you may have other commitments. If this amounts to anything I'll send you a pull request :-)

I had a quick look at your other Branch this afternoon and it seems that a lot of code has been committed to master since, and it's become quite hard to follow the changes directly related to the directory support.

So I went ahead and forked uPyLoader and added my additions to send_line to retrieve whether a path on the mcu is a folder or a file, this without the need for additional files stored on the microcontroller.

Then as a very quick and dirty test, I changed the right panel to a QStandardItemModel like you did and re-used your icons and (modified) remote_file_system_model.py. Although I didn't re-use your remote_file_system_model.py and (for now) have confined all the changes to main_window.py -- But yes, your code is way cleaner and I'm sure I'll regret not pursuing the remote_file_system_model route soon enough...

This is where I'm at: Display works, clicking on folders works, editing files works (load and save. root and subfolders), removing files work, executing files on the MCU should work. Anyway, here's a screenshot:

Cheers,
Egor

[BetaRavener]

Nice, I'm actually surprised that you accepted the challenge. It's true that the branch is ancient and lot of things changed since. Nice work, I'll try to run your fork in near future. Can you also go inside the folder on the MCU side? The idea was to change both panes to something akin total commander (now the left pane is tree structure. It would get replaced with simple list and after double-clicking folder you would enter it). This change would help to make file transfer more intuitive - you could enter a folder on MCU side and files would get transferred into that folder. That's why we also needed "root folder" and "one level up" buttons in the branch. And also both sides would behave the same after the change, only difference is the model that is pulling the data.

[zindy]

No worries! Yes, navigation works as expected by double clicking on any folder and using the ".." to go back one level. You can edit files in the current folder on the MCU side, and files transferred to the MCU are also transferred to the current folder on the right hand side:

I remember midnight commander from way back when, I can see the point of it, but it was never my thing, sorry! Also, remember that I don't query the actual FS tree for the MCU, just the files within a single directory at a time. But yes, fixing the icons so that they look the same on both side would be nice of course! Size and timestamps could also be super useful. I'm sure there's still a lot of code to loot for the dev-transfer branch!

For now, I will try to actually use uPyLoader, and implement functions I think are important for my own workflows. I could definitely use a function to copy folders to the MCU, without copying ".git*" and other easy to avoid files: *.bak, *~, and possibly copy only compiled files when both the original and the byte-compiled file are present in the source directory. Good thing you already programmed a context menu when clicking items on the left ;-)

Anyway. I'll use my branch for a few days and if everything works as expected, I'll send you a pull request. Not a complete port of dev-transfer to master, but let's call it an initial attempt at MCU navigation.

Take care mate!

[BetaRavener]

Ok cool. I didn't mean to change MCU side to tree but the opposite - change PC side to behave like MCU. I used extra buttons for navigation before because I used vanilla model for PC side and didn't want to mess around adding .. item to it as it seemed rather complicated at the time (sorting rules and stuff). But it sure is doable and would result in cleaner interface.

Actually querying only current directory is fine - in certain cases it's better (many files in different folders on the device) although opening directory will have some delay. But I think this is nothing to worry about :) Regarding timestamps - this depends on what micropython FS supports. You could at least get file size with os.stat but I didn't work on this.

Anyway, great work and I'm looking forward to that pull request. I really appreciate the effort.

[zindy]

Hi Ivan,

I didn't mean to change MCU side to tree but the opposite - change PC side to behave like MCU. I used extra buttons for navigation before because I used vanilla model for PC side and didn't want to mess around adding .. item to it as it seemed rather complicated at the time (sorting rules and stuff). But it sure is doable and would result in cleaner interface.

This is work in progress, but definitely worth a screenshot! I followed your advice (changing the local side to behave like the MCU), and implemented .. and context menus for both sides.

The new code still needs "a bit" of refactoring, but the interface does work as it's supposed to. I just need to reparent a few things like the context menus, add key bindings, then "later", I will add a column for file sizes (but yes, you're right, timestamps on the MCU side is was a bit too optimistic).

That's it for now! A bit of clean-up and I'll push my changes later tonight :-)

Cheers,
Egor

[zindy]

OK, one more for the road, sorry about the flood.

This is where Windows 10's quarter dock can really come into play (editor | terminal / mainwindow):

Compile + Auto transfer works perfectly, and I modified the "Execute" button to simply import the selected module. This works especially well if a main function called at the end... But remember to reset the mcu, simply re-uploading a module and importing it won't do anything if it was already imported once before. So basically, the workflow becomes Compile -> Upload -> Reset -> Import.

At this point, the GUI could be re-arranged a bit, but I think I have enough functionality to try and do some actual work! If I find any regressions I will of course update my fork (until I send a PR).

Cheers for now!

[BetaRavener]

I'll definitely take a look at it tomorrow evening! And you've probably discovered one of the traps of working on project like this - it's important not to forget that it's just tool for creating something else. Go do the actual work :)

Anyway, it's nice to see the progress. Cheers.

[laukejas]

@BetaRavener Has there been any progress on this? I downloaded the latest source, but it doesn't have this feature, though I saw it in some of the screenshots in this issue page. Is it coming?

[BetaRavener]

@laukejas Hello. yes it's been a while, the feature is in separate branch that was kind of put on pause as it was deemed too much effort for the time I had on hands. I'm not even sure if it could be resumed or it's too obsolete. But overall, I personally came to conclusion that MCUs are such a simple devices that one shouldn't really need folders.