consider exclude in shed update

[bernt-matthias]

by considering exclude when

• build and downloading tar balls
• diff

shed update should now consider exclude in .shed.yml

I checked this (a1e2cae) in the following way on the TTS.

• added a file (exclude.txt) to a repo (https://testtoolshed.g2.bx.psu.edu/repos/mbernt/fail)
• shed_update
• added the exclude to shed.yml
• shed_update

Afterwards the file was gone again (on the TTS).

Would be great to have for updating the OpenMS tools (where we have quite a few extra files in the TS that shouldn't be there, i.e. we would like to remove them from the TS and not consider them for the question if a repo should be updated).

TODO(?):

• I guess we could/should do the same for include, but I'm not sure what the default '**' does
• not sure if the use of exclude in diff, os.walk, and tar is equivalent (which would be nice) .. also it should be consistent with the docs (https://planemo.readthedocs.io/en/latest/standards/docs/best_practices/shed_yml.html#shed-upload-includes-excludes) and I think it is not yet
• which other planemo commands no not consider exclude/include and should

may fix: #554

[jmchilton]

This looks good to me, is it ready to be merged?

[bernt-matthias]

This looks good to me, is it ready to be merged?

Wondering about the questions in the TODOs. In particular question 2.

Is there anything that deserves a test in this changes? If yes ideas?

[nsoranzo]

build_tarball is also used in planemo/commands/cmd_shed_build.py , so that needs to be also updated or exclude made optional.

[bernt-matthias]

build_tarball is also used in planemo/commands/cmd_shed_build.py , so that needs to be also updated or exclude made optional.

No problem, but I'm still quite confused by the code. It seems that RawRepositoryDirectory already considers excludes (

planemo/planemo/shed/__init__.py

Line 1025 in 38227c1

excludes = _shed_config_excludes(config)

) partially (for instance not in _realized_files) but RealizedRepositry not (btw. do we want to fix the typo).

Both tools essentially start by calling shed.for_each_repository -> _realize_effective_repositories -> there _find_raw_repositories constructs RawRepositoryDirectorys which are then turned into RealizedRepositry. So I guess the excludes could be ignored in the calls to build_tarball from the commands.

Is this correct?

[jmchilton]

I don't remember all the details - but the idea is that the raw directory structures are materialized into effective repository structures with includes and excludes handled and tools and tool-data demultiplexed. So should any of this be needed? Maybe exclude is only used for automatic demultiplexing though?