Adding the Better Tracing Suite into Main

[MaxGamill-Sheffield]

Closes #800

This aims to add features which split out the DNATracing pipeline into smaller parts (disordered tracing, ordered tracing and splining) while adding individual analyses in each of these to be more modular. It also adds topological functions and analyses into ordered tracing, facilitating the processing of catenated molecules as separate objects, and includes a new module to handle and analyse crossings of DNA segments.

[SylviaWhittle]

All checks passed, is this ready? 👀

[SylviaWhittle]

All checks passed 👀

[MaxGamill-Sheffield]

I think it's just the multi-version documentation (v2.0.0) docs that aren't building for this to go through.

Are they building for this version @ns-rse?

[ns-rse]

I think it's just the multi-version documentation (v2.0.0) docs that aren't building for this to go through.

Are they building for this version @ns-rse?

No, there was an error processing the v2.2.0 documentation that halted all subsequent tagged and main sets of documentation from building. The logs are here.

sphinx-multiversion is configured by the options at the bottom of docs/conf.py and uses patterns to determine which branches or versions to build documentation for. Typically we don't want most branches to have documentation built, we only want main and tagged releases.

# sphinx-multiversion https://holzhaus.github.io/sphinx-multiversion/master/configuration.html
smv_tag_whitelist = r"^v\d+.*$"  # Tags beginning with v#
smv_branch_whitelist = r"^main$"  # main branch
# If testing changes locally comment out the above and use the smv_branch_whitelist below instead. Replace the branch
# name you are working on ("ns-rse/466-doc-versions" in the example below) with the branch you are working on and run...
#
# cd docs
# sphinx-multiversion . _build/html
#
# smv_branch_whitelist = r"^(main|ns-rse/466-doc-versions)$"  # main branch
smv_released_pattern = r"^tags/.*$"  # Tags only
# smv_released_pattern = r"^(/.*)|(main).*$"  # Tags and HEAD of main
smv_outputdir_format = "{ref.name}"

This builds for the smv_tag_whitelist tags that match the regular expression (i.e. v<some number>) and for the smv_branch_whitelist the main branch. Thus maxgamill-sheffield/800-better-tracing won't match that and the documentation won't be built.

But I left notes on how to check the documentation for development branches and have worked out that the cause of the errors was the sphinx-autodoc-typehints package. Removing that and unpinning the restriction on sphinx-autoapi that was lingering and following the instructions to include the maxgamill-sheffield/800-better-tracing branch the documentation built locally for all specified branches and tags.

I'm now waiting to see if that works correctly on the workflow.

[ns-rse]

Documentation now builds for...

• all commits with tags matching v\d+.*
• the main branch.

When building locally I've added in the maxgamill-sheffield/800-better-tracing branch and it includes the Advanced Documentation in the single Table of Contents...

Index

Which takes the reader to an index of advanced documentation (I figured this would over time grow)...

Advanced Documentation

Disordered Tracing

Nodestats

Ordered Tracing

Splining

[MaxGamill-Sheffield]

That's amazing thanks @ns-rse!

I've just removed the "DNA Tracing" subtitle from the page as we hope some of these can be used for samples other than DNA.

But after this is this all good to go then?

[SylviaWhittle]

Thank you so much @ns-rse ❤️

[MaxGamill-Sheffield]

Ahhhh the sphinx build fails on installing the fixed topoly version:

ERROR: Ignored the following versions that require a different python version: 1.10.0 Requires-Python <3.12,>=3.8; 1.10.0rc1 Requires-Python <3.12,>=3.8; 1.10.0rc2 Requires-Python <3.12,>=3.8; 1.10.1 Requires-Python <3.12,>=3.8; 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 1.7.2 Requires-Python >=3.7,<3.11; 1.7.3 Requires-Python >=3.7,<3.11; 1.8.0 Requires-Python >=3.8,<3.11; 1.8.0rc1 Requires-Python >=3.8,<3.11; 1.8.0rc2 Requires-Python >=3.8,<3.11; 1.8.0rc3 Requires-Python >=3.8,<3.11; 1.8.0rc4 Requires-Python >=3.8,<3.11; 1.8.1 Requires-Python >=3.8,<3.11; 1.9.0 Requires-Python >=3.8,<3.12; 1.9.0rc1 Requires-Python >=3.8,<3.12; 1.9.0rc2 Requires-Python >=3.8,<3.12; 1.9.0rc3 Requires-Python >=3.8,<3.12; 1.9.1 Requires-Python >=3.8,<3.12
ERROR: Could not find a version that satisfies the requirement topoly==1.0.2 (from topostats) (from versions: 1.0.3, 1.0.4)
ERROR: No matching distribution found for topoly==1.0.2
Error: Process completed with exit code 1.```

[ns-rse]

@MaxGamill-Sheffield I've just been going through the open issues, not all are incorporated in the #800 epic unfortunately.

I found #912 which I thought we could close but on closer reading the problem you highlighted was with the rendering of the docstring example which shows the dictionary structure. Have suggested how to resolve that.

There may be others, I've got about 20 minutes (whilst eating lunch) and then have to start work on my other project. It would be worth going through the last couple of pages of issues seeing what relates to this, whether it can be closed as its completed in some manner or not. I've been pretty poor at adding issues to the Milestone but did try to add them to #800 so they could be ticked off.

NB - I see the gh-pages CI is failing I'll fix that and next week I will be writing to the authors asking for updated versions of Topoly.

[SylviaWhittle]

Ahhhh the sphinx build fails on installing the fixed topoly version:

ERROR: Ignored the following versions that require a different python version: 1.10.0 Requires-Python <3.12,>=3.8; 1.10.0rc1 Requires-Python <3.12,>=3.8; 1.10.0rc2 Requires-Python <3.12,>=3.8; 1.10.1 Requires-Python <3.12,>=3.8; 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 1.7.2 Requires-Python >=3.7,<3.11; 1.7.3 Requires-Python >=3.7,<3.11; 1.8.0 Requires-Python >=3.8,<3.11; 1.8.0rc1 Requires-Python >=3.8,<3.11; 1.8.0rc2 Requires-Python >=3.8,<3.11; 1.8.0rc3 Requires-Python >=3.8,<3.11; 1.8.0rc4 Requires-Python >=3.8,<3.11; 1.8.1 Requires-Python >=3.8,<3.11; 1.9.0 Requires-Python >=3.8,<3.12; 1.9.0rc1 Requires-Python >=3.8,<3.12; 1.9.0rc2 Requires-Python >=3.8,<3.12; 1.9.0rc3 Requires-Python >=3.8,<3.12; 1.9.1 Requires-Python >=3.8,<3.12
ERROR: Could not find a version that satisfies the requirement topoly==1.0.2 (from topostats) (from versions: 1.0.3, 1.0.4)
ERROR: No matching distribution found for topoly==1.0.2
Error: Process completed with exit code 1.```

But if topoly > 1.0.2 then topology tests fail since topology >= 1.0.3 uses new nomenclature right?

Dunno what to do here

[ns-rse]

Dunno what to do here

I bumped the Python version to 3.12 from 3.9 to ensure latest versions of all the optional docs dependencies were being pulled in whilst trying to trouble shoot. Dropping to 3.11 should sort this.

[ns-rse]

The documentation lives! 🧟

Still recommend reviewing Issues to see if there is anything that can be closed or hasn't been addressed (e.g. #912) but 🤞

[SylviaWhittle]

LGTM?

[ns-rse]

I haven't gone through this in one big sitting but have reviewed bits here and there.

If @MaxGamill-Sheffield @SylviaWhittle and @llwiggins are happy it performs as required its ready to go.