AFM-SPM · MaxGamill-Sheffield · Oct 16, 2024 · Aug 31, 2024 · Sep 2, 2024 · Sep 3, 2024
diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs
@@ -1,3 +1,7 @@
+# nodestats codespell corrections
+f13f8d40d6768f14a46315d105b3d80beaa33aaf
+284f113cf8416eed43b88b4c438d7552456616e3
+
 # numpydoc validation - Linting docstrings with pre-commit numpydoc
 836e1bf5347eb7a9f97e784e783045c0287b3fe9
 2580811edefac867938ee0c4b705649a493e5d4f

diff --git a/.github/workflows/sphinx_docs_to_gh_pages.yaml b/.github/workflows/sphinx_docs_to_gh_pages.yaml
@@ -34,8 +34,8 @@ jobs:
           # defined as it defaults to 'main'.
           branch: main
           dir_docs: docs
-          sphinxapiexclude: "../*setup* ../*tests* ../*.ipynb ../demo.py ../make_baseline.py ../jupyter_notebook_config.py ../demo_ftrs.py"
-          sphinxapiopts: "--separate -o . ../"
-          sphinxopts: ""
+          sphinx-apidoc-exclude: "../*setup* ../*tests* ../*.ipynb ../demo.py ../make_baseline.py ../jupyter_notebook_config.py ../demo_ftrs.py"
+          sphinx-apidoc-opts: "--separate -o . ../"
+          sphinx-opts: ""
           multiversion: true
           multiversionopts: ""
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
@@ -28,7 +28,7 @@ jobs:
     strategy:
       matrix:
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.9", "3.10", "3.11"]
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python

diff --git a/.gitignore b/.gitignore
@@ -23,6 +23,7 @@ output/config.yaml
 *.npy
 *.csv
 *.spm
+*.svg
 
 # Notebook checkpoints
 notebooks/.ipynb_checkpoints/
@@ -34,6 +35,7 @@ notebooks/.ipynb_checkpoints/
 .vscode/
 .idea/
 *~
+pytest-debug.ini
 
 # Documentation
 _build/
@@ -57,5 +59,10 @@ topostats/_version.py
 # default output directory, often common from testing
 output/
 
+# Include all files in tests and all subdirectories except processed and __pycache__
+!tests/**
+tests/resources/processed/
+__pycache__/
+
 # Debugging
-pytest-debug.ini
+pytest-debug.ini
diff --git a/.markdownlint-cli2.yaml b/.markdownlint-cli2.yaml
@@ -8,6 +8,7 @@ config:
   html:
     allowed_elements:
       - div
+      - br
 
 # Globs
 globs:

diff --git a/.pylintrc b/.pylintrc
@@ -22,7 +22,6 @@ fail-under=10.0
 # Files or directories to be skipped. They should be base names, not paths.
 ignore=CVS,
        _version.py,
-       dnatracing.py,
        dnacurvature.py,
        conf.py,
        test_dnacurvature.py,

diff --git a/contributing.md b/contributing.md
@@ -22,9 +22,17 @@ not currently considered core elements of `topostats` apart from `Plotting.py`.
 Currently the `topostats` module consists of:
 
 - `default_config.ini` The default config file.
-- `dnatracing.py` Applies tracing functions to each molecule.
-- `pygwytracing.py` The "main" routine.
-- `tracingfuncs.py` Skeletonises and generates backbone traces from masks.
+- `run_topostats.py` The main script for running `topostats` and handles multiprocessing.
+- `entry_point.py` Handles the entry point commandline commands.
+- `processing.py` Handles processing a single AFM file.
+- `filters.py` Handles flattening and pre-processing images.
+- `grains.py` Handles grain segmentation.
+- `grainstats.py` Calculates statistics for each grain.
+- `disordered_tracing.py` Initial tracing of molecules.
+- `nodestats.py` Handles any crossings / branches in DNA grains.
+- `ordered_tracing.py` Proper tracing of molecules.
+- `splining.py` Smooths / splines the traces for a more representative trace.
+- `plotting.py` Handles plotting of the data.
 
 The current working plan is to move to a more modular architecture with new (and existing) functionality being grouped
 by theme within files. We expect to add such files as:

diff --git a/docs/_static/custom.css b/docs/_static/custom.css
@@ -0,0 +1,3 @@
+html[data-theme="dark"] .bd-content img:not(.only-dark, .dark-light) {
+  background-color: transparent !important;
+}
diff --git a/docs/_static/images/disordered_tracing/diagnostics.png b/docs/_static/images/disordered_tracing/diagnostics.png
diff --git a/docs/_static/images/disordered_tracing/height_bias.png b/docs/_static/images/disordered_tracing/height_bias.png
diff --git a/docs/_static/images/disordered_tracing/overview.png b/docs/_static/images/disordered_tracing/overview.png
diff --git a/docs/_static/images/disordered_tracing/pruning.png b/docs/_static/images/disordered_tracing/pruning.png
diff --git a/docs/_static/images/disordered_tracing/skeletons.png b/docs/_static/images/disordered_tracing/skeletons.png
diff --git a/docs/_static/images/disordered_tracing/smoothing.png b/docs/_static/images/disordered_tracing/smoothing.png
diff --git a/docs/_static/images/nodestats/convolved.png b/docs/_static/images/nodestats/convolved.png
diff --git a/docs/_static/images/nodestats/diagnostics.png b/docs/_static/images/nodestats/diagnostics.png
diff --git a/docs/_static/images/nodestats/height_trace.png b/docs/_static/images/nodestats/height_trace.png
diff --git a/docs/_static/images/nodestats/joining.png b/docs/_static/images/nodestats/joining.png
diff --git a/docs/_static/images/nodestats/overview.png b/docs/_static/images/nodestats/overview.png
diff --git a/docs/_static/images/nodestats/pairing.png b/docs/_static/images/nodestats/pairing.png
diff --git a/docs/_static/images/nodestats/tidy.png b/docs/_static/images/nodestats/tidy.png
diff --git a/docs/_static/images/nodestats/topostats_file.png b/docs/_static/images/nodestats/topostats_file.png
diff --git a/docs/_static/images/ordered_tracing/diagnostics.png b/docs/_static/images/ordered_tracing/diagnostics.png
diff --git a/docs/_static/images/ordered_tracing/ns_3d.png b/docs/_static/images/ordered_tracing/ns_3d.png
diff --git a/docs/_static/images/ordered_tracing/ns_segments.png b/docs/_static/images/ordered_tracing/ns_segments.png
diff --git a/docs/_static/images/ordered_tracing/ns_sep_mols.png b/docs/_static/images/ordered_tracing/ns_sep_mols.png
diff --git a/docs/_static/images/ordered_tracing/ns_tracing.png b/docs/_static/images/ordered_tracing/ns_tracing.png
diff --git a/docs/_static/images/ordered_tracing/ns_writhe.png b/docs/_static/images/ordered_tracing/ns_writhe.png
diff --git a/docs/_static/images/ordered_tracing/overview.png b/docs/_static/images/ordered_tracing/overview.png
diff --git a/docs/_static/images/ordered_tracing/topostats_file.png b/docs/_static/images/ordered_tracing/topostats_file.png
diff --git a/docs/_static/images/ordered_tracing/ts_circular.png b/docs/_static/images/ordered_tracing/ts_circular.png
diff --git a/docs/_static/images/ordered_tracing/ts_trace.png b/docs/_static/images/ordered_tracing/ts_trace.png
diff --git a/docs/_static/images/splining/overview.png b/docs/_static/images/splining/overview.png
diff --git a/docs/_static/images/splining/rw_schematic.png b/docs/_static/images/splining/rw_schematic.png
diff --git a/docs/_static/images/splining/rw_sizes.png b/docs/_static/images/splining/rw_sizes.png
diff --git a/docs/_static/images/splining/spl_multi.png b/docs/_static/images/splining/spl_multi.png
diff --git a/docs/_static/images/splining/spl_smooth.png b/docs/_static/images/splining/spl_smooth.png
diff --git a/docs/_static/images/splining/topostats_file.png b/docs/_static/images/splining/topostats_file.png
diff --git a/docs/advanced.md b/docs/advanced.md
@@ -0,0 +1,8 @@
+# Advanced Documentation
+
+You can read more detailed information about the methods implemented in TopoStats in the pages below.
+
+- [Disordered Tracing](advanced/disordered_tracing.md)
+- [Nodestats](advanced/nodestats.md)
+- [Ordered Tracing](advanced/ordered_tracing.md)
+- [Splining](advanced/splining.md)
diff --git a/docs/advanced/disordered_tracing.md b/docs/advanced/disordered_tracing.md
@@ -0,0 +1,171 @@
+# Disordered Tracing
+
+This section gives an in-depth overview of the steps taken in the disordered tracing module.
+
+## At a Glance : Simple Representations
+
+The `disordered_tracing.py` module handles all the functions associated with obtaining single-pixel wide, line
+representations of masked objects.
+
+The quality and likeness of the resultant pruned skeleton thus depends on the quality of the mask, the effectiveness of
+smoothing parameters, the method of skeletonisation, and the quality of automating the pruning of incorrect skeletal
+branches.
+
+![object mask to pruned skeleton](../_static/images/disordered_tracing/overview.png)
+
+This module measures the number of junctions and endpoints for each pruned skeleton object and appends these columns to
+the `all_statistics.csv`. In addition, the `all_disordered_segment_statistics.csv` file is produced which measures the
+length, type, connections, and pixel value (typically height); minimum, middle, median, mean and standard deviation for
+each skeleton segment between junctions using [Skan](https://skeleton-analysis.org/stable/index.html). The branch types
+are given by:
+
+- 0: Endpoint-to-endpoint
+- 1: Endpoint-to-junction
+- 2: Junction-to-junction
+- 3: Isolated cycle
+
+Some useful points to bear in mind:
+
+- **Bad mask, bad skeleton** - If the mask closes holes seen in the image, all skeletonisation methods will produce a
+  single line for this region.
+- **No skeletons in image** - The disordered trace `core` image may not show the resultant skeletons if the plotting
+  `dpi` value is too low (varies based on image size).
+- **Hard to remove branches** - If there are still spurious branches to prune after modifying the
+  `mask_smoothing_params` and `pruning_params`, try increasing the `filters.gaussian_size` value to smooth the image the
+  mask is created from.
+- **Masked colours are relative** - Any mask colours that may be produced by plots are relative to the mask values in
+  that image as they will always span the masked colourmap, and will not compare well across images if the range of mask
+  values differ.
+
+## Processing Steps
+
+### 1. Smoothing the Mask
+
+Generated masks can be quite jagged which causes a large increase in spurious skeletal branches which do not best
+represent the image. This can be resolved by first smoothing the incoming mask.
+
+![Image of smoothed vs un-smoothed skeletonisation result](../_static/images/disordered_tracing/smoothing.png)
+
+Smoothing uses either binary dilation (expanding the borders of the mask `dilation_iterations` times) or an otsu
+threshold applied to a gaussian filtered mask (with a smoothing factor of `gaussian_sigma`). If both values are provided
+an input in the configuration file, they will compete, and the winning result is the one with the smallest number of
+added pixels.
+
+The reason for the competition is an attempt to conserve mask topology i.e. any small holes it has which may become
+closed upon dilation / gaussian blurring. The dilation method seems to work best for larger masks where there are more
+mask pixels, and the gaussian smoothing better when there are small holes.
+
+In addition, this smoothing step also tries to preserve mask topology by re-adding any holes back into the mask that lie
+within the `holearea_min_max` threshold. This has the resulting effect of only smoothing the outer edge of the mask.
+
+### 2. Skeletonisation
+
+Skeletonisation is the process of reducing a binary image to a single-pixel wide representation. This can be done using
+the algorithms provided by [scikit-image](https://scikit-image.org/docs/stable/auto_examples/edges/plot_skeleton.html)
+such as `zhang` (rule based + erosion), `lee` (connectivity preserving), `medial_axis` (pixels with >1 closest boundary
+pixels), or `topostats` - a modification of Zhang's algorithm which uses the underlying height information to order and
+then remove only a percentage of the pixels marked for removal on each skeletonisation iteration. The skeletonisation
+methods are handled by the `tracing/skeletonize.py` module.
+
+![Comparison between skeletonisation algorithms](../_static/images/disordered_tracing/skeletons.png)
+
+We have found that by including the height information into the skeletonisation process and removing the lowest
+`height_bias` percent, we can bias the skeleton to lie on the DNA backbone and generate a better representation of the
+molecule, especially at crossing points and regions where the mask is less accurate.
+
+![Different skeletonisation height bias examples](../_static/images/disordered_tracing/height_bias.png)
+
+### 3. Pruning
+
+Pruning is the act of removing spurious branches from a skeleton which does not follow the underlying masks' shape. To
+this end, TopoStats provides a variety of methods and parameters to help clean up the skeletons.
+
+![Pre and post pruning of skeletons](../_static/images/disordered_tracing/pruning.png)
+
+Pruning can be done by branch length (using the `max_length` configuration parameter) and / or the branch height (using
+the `height_threshold`, `method_values` and `method_outliers` configuration parameters).
+
+Length pruning is the simplest, iteratively comparing the length (in nm) of each branch containing an endpoint with the
+`max_length` parameter. If it's length is below this value, it is deemed as a spurious branch and removed, along with
+any junction pixels until a single-pixel skeleton remains.
+
+Height pruning also iteratively compares the branches pixel values (height) to a `height_threshold` parameter, again,
+any branch heights which are below this value are pruned. However, as there are multiple pixels in a branch, we provide
+different methods to obtain a single branch height value for comparison i.e. `method_values`:
+
+- `min` - the minimum value of all the branch pixel values.
+- `median`- the median value of all the branch pixel values.
+- `mid` - the middle value (averaged if two) of the ordered branch pixels. This is particularly useful for pruning false
+  bridges where the height dips in the middle of the branch.
+
+Additionally, how these branch height values compare to the `height_threshold` is also considered i.e `method_outliers`:
+
+- `abs` - prunes branch values below the absolute value of the `height_threshold`.
+- `mean_abs` - prunes branch values below the whole skeleton mean pixel value - absolute threshold. This is useful for
+  non-surface samples or periodic structures e.g. in DNA we expect the mean height to be around 2nm, but high resolution
+  imaging may cause this to dip to 0.8nm (the depth of a major groove), so we'd want to prune branch heights below this.
+- `iqr` - prunes branch values below 1.5x inter-quartile range (IQR) of all the branches. Height pruning cannot produce
+  more than one skeleton and so avoids breaking up the skeleton into multiple parts.
+
+## Outputs
+
+The `<image>_<threshold>_disordered_trace` image shows the pruned skeletons that are used to obtain the below metrics
+and passed onto future processing stages.
+
+For each grain, the following new columns are added to the `grainstats.csv` file:
+
+| Column Name           | Description                                                                             | Data Type |
+| --------------------- | --------------------------------------------------------------------------------------- | --------- |
+| `grain_endpoints`     | The number of pixels designated as endpoints (only 1 neighbour) in the pruned skeleton. | `integer` |
+| `grain_junctions`     | The number of pixels designated as junctions (>2 neighbours) in the pruned skeleton.    | `integer` |
+| `total_branch_length` | The sum of all branch lengths in the pruned skeleton.                                   | `float`   |
+
+&nbsp;
+
+An `all_disordered_segment_statistics.csv` file is produced for each image which measures the following metrics from
+each segment in each pruned skeleton:
+
+| Column Name          | Description                                                                                                                | Data Type |
+| -------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------- |
+| `image`              | The image name being processed.                                                                                            | `string`  |
+| `threshold`          | The direction of the grain threshold being applied.                                                                        | `string`  |
+| `grain_number`       | The number of the grain being processed in the image.                                                                      | `integer` |
+| `index`              | The branch index.                                                                                                          | `integer` |
+| `branch-distance`    | The distance (in nm) of the branch.                                                                                        | `float`   |
+| `branch-type`        | Branch classification of endpoint-to-endpoint (0), endpoint-to-junction (1), junction-to-junction (2), isolated cycle (3). | `integer` |
+| `connected_segments` | The index of the branch segments that this current branch is connected to via a junction point.                            | `list`    |
+| `mean-pixel-value`   | The mean of the branch pixel values (height), in nm.                                                                       | `float`   |
+| `stdev-pixel-value`  | The standard deviation of the branch pixel values (height), in nm.                                                         | `float`   |
+| `min-value`          | The minimum value of the branch pixel values (height), in nm.                                                              | `float`   |
+| `median-value`       | The median value of the branch pixel values (height), in nm.                                                               | `float`   |
+| `mid-value`          | The value of a pixel halfway along the ordered branch (height), in nm.                                                     | `float`   |
+| `basename`           | The directory path containing the image.                                                                                   | `string`  |
+
+## Diagnostic Images
+
+Images produced by the `plotting.image_set: all` for this module are:
+
+- `21-smoothed_grains` - The smoothed mask, used to check that the image topology is retained (holes) before
+  skeletonisation.
+- `22-original_skeletons` - Skeletonised mask, used to ensure the skeletons follow the underlying structures. Used to
+  check if the skeletonisation parameters are suitable.
+- `23-branch_indexes` - An integer mask of the pruned skeleton with branch pixel values matching the index value in the
+  data. Used to cross reference the `all_disordered_segment_statistics.csv` data with an image. Using the default
+  colourmap for this (viridris), darker (purple) colours are lower indexes, and brighter (yellow) colours are higher.
+- `24-branch_types` - An integer mask of the pruned skeleton with branch pixel values matching the `branch-type`
+  (numbers and definitions in the "At a Glance" section). This can be used to count and check if the skeletonisation
+  process correctly identifies the different branch types. Using the default colourmap for this (viridris), darker
+  (purple) colours are lower indexes, and brighter (yellow) colours are higher.
+
+![Examples of diagnostic images](../_static/images/disordered_tracing/diagnostics.png)
+
+## Possible Uses
+
+This module would lend itself to measuring branched structures and may aid the identification of particular regions by
+filtering-out segments based on their branch type.
+
+We have used this module to identify and measure the length of reverse forks in stalled DNA replication intermediates as
+these structures should produce closed loops, however, the reverse forks can be identified as branches with endpoints
+and their metrics identified from the data. It has also been used as part of the pipeline to obtain ordered traces along
+topologically complex DNA molecules and topological classifications in our paper; [Under or Over? Tracing Complex DNA
+Topologies with High Resolution Atomic Force Microscopy](https://www.biorxiv.org/content/10.1101/2024.06.28.601212v1).