Dev refactor #401

alexfore · 2024-05-01T10:58:20Z

Partial refactor of ARIA-tools

More domain specific knowledge needed to finish refactoring
add util submodule
fix executable permissions
Rename methods in util.shp
Add default args to util.shp.open / save methods
Clean up logging and add --log-level command line args
improve clarity of main computational loop
Remove lat/lon from export_products_worker interface
Add ability to run export_products in parallel with GNU parallel
Add multiproc_method argument to export_products with three options:
- gnu_parallel: will use subjobs, executed in parallel, to process export_products_worker
- threads: will use multiple threads, faster startup/tear down but has the python GIL issue
- single: will run single-threaded

pep8speaks · 2024-05-01T10:58:40Z

Hello @alexfore! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file tools/bin/ariaAOIassist.py:

Line 665:80: E501 line too long (89 > 79 characters)

In the file tools/bin/ariaPlot.py:

Line 185:17: E131 continuation line unaligned for hanging indent
Line 197:80: E501 line too long (149 > 79 characters)
Line 198:36: E127 continuation line over-indented for visual indent
Line 198:80: E501 line too long (168 > 79 characters)
Line 199:35: E131 continuation line unaligned for hanging indent
Line 200:35: E131 continuation line unaligned for hanging indent
Line 201:35: E131 continuation line unaligned for hanging indent
Line 202:35: E131 continuation line unaligned for hanging indent
Line 203:35: E131 continuation line unaligned for hanging indent
Line 204:35: E131 continuation line unaligned for hanging indent
Line 205:35: E131 continuation line unaligned for hanging indent
Line 206:35: E131 continuation line unaligned for hanging indent
Line 207:35: E131 continuation line unaligned for hanging indent
Line 212:80: E501 line too long (90 > 79 characters)
Line 213:80: E501 line too long (119 > 79 characters)
Line 214:32: E128 continuation line under-indented for visual indent
Line 214:80: E501 line too long (91 > 79 characters)
Line 215:31: E126 continuation line over-indented for hanging indent
Line 225:80: E501 line too long (117 > 79 characters)
Line 226:31: E126 continuation line over-indented for hanging indent
Line 226:80: E501 line too long (126 > 79 characters)
Line 232:80: E501 line too long (97 > 79 characters)
Line 233:80: E501 line too long (110 > 79 characters)
Line 234:32: E128 continuation line under-indented for visual indent
Line 234:80: E501 line too long (91 > 79 characters)
Line 235:31: E126 continuation line over-indented for hanging indent
Line 246:80: E501 line too long (110 > 79 characters)
Line 247:32: E128 continuation line under-indented for visual indent
Line 247:80: E501 line too long (91 > 79 characters)
Line 248:31: E126 continuation line over-indented for hanging indent
Line 260:80: E501 line too long (91 > 79 characters)
Line 261:80: E501 line too long (115 > 79 characters)
Line 262:32: E128 continuation line under-indented for visual indent
Line 264:80: E501 line too long (91 > 79 characters)
Line 265:31: E126 continuation line over-indented for hanging indent

Comment last updated at 2024-05-13 10:44:07 UTC

* More domain specific knowledge needed to finish refactoring * add util submodule * fix executable permissions * Rename methods in util.shp * Add default args to util.shp.open / save methods * Clean up logging and add --log-level command line args * improve clarity of main computational loop * Remove lat/lon from export_products_worker interface * Add ability to run export_products in parallel with GNU parallel * Add multiproc_method argument to export_products with three options: * gnu_parallel: will use subjobs, executed in parallel, to process export_products_worker * threads: will use multiple threads, faster startup/tear down but has the python GIL issue * single: will run single-threaded

For a given IFG, avoid mixing v3 and v2 products such that only v3 products are passed for a given IFG where available.

Before, the program checked for gaps between products for a given IFG before filtering out v2 in the presence of v3 products

When running this command: `ariaDownload.py --bbox "34.6 34.8 -118.1 -117.9" --track 71 --output Url --start 20211214 --end 20220102` Vestigial 'inps' variable lead to a crash: ``` /u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/envs/dev_ARIA-tools/bin/ariaDownload.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html __import__('pkg_resources').require('ARIAtools==1.1.6') Traceback (most recent call last): File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/envs/dev_ARIA-tools/bin/ariaDownload.py", line 7, in <module> exec(compile(f.read(), __file__, 'exec')) File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/dev_ARIA-tools/ARIA-tools/tools/bin/ariaDownload.py", line 332, in <module> main() File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/dev_ARIA-tools/ARIA-tools/tools/bin/ariaDownload.py", line 329, in main Downloader(args)() File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/dev_ARIA-tools/ARIA-tools/tools/bin/ariaDownload.py", line 250, in __call__ dst = fmt_dst(inps) ^^^^ NameError: name 'inps' is not defined. Did you mean: 'input'? ```

* tests ariaDownload, ariaExtract, and ariaTssetup * golden input and output data are stored at s3://aria-tools/tests/regression/

* Add NISAR support * ariaExtract updates * Make updates to ariaTSsetup * pep8 changes * Issue pep8 changes to long lines * Push pep8 changes * Adjust long line * Adjust product.py imports * minor style changes * Update masking function (#400) * Update masking function to leverage the OPERA project tile-mate tool * Access to reliable, actively supported high-resolution water masks * Native UTM projections of the NISAR GUNW supported --------- Co-authored-by: Simran S Sangha <[email protected]> Co-authored-by: fore <[email protected]> * Indent if statements to reflect pep8 convention * Indent continuation lines following pep8 suggestion * Split long if statement conditional into multiple lines * Fix minor bug in distinguishing NISAR from S1 GUNWs --------- Co-authored-by: Simran S Sangha <[email protected]> Co-authored-by: fore <[email protected]>

* Found when debugging github issue 396

…r lat

need to use either None or 1 but not a mix of them as default values

alexfore · 2024-05-13T10:40:07Z

I'm not going to address the style issues in ariaPlot at this time -- they are mostly related to the nested list comps which I lack the energy to decompose.

autopep8 code reformat

a5b9c2a

alexfore force-pushed the dev_refactor branch 2 times, most recently from 3b8331a to 5d7b87f Compare May 7, 2024 17:52

alexfore and others added 26 commits May 13, 2024 03:18

Prioritize v3 over v2 products

12eebf0

For a given IFG, avoid mixing v3 and v2 products such that only v3 products are passed for a given IFG where available.

Update to first filter v2 products out per IFG

80f17ba

Before, the program checked for gaps between products for a given IFG before filtering out v2 in the presence of v3 products

make sure version is entered correctly

aff676b

Dev refactor tests (#398)

33a14c2

* tests ariaDownload, ariaExtract, and ariaTssetup * golden input and output data are stored at s3://aria-tools/tests/regression/

Update readme to reflect stable install

07b4586

Add clarification about mamba in readme

2213cee

Add back requirements txt and remove vestigial modules

448a1b4

Resolve jupyter-related install issue

c917d20

Make consistent with env yml

d705a9b

Commenting jupyter dependencies and addting netcdf4 and dask

42ad7e2

Updating requirements again to reflect env yml

83288ac

Adjusted readme to reflect miniforge install

200ea83

Make a note that users should specify fresh output directories

d05e83a

Update PATH for conda

f76231b

Clarify supported OS, shells, and installation directory

0b6ed02

Removing vestigial relax code

625ee33

Miniforge clarification

7aaa81c

Add config file for pep8speaks

54954da

pep8fixes

8ad43c2

Fix two issues in dev_refactor_nisar merge

58d484e

* Found when debugging github issue 396

improve logging of errors when input file doesn't exist

fd1fed3

Fix bug in concatenating products with same footprint, different dates

0463c1a

Fix bug in printing rejected scenes

60f8bb6

Simran S Sangha and others added 8 commits May 13, 2024 03:25

Fix bug in sorting products with same time stamp, but different cente…

af4a285

…r lat

Add missing rankedresample argument back in

0b93fc4

Fix multilooking bug for ancilliary layers

e6db858

Fix bug in default values for multilooking

a21e405

need to use either None or 1 but not a mix of them as default values

Fix some logging issues in ariaDownload.py

be6b48b

reorder imports

800a8da

Add nisar extract test to regression test suite

f7fee2b

Fix bug in ionosphere verbose arguement

7961f6a

alexfore force-pushed the dev_refactor branch from 07e4cd7 to 7961f6a Compare May 13, 2024 10:35

fix line breaks

b5cdb60

alexfore force-pushed the dev_refactor branch from 83dc30a to b5cdb60 Compare May 13, 2024 10:37

Merge branch 'dev' into dev_refactor

144b78c

alexfore merged commit e66ea18 into dev May 13, 2024

alexfore deleted the dev_refactor branch May 14, 2024 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev refactor #401

Dev refactor #401

alexfore commented May 1, 2024 •

edited

Loading

pep8speaks commented May 1, 2024 •

edited

Loading

alexfore commented May 13, 2024

Dev refactor #401

Dev refactor #401

Conversation

alexfore commented May 1, 2024 • edited Loading

pep8speaks commented May 1, 2024 • edited Loading

Comment last updated at 2024-05-13 10:44:07 UTC

alexfore commented May 13, 2024

alexfore commented May 1, 2024 •

edited

Loading

pep8speaks commented May 1, 2024 •

edited

Loading