From 41b0b5d7ecf9b0ff319c3666143c0418fab0de0a Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Fri, 29 Mar 2024 17:30:51 -0700 Subject: [PATCH 01/16] Update readme to reflect stable install --- README.md | 13 +++---------- requirements.txt | 22 ---------------------- 2 files changed, 3 insertions(+), 32 deletions(-) delete mode 100644 requirements.txt diff --git a/README.md b/README.md index 5e4d861f..905195f6 100644 --- a/README.md +++ b/README.md @@ -70,10 +70,9 @@ Below we list the dependencies for ARIA-tools ARIA-tools package can be easily installed and used after the dependencies are installed and activated. The third-party RelaxIV package is optional (not required), and only used when opting to minimizing phase-discontinuities. Prior to use of RelaxIV, users should conform to the RelaxIV license agreement. The easiest way of installing RelaxIV is by downloading the min-cost-flow repository in the third-party folder of the ARIAtools and using the setup.py script as outlined below. For the required dependencies, we strongly recommend using [Anaconda](https://www.anaconda.com/distribution/) package manager for easy installation of dependencies in the python environment. ### Conda -Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. Run the commands below to download/clone the ARIA-tools package to your local directory. It is advised to use `mamba` as prompted to speed the install up: +Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. Run the commands below to download/clone the ARIA-tools package to your local directory.: ```.tcsh -conda install mamba git clone https://github.com/aria-tools/ARIA-tools.git cd ARIA-tools ``` @@ -81,16 +80,10 @@ cd ARIA-tools Run the commands below to install dependencies to a new conda environment `ARIA-tools` and activate it: ```.tcsh -mamba env create -f environment.yml +conda env create -f environment.yml conda activate ARIA-tools ``` -Or run the commands below to install dependencies to an existing conda environment (`base` by default): - -```.tcsh -mamba install -c conda-forge --yes --file requirements.txt -``` - We have included a `setup.py` script which allows for easy compilation and installation of third-party dependencies (c-code), as well as for setting up the ARIA-tools package itself (python and command line tools). ```.tcsh python -m pip install -e . @@ -104,7 +97,7 @@ setenv PATH ${PATH}:${PWD}/tools/ARIAtools To avoid potential issues associated with dependencies when cloning new ARIA-tools commits, it is advised to regularly maintain your conda environment as so (making sure to adjust the conda environment argument name `--name ARIA-tools` as appropriate): ```.tcsh -mamba env update --name ARIA-tools --file environment.yml --prune +conda env update --name ARIA-tools --file environment.yml --prune ``` ### Other installation options diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index c593eec6..00000000 --- a/requirements.txt +++ /dev/null @@ -1,22 +0,0 @@ -python>=3.8 -asf_search -cartopy -dem_stitcher>=2.5.0 -gdal>=3.4.1 -h5py -joblib -matplotlib -netcdf4 -pandas -xarray -rasterio -rioxarray -pip -pyproj -requests -scipy>1.10.0 -shapely -## notebook dependencies, keep commented out for better compatiability with other codes -#jupyterlab -#jupyter_contrib_nbextensions -#rise From 6901cc04a5e8d18acf9e269025188e9f43c2faad Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Mon, 1 Apr 2024 16:36:45 -0700 Subject: [PATCH 02/16] Add clarification about mamba in readme --- README.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 905195f6..6568bdc5 100644 --- a/README.md +++ b/README.md @@ -67,12 +67,16 @@ Below we list the dependencies for ARIA-tools ------ ## Installation -ARIA-tools package can be easily installed and used after the dependencies are installed and activated. The third-party RelaxIV package is optional (not required), and only used when opting to minimizing phase-discontinuities. Prior to use of RelaxIV, users should conform to the RelaxIV license agreement. The easiest way of installing RelaxIV is by downloading the min-cost-flow repository in the third-party folder of the ARIAtools and using the setup.py script as outlined below. For the required dependencies, we strongly recommend using [Anaconda](https://www.anaconda.com/distribution/) package manager for easy installation of dependencies in the python environment. +ARIA-tools package can be easily installed and used after the dependencies are installed and activated. The third-party RelaxIV package is optional (not required), and only used when opting to minimizing phase-discontinuities. Prior to use of RelaxIV, users should conform to the RelaxIV license agreement. The easiest way of installing RelaxIV is by downloading the min-cost-flow repository in the third-party folder of the ARIAtools and using the setup.py script as outlined below. + +__[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments," which allows for the easy installation and management of all of the required dependencies. We recommend using the [Miniforge](https://github.com/conda-forge/miniforge) conda environment manager, which uses conda-forge as its default code repo. Alternatively, see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda. ### Conda Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. Run the commands below to download/clone the ARIA-tools package to your local directory.: ```.tcsh +conda config --add channels conda-forge +conda install mamba git clone https://github.com/aria-tools/ARIA-tools.git cd ARIA-tools ``` @@ -80,10 +84,16 @@ cd ARIA-tools Run the commands below to install dependencies to a new conda environment `ARIA-tools` and activate it: ```.tcsh -conda env create -f environment.yml +mamba env create -f environment.yml conda activate ARIA-tools ``` +Or run the commands below to install dependencies to an existing conda environment (`base` by default): + +```.tcsh +mamba install -c conda-forge --yes --file requirements.txt +``` + We have included a `setup.py` script which allows for easy compilation and installation of third-party dependencies (c-code), as well as for setting up the ARIA-tools package itself (python and command line tools). ```.tcsh python -m pip install -e . From ed18c348f1af97f74165ee2be36bdf4018f04acf Mon Sep 17 00:00:00 2001 From: ssaangha Date: Wed, 10 Apr 2024 14:55:29 -0700 Subject: [PATCH 03/16] Add back requirements txt and remove vestigial modules --- environment.yml | 3 --- requirements.txt | 21 +++++++++++++++++++++ 2 files changed, 21 insertions(+), 3 deletions(-) create mode 100644 requirements.txt diff --git a/environment.yml b/environment.yml index 87b55d95..0d4ef223 100644 --- a/environment.yml +++ b/environment.yml @@ -10,13 +10,11 @@ channels: dependencies: - python>=3.8 - asf_search - - cartopy - dem_stitcher>=2.5.0 - gdal>=3.4.1 - h5py - joblib - matplotlib - - netcdf4 - pandas - xarray - rasterio @@ -25,7 +23,6 @@ dependencies: - pyproj - requests - scipy>1.10.0 - - mintpy - shapely #for ARIA-tools-docs - jupyterlab diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..786ac2dc --- /dev/null +++ b/requirements.txt @@ -0,0 +1,21 @@ +python>=3.8 +asf_search +dem_stitcher>=2.5.0 +gdal>=3.4.1 +h5py +joblib +matplotlib +netcdf4 +pandas +xarray +rasterio +rioxarray +pip +pyproj +requests +scipy>1.10.0 +shapely +## notebook dependencies, keep commented out for better compatiability with other codes +#jupyterlab +#rise +#jupyter_contrib_nbextensions From cc6b2b58014c802fc440ba5414473aea5e686560 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Thu, 11 Apr 2024 01:33:58 -0700 Subject: [PATCH 04/16] Resolve jupyter-related install issue --- environment.yml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/environment.yml b/environment.yml index 0d4ef223..3c40e6e2 100644 --- a/environment.yml +++ b/environment.yml @@ -26,7 +26,4 @@ dependencies: - shapely #for ARIA-tools-docs - jupyterlab - - - pip: - - rise - - jupyter_contrib_nbextensions + - jupyter_contrib_nbextensions From d174685583629fa4423144a8982f9ff9f25fb8a4 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Thu, 11 Apr 2024 01:34:20 -0700 Subject: [PATCH 05/16] Make consistent with env yml --- requirements.txt | 2 -- 1 file changed, 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index 786ac2dc..34ef1207 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,7 +5,6 @@ gdal>=3.4.1 h5py joblib matplotlib -netcdf4 pandas xarray rasterio @@ -17,5 +16,4 @@ scipy>1.10.0 shapely ## notebook dependencies, keep commented out for better compatiability with other codes #jupyterlab -#rise #jupyter_contrib_nbextensions From c71460dfea40e315d476195c351fe511ebf9c22e Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Thu, 11 Apr 2024 11:30:00 -0700 Subject: [PATCH 06/16] Commenting jupyter dependencies and addting netcdf4 and dask --- environment.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/environment.yml b/environment.yml index 3c40e6e2..a7af3bf3 100644 --- a/environment.yml +++ b/environment.yml @@ -15,6 +15,7 @@ dependencies: - h5py - joblib - matplotlib + - netCDF4 - pandas - xarray - rasterio @@ -24,6 +25,7 @@ dependencies: - requests - scipy>1.10.0 - shapely + - dask #for ARIA-tools-docs - - jupyterlab - - jupyter_contrib_nbextensions + #- jupyterlab + #- jupyter_contrib_nbextensions From fab1398cf04027049f544dca590a5a27c2490b0b Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Thu, 11 Apr 2024 11:31:00 -0700 Subject: [PATCH 07/16] Updating requirements again to reflect env yml --- requirements.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/requirements.txt b/requirements.txt index 34ef1207..bd446919 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,6 +5,7 @@ gdal>=3.4.1 h5py joblib matplotlib +netcdf4 pandas xarray rasterio @@ -14,6 +15,8 @@ pyproj requests scipy>1.10.0 shapely +dask ## notebook dependencies, keep commented out for better compatiability with other codes #jupyterlab +#rise #jupyter_contrib_nbextensions From 0d4bc46d3cbbd8baf26fcc6f38eb8e55c4a45e49 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Thu, 11 Apr 2024 12:02:22 -0700 Subject: [PATCH 08/16] Adjusted readme to reflect miniforge install --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6568bdc5..793f2882 100644 --- a/README.md +++ b/README.md @@ -75,8 +75,9 @@ __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. Run the commands below to download/clone the ARIA-tools package to your local directory.: ```.tcsh -conda config --add channels conda-forge -conda install mamba +wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh +bash Miniforge3-Linux-x86_64.sh -b -p miniforge +miniforge/bin/mamba init tcsh git clone https://github.com/aria-tools/ARIA-tools.git cd ARIA-tools ``` From e543e021b3bddd7f3ab21eb652344dbe19e19553 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Mon, 15 Apr 2024 10:38:09 -0700 Subject: [PATCH 09/16] Remove `rise` package from comments --- requirements.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index bd446919..b4760aee 100644 --- a/requirements.txt +++ b/requirements.txt @@ -18,5 +18,4 @@ shapely dask ## notebook dependencies, keep commented out for better compatiability with other codes #jupyterlab -#rise #jupyter_contrib_nbextensions From ad044892002ea14e58bb8f321a5598de8db3c817 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Mon, 15 Apr 2024 10:39:08 -0700 Subject: [PATCH 10/16] Fix casing for `netcdf4` --- environment.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/environment.yml b/environment.yml index a7af3bf3..eca8b0cb 100644 --- a/environment.yml +++ b/environment.yml @@ -15,7 +15,7 @@ dependencies: - h5py - joblib - matplotlib - - netCDF4 + - netcdf4 - pandas - xarray - rasterio From d7a4f9715c2843342d58ea0fdaabbb0b29d3c5b5 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Mon, 15 Apr 2024 11:17:47 -0700 Subject: [PATCH 11/16] Make a note that users should specify fresh output directories --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 793f2882..77bf8e35 100644 --- a/README.md +++ b/README.md @@ -145,6 +145,8 @@ export VSI_CACHE=YES The ARIA-tools scripts are highly modulized in Python and therefore allows for building your own processing workflow. Below, we show how to call some of the functionality. For detailed documentation, examples, and Jupyter notebooks see the [ARIA-tools-docs repository](https://github.com/aria-tools/ARIA-tools-docs). We welcome the community to contribute other examples on how to leverage the ARIA-tools (see [here](https://github.com/aria-tools/ARIA-tools/blob/master/CONTRIBUTING.md) for instructions). +* NOTE, currently ARIA-tools does not support deduplication (we are working to add it soon though), so for each commandline input please make sure to specify a fresh output directory with the `--workdir` option. + ### Commandline download of GUNW Products GUNW products can be downloaded through the commandline using the *ariaDownload.py* program, which wraps around the ASF DAAC api. From 0da8ade9e3cc07f350b28cb80832964ce5dc8391 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Tue, 16 Apr 2024 11:44:02 -0700 Subject: [PATCH 12/16] Update PATH for conda --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 77bf8e35..69d8ef4a 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,7 @@ python -m pip install -e . If not using the setup.py, users should compile third-party packages manually and ensure ARIA-tools and dependencies are included on their PATH and PYTHONPATH. For c-shell this can be done as follows (replace "ARIAtoolsREPO" to the location where you have cloned the ARIAtools repository): ```.tcsh setenv PYTHONPATH ${PYTHONPATH}:{$PWD}/tools/ARIAtools -setenv PATH ${PATH}:${PWD}/tools/ARIAtools +setenv PATH ${PATH}:${PWD}/tools/bin ``` To avoid potential issues associated with dependencies when cloning new ARIA-tools commits, it is advised to regularly maintain your conda environment as so (making sure to adjust the conda environment argument name `--name ARIA-tools` as appropriate): From 07ed0ff2136abad07909a836366883f08d91a72b Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Tue, 16 Apr 2024 12:53:43 -0700 Subject: [PATCH 13/16] Clarify supported OS, shells, and installation directory --- README.md | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 69d8ef4a..64a4abfb 100644 --- a/README.md +++ b/README.md @@ -67,22 +67,41 @@ Below we list the dependencies for ARIA-tools ------ ## Installation + + +ARIA-tools has been tested on the following system: +- Linux v.7 and up + +Below we demonstrate how to build and setup an environment from scratch through Linux through a `TCSH` shell. + ARIA-tools package can be easily installed and used after the dependencies are installed and activated. The third-party RelaxIV package is optional (not required), and only used when opting to minimizing phase-discontinuities. Prior to use of RelaxIV, users should conform to the RelaxIV license agreement. The easiest way of installing RelaxIV is by downloading the min-cost-flow repository in the third-party folder of the ARIAtools and using the setup.py script as outlined below. __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments," which allows for the easy installation and management of all of the required dependencies. We recommend using the [Miniforge](https://github.com/conda-forge/miniforge) conda environment manager, which uses conda-forge as its default code repo. Alternatively, see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda. ### Conda -Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. Run the commands below to download/clone the ARIA-tools package to your local directory.: +Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. + +Run the commands below to download and setup your miniforge environment manager: ```.tcsh +cd ~/tools wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh bash Miniforge3-Linux-x86_64.sh -b -p miniforge miniforge/bin/mamba init tcsh +# reset shell +csh +``` + +Run the commands below to download/clone the ARIA-tools package to your local directory: + +```.tcsh +cd ~/tools git clone https://github.com/aria-tools/ARIA-tools.git cd ARIA-tools ``` -Run the commands below to install dependencies to a new conda environment `ARIA-tools` and activate it: +Run the commands below to install dependencies to a new conda environment `ARIA-tools` and activate it. +Make sure to activate your environment each time you open a new session: ```.tcsh mamba env create -f environment.yml @@ -100,7 +119,7 @@ We have included a `setup.py` script which allows for easy compilation and insta python -m pip install -e . ``` -If not using the setup.py, users should compile third-party packages manually and ensure ARIA-tools and dependencies are included on their PATH and PYTHONPATH. For c-shell this can be done as follows (replace "ARIAtoolsREPO" to the location where you have cloned the ARIAtools repository): +If not using the setup.py, users should compile third-party packages manually and ensure ARIA-tools and dependencies are included on their PATH and PYTHONPATH. For `TCSH` shell this can be done as follows (replace `{$PWD}/tools/ARIAtools` to the location where you have cloned the ARIAtools repository): ```.tcsh setenv PYTHONPATH ${PYTHONPATH}:{$PWD}/tools/ARIAtools setenv PATH ${PATH}:${PWD}/tools/bin @@ -108,7 +127,7 @@ setenv PATH ${PATH}:${PWD}/tools/bin To avoid potential issues associated with dependencies when cloning new ARIA-tools commits, it is advised to regularly maintain your conda environment as so (making sure to adjust the conda environment argument name `--name ARIA-tools` as appropriate): ```.tcsh -conda env update --name ARIA-tools --file environment.yml --prune +mamba env update --name ARIA-tools --file environment.yml --prune ``` ### Other installation options From e47f5a79447140eb07939f8f4d976591d1e78e8a Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Tue, 16 Apr 2024 12:57:35 -0700 Subject: [PATCH 14/16] Removing vestigial relax code --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 64a4abfb..99579996 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ ARIA-tools has been tested on the following system: Below we demonstrate how to build and setup an environment from scratch through Linux through a `TCSH` shell. -ARIA-tools package can be easily installed and used after the dependencies are installed and activated. The third-party RelaxIV package is optional (not required), and only used when opting to minimizing phase-discontinuities. Prior to use of RelaxIV, users should conform to the RelaxIV license agreement. The easiest way of installing RelaxIV is by downloading the min-cost-flow repository in the third-party folder of the ARIAtools and using the setup.py script as outlined below. +ARIA-tools package can be easily installed and used after the dependencies are installed and activated. __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments," which allows for the easy installation and management of all of the required dependencies. We recommend using the [Miniforge](https://github.com/conda-forge/miniforge) conda environment manager, which uses conda-forge as its default code repo. Alternatively, see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda. From b87ee06b0274fc3294f1f680cc2bda38fc5886f6 Mon Sep 17 00:00:00 2001 From: Simran S Sangha Date: Tue, 16 Apr 2024 12:58:47 -0700 Subject: [PATCH 15/16] Miniforge clarification --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 99579996..bb38e66a 100644 --- a/README.md +++ b/README.md @@ -79,9 +79,9 @@ ARIA-tools package can be easily installed and used after the dependencies are i __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments," which allows for the easy installation and management of all of the required dependencies. We recommend using the [Miniforge](https://github.com/conda-forge/miniforge) conda environment manager, which uses conda-forge as its default code repo. Alternatively, see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda. ### Conda -Below we outline the different steps for setting up the ARIA-tools while leveraging Anaconda for installation of the requirements. +Below we outline the different steps for setting up the ARIA-tools while leveraging Miniforge for installation of the requirements. -Run the commands below to download and setup your miniforge environment manager: +Run the commands below to download and setup your Miniforge environment manager: ```.tcsh cd ~/tools From 0cbd0be4847fcf2a5c18e85b6b182542d9c33534 Mon Sep 17 00:00:00 2001 From: ssaangha Date: Wed, 17 Apr 2024 09:59:04 -0700 Subject: [PATCH 16/16] Add parallel and pytest --- environment.yml | 2 ++ requirements.txt | 2 ++ 2 files changed, 4 insertions(+) diff --git a/environment.yml b/environment.yml index eca8b0cb..3c3a69dc 100644 --- a/environment.yml +++ b/environment.yml @@ -20,8 +20,10 @@ dependencies: - xarray - rasterio - rioxarray + - parallel - pip - pyproj + - pytest - requests - scipy>1.10.0 - shapely diff --git a/requirements.txt b/requirements.txt index b4760aee..fa16fa44 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,8 +10,10 @@ pandas xarray rasterio rioxarray +parallel pip pyproj +pytest requests scipy>1.10.0 shapely