You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running without --jobfiles (on idev) do we need run_files directory? Could we write the commands that are written into e.g. run_files/run_files/run_02_minopy_phase_inversion into minopy/log?
Consider to clearly say when mintpy is used. E.g.*********** step 7 - ifgram_correction using MintPy ************
rename phase_series.h5 to phase_timeseries.h5
in minopyApp.cfg use minopy.network.type and title it "select the network to unwrap" (maybe the run_file run_04_minopy_generate_ifgram_network)
make the numbers is minopyApp.cfg consistent with the step numbers. E.g. don't use number for minopy.textCmd and minopy.multiprocessing.numProcessor (see MintPy) but give a number for concatenate_patch (just explaining, no options)
in the steps description , explain which files are created in each step (for the pure minopy steps, of course), also why an empty minopy/inverted/interferograms_sequential is created as well as what weighting s used in the network_inversion step.
introduce minopy.inversion.startDate and minopy.inversion.endDate (to be overwritten by mintpy.network.startDate if given (there should be a mintpy.load.startDate but unfortunately it does not exist).
somewhere document properly which mintpy option are used if given (will mintpy.subset be used if given)
how about minopy.plot.coherenceMatrix option. You can give lat,lon or num. For num=3 it will divide equally space the subset to generate a 3x3 grid of points (side note: it would be could to have the ability to give multiple points (minopy.plot.coherenceMatrix=lat1,lon1;lat2,lon2 -- does MintPy has a convention for this? ';' is not great)
weird unwrap/snaphu behavior. When running out of memory it starts again.
top - 22:50:50 up 315 days, 14:07, 1 user, load average: 5.50, 6.04, 6.00
Tasks: 1101 total, 1 running, 1100 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 173.9 free, 3.5 used, 9.9 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 179.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27574 tg851601 20 0 152932 3512 1676 R 0.7 0.0 0:02.59 top
21830 tg851601 20 0 177064 4364 1056 S 0.0 0.0 0:00.07 sshd
21831 tg851601 20 0 115836 2668 1796 S 0.0 0.0 0:00.05 bash
29020 tg851601 20 0 150760 5404 4208 S 0.0 0.0 0:00.04 ssh
29023 tg851601 20 0 177064 4356 1052 S 0.0 0.0 0:00.00 sshd
29024 tg851601 20 0 113312 1828 1464 S 0.0 0.0 0:00.01 bash
top - 23:01:47 up 315 days, 14:18, 1 user, load average: 9.12, 6.57, 6.16
Tasks: 1118 total, 17 running, 1101 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.4 us, 1.0 sy, 0.0 ni, 82.7 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 0.8 free, 167.5 used, 19.0 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 15.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32387 tg851601 20 0 18.9g 18.9g 628 R 100.0 10.1 1:15.89 snaphu
32388 tg851601 20 0 18.9g 18.9g 628 R 100.0 10.1 1:15.93 snaphu
32392 tg851601 20 0 18.9g 18.9g 628 R 100.0 10.1 1:15.62 snaphu
32461 tg851601 20 0 2041844 1.9g 608 R 100.0 1.0 0:23.74 snaphu
32385 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:19.60 snaphu
32390 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:16.33 snaphu
32394 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:13.46 snaphu
32396 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:12.70 snaphu
32397 tg851601 20 0 18.9g 18.9g 628 R
top - 23:02:32 up 315 days, 14:19, 1 user, load average: 5.00, 5.89, 5.96
Tasks: 1102 total, 1 running, 1101 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 173.9 free, 3.5 used, 9.9 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 179.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27574 tg851601 20 0 152932 3552 1676 R 0.7 0.0 0:07.10 top
21830 tg851601 20 0 177064 4364 1056 S 0.0 0.0 0:00.12 sshd
21831 tg851601 20 0 115836 2668 1796 S 0.0 0.0 0:00.05 bash
32492 tg851601 20 0 150760 5404 4208 S 0.0 0.0 0:00.04 ssh
32495 tg851601 20 0 177064 4356 1044 S 0.0 0.0 0:00.00 sshd
32496 tg851601 20 0 113312 1828 1464 S 0.0 0.0 0:00.01 bash
32573 tg851601 20 0 113184 1540 1268 S 0.0 0.0 0:00.00 init_launcher
32575 tg851601 20 0 113320 1580 1292 S 0.0 0.0 0:00.00 launcher
32576 tg851601 20 0 113320 1580 1292 S
Check memory requirement running only one unwrap.py job
top - 23:13:37 up 8 days, 9:49, 1 user, load average: 1.99, 1.09, 0.70
Tasks: 1090 total, 3 running, 1087 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.1 us, 0.0 sy, 0.0 ni, 97.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 142.4 free, 39.7 used, 5.3 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 144.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
380871 tg851601 20 0 18.2g 18.2g 636 R 99.7 9.7 4:00.52 snaphu
380915 tg851601 20 0 18.2g 18.2g 636 R 99.7 9.7 3:09.16 snaphu
380911 tg851601 20 0 152840 3464 1656 R 0.7 0.0 0:01.16 top
380570 tg851601 20 0 113300 1620 1316 S 0.0 0.0 0:00.00 slurm_script
380577 tg851601 20 0 107960 364 284 S 0.0 0.0 0:00.00 sleep
To discuss:
How to deal with cases where we are interested in SHP over a long period but deformation only for a subset of this period?
interferograms_sequential-3for 3 connections if it makes it easier to use different networks.
what is remove filter after unwrap ?
minopy.unwrap.method = auto (snaphu for auto)?
(For the logging, can we have after giving the step the command? Not sure...
******************** step - phase_inversion ********************
Generate /scratch/05861/tg851601/qunittestGalapagosSenDT128/minopy/run_files/run_02_minopy_phase_inversion
Total number of PATCHES: 30
Number of tasks for step phase inversion: 8
20220111:175541 * phase_inversion.py --work_dir /scratch/05861/tg851601/qunittestGalapagosSenDT128/minopy --range_window 19 --azimuth_window 9 --patch_size 200 --method sequential_EMI --test ks --num_worker 4 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --index 0 --slc_stack /scratch/05861/tg851601/qunittestGalapagosSenDT128/minopy/inputs/slcStack.h5
)
The text was updated successfully, but these errors were encountered:
On GitHub page we should document expected processing time and file sizes (slcStack) including typical processing times. At the beginning of the processing these should be returned so the the user can decide whether he/she really wants to process everything. The could be important for processing on the AWS. What resources do we need given the problem at hand?
Example Pichincha data: rename from *.template to *txt following MintPy
******************** step - generate_ifgram ********************etc, give the number of the stepshould be in
qunittestGalapagosSenDT128/logrun_filesdirectory? Could we write the commands that are written into e.g.run_files/run_files/run_02_minopy_phase_inversionintominopy/log?*********** step 7 - ifgram_correction using MintPy ************phase_series.h5tophase_timeseries.h5minopyApp.cfguseminopy.network.typeand title it "select the network to unwrap" (maybe the run_filerun_04_minopy_generate_ifgram_network)minopyApp.cfgconsistent with the step numbers. E.g. don't use number forminopy.textCmdandminopy.multiprocessing.numProcessor(see MintPy) but give a number forconcatenate_patch(just explaining, no options)minopy/inverted/interferograms_sequentialis created as well as what weighting s used in thenetwork_inversionstep.minopy.inversion.startDateandminopy.inversion.endDate(to be overwritten bymintpy.network.startDateif given (there should be amintpy.load.startDatebut unfortunately it does not exist).mintpy.subsetbe used if given)minopy.plot.coherenceMatrixoption. You can give lat,lon or num. For num=3 it will divide equally space the subset to generate a 3x3 grid of points (side note: it would be could to have the ability to give multiple points (minopy.plot.coherenceMatrix=lat1,lon1;lat2,lon2 -- does MintPy has a convention for this? ';' is not great)Check memory requirement running only one unwrap.py job
To discuss:
interferograms_sequential-3for 3 connections if it makes it easier to use different networks.remove filter after unwrap?minopy.unwrap.method = auto (snaphu for auto)?)
The text was updated successfully, but these errors were encountered: