create_water_mask fails with TIFFReadEncodedTile() failed during InSAR processing

[asjohnston-asf]

Observed this error for ~100 jobs by two users in the vicinity of Beijing, China on 2024-05-21:

ERROR 1: Request for 2826154-2826577 failed
ERROR 1: Request for 3670980-3671403 failed
ERROR 1: Request for 4325250-4325673 failed
ERROR 1: Request for 1154910-1155333 failed
ERROR 1: Request for 1898663-1899086 failed
ERROR 1: Request for 2006723-2007146 failed
ERROR 1: Request for 2328297-2328720 failed
ERROR 1: Request for 2433557-2433980 failed
ERROR 1: Request for 3061653-3062076 failed
ERROR 1: Request for 3927722-3928145 failed
ERROR 1: Request for 4584744-4585167 failed
ERROR 1: Request for 4694786-4695695 failed
ERROR 1: Request for 5775915-5776338 failed
ERROR 1: Request for 7078566-7079392 failed
ERROR 1: Request for 8206198-8206621 failed
ERROR 1: Request for 8667848-8668517 failed
ERROR 1: Request for 8898555-8899891 failed
ERROR 1: Request for 9130282-9131047 failed
ERROR 1: Request for 9247100-9248096 failed
ERROR 1: Request for 9617062-9617956 failed
ERROR 1: Request for 10342569-10343444 failed
ERROR 1: Request for 10469761-10470874 failed
ERROR 1: Request for 10599029-10599896 failed
ERROR 1: Request for 10993785-10994439 failed
ERROR 1: Request for 11392378-11392908 failed
ERROR 1: Request for 12266256-12266732 failed
ERROR 1: Request for 12386616-12387039 failed
ERROR 1: Request for 13620334-13620757 failed
ERROR 1: Request for 14340804-14341227 failed
ERROR 1: Request for 14703535-14703958 failed
ERROR 1: Request for 14821547-14821970 failed
ERROR 1: Request for 14933993-14934416 failed
ERROR 1: Request for 15046516-15046939 failed
ERROR 1: Request for 15157029-15157452 failed
ERROR 1: Request for 16284432-16284855 failed
ERROR 1: Request for 16950412-16950835 failed
ERROR 1: TIFFFillTile:Read error at row 14848, col 14848, tile 13513; got 0 bytes, expected 715
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: /vsicurl/https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/n35e115.tif, band 1: IReadBlock failed at X offset 59, Y offset 62: TIFFReadEncodedTile() failed.

Traceback (most recent call last):
  File "/usr/local/bin/hyp3_gamma", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 38, in main
    process_entry_point.load()()
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 185, in insar
    product_name = insar_sentinel_gamma(
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/ifm_sentinel.py", line 427, in insar_sentinel_gamma
    coords, ref_point_info = unwrapping_geocoding(reference, secondary, step="man", rlooks=rlooks, alooks=alooks,
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/unwrapping_geocoding.py", line 253, in unwrapping_geocoding
    get_water_mask(f"{ifgname}.adf.cc", width, lt, demw, demn, dempar)
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/unwrapping_geocoding.py", line 150, in get_water_mask
    create_water_mask(f'{temp_dir}/tmpgtiff_mask_geo.tif', 'water_mask.tif')
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/water_mask.py", line 144, in create_water_mask
    subprocess.run(translate_command, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gdal_translate', '-co', 'COMPRESS=LZW', '-co', 'NUM_THREADS=all_cpus', 'merged.vrt', 'merged.tif']' returned non-zero exit status 1.

For these InSAR pairs:

['S1A_IW_SLC__1SDV_20230724T101408_20230724T101435_049564_05F5BB_1F9A', 'S1A_IW_SLC__1SDV_20230817T101409_20230817T101436_049914_0600FF_214A']
['S1A_IW_SLC__1SDV_20231215T101409_20231215T101436_051664_063D15_7FE0', 'S1A_IW_SLC__1SDV_20231227T101408_20231227T101435_051839_064321_568B']
['S1A_IW_SLC__1SDV_20231109T101410_20231109T101437_051139_062AE5_3594', 'S1A_IW_SLC__1SDV_20231203T101410_20231203T101437_051489_0636F1_0E4D']
['S1A_IW_SLC__1SDV_20231109T101410_20231109T101437_051139_062AE5_3594', 'S1A_IW_SLC__1SDV_20231121T101410_20231121T101437_051314_0630F7_9CCD']
['S1A_IW_SLC__1SDV_20240201T101407_20240201T101434_052364_0654F5_732A', 'S1A_IW_SLC__1SDV_20240225T101407_20240225T101434_052714_0660D0_80F3']
['S1A_IW_SLC__1SDV_20230218T101402_20230218T101429_047289_05ACDF_36E2', 'S1A_IW_SLC__1SDV_20230302T101402_20230302T101429_047464_05B2C8_AAF7']
['S1A_IW_SLC__1SDV_20231028T101411_20231028T101438_050964_0624DE_2883', 'S1A_IW_SLC__1SDV_20231109T101410_20231109T101437_051139_062AE5_3594']
['S1A_IW_SLC__1SDV_20230525T101404_20230525T101431_048689_05DB13_340E', 'S1A_IW_SLC__1SDV_20230618T101406_20230618T101433_049039_05E5A8_6B31']
['S1A_IW_SLC__1SDV_20230817T101409_20230817T101436_049914_0600FF_214A', 'S1A_IW_SLC__1SDV_20230829T101410_20230829T101437_050089_060703_382C']
['S1A_IW_SLC__1SDV_20230712T101407_20230712T101434_049389_05F067_4D27', 'S1A_IW_SLC__1SDV_20230724T101408_20230724T101435_049564_05F5BB_1F9A']
['S1A_IW_SLC__1SDV_20231121T101410_20231121T101437_051314_0630F7_9CCD', 'S1A_IW_SLC__1SDV_20231203T101410_20231203T101437_051489_0636F1_0E4D']
['S1A_IW_SLC__1SDV_20231016T101411_20231016T101438_050789_061EF5_975D', 'S1A_IW_SLC__1SDV_20231109T101410_20231109T101437_051139_062AE5_3594']
['S1A_IW_SLC__1SDV_20230206T101402_20230206T101429_047114_05A6F5_2E8C', 'S1A_IW_SLC__1SDV_20230218T101402_20230218T101429_047289_05ACDF_36E2']
['S1A_IW_SLC__1SDV_20210511T101352_20210511T101419_037839_04774B_183E', 'S1A_IW_SLC__1SDV_20210604T101354_20210604T101421_038189_0481CE_5900']
['S1A_IW_SLC__1SDV_20200715T101350_20200715T101417_033464_03E0B4_CA2C', 'S1A_IW_SLC__1SDV_20200727T101351_20200727T101418_033639_03E612_4DD2']
['S1A_IW_SLC__1SDV_20210312T101350_20210312T101417_036964_045955_FB18', 'S1A_IW_SLC__1SDV_20210324T101350_20210324T101417_037139_045F65_2DEE']
['S1A_IW_SLC__1SDV_20210228T101350_20210228T101417_036789_04533C_7721', 'S1A_IW_SLC__1SDV_20210312T101350_20210312T101417_036964_045955_FB18']
['S1A_IW_SLC__1SDV_20200609T101348_20200609T101415_032939_03D0BE_1D89', 'S1A_IW_SLC__1SDV_20200703T101349_20200703T101416_033289_03DB5D_780E']
['S1A_IW_SLC__1SDV_20200901T101353_20200901T101420_034164_03F7ED_1BA0', 'S1A_IW_SLC__1SDV_20200925T101354_20200925T101421_034514_040436_2C4F']
['S1A_IW_SLC__1SDV_20210429T101352_20210429T101419_037664_047191_9D76', 'S1A_IW_SLC__1SDV_20210523T101353_20210523T101420_038014_047C93_FE10']
['S1A_IW_SLC__1SDV_20211002T101400_20211002T101427_039939_04B9F7_91A6', 'S1A_IW_SLC__1SDV_20211026T101400_20211026T101427_040289_04C61B_CFB7']
['S1A_IW_SLC__1SDV_20211201T101359_20211201T101426_040814_04D85B_9C88', 'S1A_IW_SLC__1SDV_20220106T101357_20220106T101424_041339_04EA18_C3CA']
['S1A_IW_SLC__1SDV_20201031T101354_20201031T101421_035039_041673_4C5A', 'S1A_IW_SLC__1SDV_20201112T101354_20201112T101421_035214_041C91_AF85']
['S1A_IW_SLC__1SDV_20201124T101354_20201124T101421_035389_04229A_55EC', 'S1A_IW_SLC__1SDV_20201218T101353_20201218T101420_035739_042EA4_B3CF']
['S1A_IW_SLC__1SDV_20200305T101344_20200305T101411_031539_03A223_105D', 'S1A_IW_SLC__1SDV_20200317T101345_20200317T101412_031714_03A833_89CF']
['S1A_IW_SLC__1SDV_20200329T101345_20200329T101412_031889_03AE5B_784C', 'S1A_IW_SLC__1SDV_20200422T101345_20200422T101412_032239_03BAAD_28FC']
['S1A_IW_SLC__1SDV_20191118T101348_20191118T101415_029964_036B7E_A053', 'S1A_IW_SLC__1SDV_20191130T101347_20191130T101414_030139_03718F_A8B3']
['S1A_IW_SLC__1SDV_20190814T101346_20190814T101413_028564_033B00_4E9F', 'S1A_IW_SLC__1SDV_20190826T101346_20190826T101413_028739_03410A_C691']
['S1A_IW_SLC__1SDV_20191224T101346_20191224T101413_030489_037DA6_729C', 'S1A_IW_SLC__1SDV_20200105T101346_20200105T101413_030664_0383B1_316C']
['S1A_IW_SLC__1SDV_20180819T101340_20180819T101407_023314_028919_39C1', 'S1A_IW_SLC__1SDV_20180912T101341_20180912T101408_023664_02943F_5E6A']
['S1A_IW_SLC__1SDV_20191118T101348_20191118T101415_029964_036B7E_A053', 'S1A_IW_SLC__1SDV_20191212T101347_20191212T101414_030314_037799_67E3']
['S1A_IW_SLC__1SDV_20200210T101345_20200210T101412_031189_039611_F087', 'S1A_IW_SLC__1SDV_20200305T101344_20200305T101411_031539_03A223_105D']
['S1A_IW_SLC__1SDV_20191212T101347_20191212T101414_030314_037799_67E3', 'S1A_IW_SLC__1SDV_20200105T101346_20200105T101413_030664_0383B1_316C']
['S1A_IW_SLC__1SDV_20171128T101334_20171128T101401_019464_021067_DB85', 'S1A_IW_SLC__1SDV_20171222T101333_20171222T101400_019814_021B3F_3E0A']
['S1A_IW_SLC__1SDV_20180714T101338_20180714T101405_022789_027870_993F', 'S1A_IW_SLC__1SDV_20180807T101339_20180807T101406_023139_02836E_B567']
['S1A_IW_SLC__1SDV_20170625T101330_20170625T101357_017189_01CAB2_677B', 'S1A_IW_SLC__1SDV_20170719T101331_20170719T101358_017539_01D54E_16A1']
['S1A_IW_SLC__1SDV_20191025T101348_20191025T101415_029614_035F3F_E567', 'S1A_IW_SLC__1SDV_20191118T101348_20191118T101415_029964_036B7E_A053']
['S1A_IW_SLC__1SDV_20170824T101333_20170824T101400_018064_01E543_DF94', 'S1A_IW_SLC__1SDV_20170905T101334_20170905T101401_018239_01EA97_0EBB']
['S1A_IW_SLC__1SDV_20190122T101339_20190122T101406_025589_02D704_AF1D', 'S1A_IW_SLC__1SDV_20190203T101338_20190203T101405_025764_02DD63_CC84']
['S1A_IW_SLC__1SDV_20190110T101339_20190110T101406_025414_02D0A2_2986', 'S1A_IW_SLC__1SDV_20190203T101338_20190203T101405_025764_02DD63_CC84']
['S1A_IW_SLC__1SDV_20180608T101335_20180608T101402_022264_0268C5_9539', 'S1A_IW_SLC__1SDV_20180702T101337_20180702T101404_022614_027335_5546']
['S1A_IW_SLC__1SDV_20171011T101335_20171011T101402_018764_01FAB1_427C', 'S1A_IW_SLC__1SDV_20171023T101335_20171023T101402_018939_020013_11DC']
['S1A_IW_SLC__1SDV_20180819T101340_20180819T101407_023314_028919_39C1', 'S1A_IW_SLC__1SDV_20180831T101340_20180831T101407_023489_028EA8_81F2']
['S1A_IW_SLC__1SDV_20181205T101341_20181205T101408_024889_02BDB4_F88C', 'S1A_IW_SLC__1SDV_20181229T101340_20181229T101407_025239_02CA59_7BDA']
['S1A_IW_SLC__1SDV_20181018T101342_20181018T101409_024189_02A561_34FE', 'S1A_IW_SLC__1SDV_20181111T101341_20181111T101408_024539_02B157_6817']
['S1A_IW_SLC__1SDV_20180714T101338_20180714T101405_022789_027870_993F', 'S1A_IW_SLC__1SDV_20180726T101338_20180726T101405_022964_027DF6_8C6F']
['S1A_IW_SLC__1SDV_20170812T101333_20170812T101400_017889_01DFF9_6E38', 'S1A_IW_SLC__1SDV_20170824T101333_20170824T101400_018064_01E543_DF94']
['S1A_IW_SLC__1SDV_20190802T101345_20190802T101412_028389_033542_5C4F', 'S1A_IW_SLC__1SDV_20190826T101346_20190826T101413_028739_03410A_C691']
['S1A_IW_SLC__1SDV_20191224T101346_20191224T101413_030489_037DA6_729C', 'S1A_IW_SLC__1SDV_20200117T101345_20200117T101412_030839_0389D1_2D10']
['S1A_IW_SLC__1SDV_20190709T101343_20190709T101410_028039_032AA8_66AB', 'S1A_IW_SLC__1SDV_20190721T101344_20190721T101411_028214_032FED_2BDE']
['S1A_IW_SLC__1SDV_20190919T101347_20190919T101414_029089_034D30_CC60', 'S1A_IW_SLC__1SDV_20191013T101348_20191013T101415_029439_03593C_9844']
['S1A_IW_SLC__1SDV_20191013T101348_20191013T101415_029439_03593C_9844', 'S1A_IW_SLC__1SDV_20191106T101348_20191106T101415_029789_036566_0108']
['S1A_IW_SLC__1SDV_20190203T101338_20190203T101405_025764_02DD63_CC84', 'S1A_IW_SLC__1SDV_20190227T101338_20190227T101405_026114_02E9D2_BF76']
['S1A_IW_SLC__1SDV_20191130T101347_20191130T101414_030139_03718F_A8B3', 'S1A_IW_SLC__1SDV_20191212T101347_20191212T101414_030314_037799_67E3']
['S1A_IW_SLC__1SDV_20190510T101340_20190510T101407_027164_030FE6_4036', 'S1A_IW_SLC__1SDV_20190603T101341_20190603T101408_027514_031ACF_A0EA']
['S1A_IW_SLC__1SDV_20180527T101335_20180527T101402_022089_026345_4D9C', 'S1A_IW_SLC__1SDV_20180620T101336_20180620T101403_022439_026E1D_3A18']
['S1A_IW_SLC__1SDV_20190522T101341_20190522T101408_027339_031562_7EA1', 'S1A_IW_SLC__1SDV_20190603T101341_20190603T101408_027514_031ACF_A0EA']
['S1A_IW_SLC__1SDV_20180421T101333_20180421T101400_021564_02529C_B255', 'S1A_IW_SLC__1SDV_20180503T101333_20180503T101400_021739_02581D_2419']
['S1A_IW_SLC__1SDV_20190603T101341_20190603T101408_027514_031ACF_A0EA', 'S1A_IW_SLC__1SDV_20190615T101342_20190615T101409_027689_03201A_37AE']
['S1A_IW_SLC__1SDV_20181123T101341_20181123T101408_024714_02B7C7_5F65', 'S1A_IW_SLC__1SDV_20181217T101340_20181217T101407_025064_02C400_D0D1']
['S1A_IW_SLC__1SDV_20200117T101345_20200117T101412_030839_0389D1_2D10', 'S1A_IW_SLC__1SDV_20200129T101345_20200129T101412_031014_038FF8_EC79']
['S1A_IW_SLC__1SDV_20171128T101334_20171128T101401_019464_021067_DB85', 'S1A_IW_SLC__1SDV_20171210T101334_20171210T101401_019639_0215DD_A725']
['S1A_IW_SLC__1SDV_20181006T101341_20181006T101408_024014_029FAB_3D9C', 'S1A_IW_SLC__1SDV_20181018T101342_20181018T101409_024189_02A561_34FE']
['S1A_IW_SLC__1SDV_20190603T101341_20190603T101408_027514_031ACF_A0EA', 'S1A_IW_SLC__1SDV_20190627T101342_20190627T101409_027864_032551_330F']
['S1A_IW_SLC__1SDV_20190416T101339_20190416T101406_026814_030378_4BD9', 'S1A_IW_SLC__1SDV_20190510T101340_20190510T101407_027164_030FE6_4036']
['S1A_IW_SLC__1SDV_20170625T101330_20170625T101357_017189_01CAB2_677B', 'S1A_IW_SLC__1SDV_20170731T101332_20170731T101359_017714_01DAA9_12EE']
['S1A_IW_SLC__1SDV_20180328T101332_20180328T101359_021214_0247AB_609B', 'S1A_IW_SLC__1SDV_20180421T101333_20180421T101400_021564_02529C_B255']
['S1A_IW_SLC__1SDV_20180831T101340_20180831T101407_023489_028EA8_81F2', 'S1A_IW_SLC__1SDV_20180924T101341_20180924T101408_023839_0299F4_39ED']
['S1A_IW_SLC__1SDV_20190919T101347_20190919T101414_029089_034D30_CC60', 'S1A_IW_SLC__1SDV_20191001T101348_20191001T101415_029264_035336_2AB9']
['S1A_IW_SLC__1SDV_20180515T101334_20180515T101401_021914_025DBA_AA37', 'S1A_IW_SLC__1SDV_20180608T101335_20180608T101402_022264_0268C5_9539']
['S1A_IW_SLC__1SDV_20190203T101338_20190203T101405_025764_02DD63_CC84', 'S1A_IW_SLC__1SDV_20190311T101338_20190311T101405_026289_02F02A_661D']
['S1A_IW_SLC__1SDV_20180620T101336_20180620T101403_022439_026E1D_3A18', 'S1A_IW_SLC__1SDV_20180714T101338_20180714T101405_022789_027870_993F']
['S1A_IW_SLC__1SDV_20170731T101332_20170731T101359_017714_01DAA9_12EE', 'S1A_IW_SLC__1SDV_20170824T101333_20170824T101400_018064_01E543_DF94']
['S1A_IW_SLC__1SDV_20190627T101342_20190627T101409_027864_032551_330F', 'S1A_IW_SLC__1SDV_20190709T101343_20190709T101410_028039_032AA8_66AB']
['S1A_IW_SLC__1SDV_20220331T101306_20220331T101333_042564_0513C2_2EA7', 'S1A_IW_SLC__1SDV_20220412T101307_20220412T101334_042739_0519A2_90F6']
['S1A_IW_SLC__1SDV_20230314T101312_20230314T101339_047639_05B8B7_9531', 'S1A_IW_SLC__1SDV_20230407T101313_20230407T101339_047989_05C485_55A3']
['S1A_IW_SLC__1SDV_20210827T101309_20210827T101336_039414_04A7E8_F113', 'S1A_IW_SLC__1SDV_20210920T101310_20210920T101337_039764_04B3E9_664E']
['S1A_IW_SLC__1SDV_20231109T101320_20231109T101347_051139_062AE5_7A00', 'S1A_IW_SLC__1SDV_20231121T101321_20231121T101348_051314_0630F7_7036']
['S1A_IW_SLC__1SDV_20220822T101315_20220822T101342_044664_0554FB_BA66', 'S1A_IW_SLC__1SDV_20220915T101316_20220915T101343_045014_0560C3_4D24']
['S1A_IW_SLC__1SDV_20230805T101319_20230805T101346_049739_05FB20_C29D', 'S1A_IW_SLC__1SDV_20230829T101320_20230829T101347_050089_060703_772E']
['S1A_IW_SLC__1SDV_20211002T101310_20211002T101337_039939_04B9F7_2DE8', 'S1A_IW_SLC__1SDV_20211026T101310_20211026T101337_040289_04C61B_1CCF']
['S1A_IW_SLC__1SDV_20211213T101309_20211213T101336_040989_04DE5C_1938', 'S1A_IW_SLC__1SDV_20220106T101308_20220106T101335_041339_04EA18_0DE3']
['S1A_IW_SLC__1SDV_20220903T101316_20220903T101343_044839_055AE4_0F6C', 'S1A_IW_SLC__1SDV_20220915T101316_20220915T101343_045014_0560C3_4D24']
['S1A_IW_SLC__1SDV_20220729T101314_20220729T101340_044314_0549E2_58AC', 'S1A_IW_SLC__1SDV_20220822T101315_20220822T101342_044664_0554FB_BA66']
['S1A_IW_SLC__1SDV_20220118T101307_20220118T101334_041514_04EFD4_E72D', 'S1A_IW_SLC__1SDV_20220130T101307_20220130T101334_041689_04F5C9_5D47']
['S1A_IW_SLC__1SDV_20220424T101308_20220424T101335_042914_051F74_25A7', 'S1A_IW_SLC__1SDV_20220518T101309_20220518T101336_043264_052AB1_CBCF']
['S1A_IW_SLC__1SDV_20220717T101313_20220717T101340_044139_0544BC_704F', 'S1A_IW_SLC__1SDV_20220729T101314_20220729T101340_044314_0549E2_58AC']
['S1A_IW_SLC__1SDV_20220915T101316_20220915T101343_045014_0560C3_4D24', 'S1A_IW_SLC__1SDV_20221009T101316_20221009T101343_045364_056C83_EEA7']
['S1A_IW_SLC__1SDV_20220223T101306_20220223T101333_042039_0501F3_1F84', 'S1A_IW_SLC__1SDV_20220307T101306_20220307T101333_042214_0507E2_9D3E']
['S1A_IW_SLC__1SDV_20220412T101307_20220412T101334_042739_0519A2_90F6', 'S1A_IW_SLC__1SDV_20220506T101308_20220506T101335_043089_05253B_0590']
['S1A_IW_SLC__1SDV_20230101T101314_20230101T101341_046589_059552_3387', 'S1A_IW_SLC__1SDV_20230113T101313_20230113T101340_046764_059B32_3A13']
['S1A_IW_SLC__1SDV_20221102T101316_20221102T101343_045714_05779A_6334', 'S1A_IW_SLC__1SDV_20221126T101316_20221126T101343_046064_05836E_CC54']
['S1A_IW_SLC__1SDV_20210920T101310_20210920T101337_039764_04B3E9_664E', 'S1A_IW_SLC__1SDV_20211014T101310_20211014T101337_040114_04BFFD_C829']
['S1A_IW_SLC__1SDV_20220822T101315_20220822T101342_044664_0554FB_BA66', 'S1A_IW_SLC__1SDV_20220903T101316_20220903T101343_044839_055AE4_0F6C']
['S1A_IW_SLC__1SDV_20220506T101308_20220506T101335_043089_05253B_0590', 'S1A_IW_SLC__1SDV_20220518T101309_20220518T101336_043264_052AB1_CBCF']
['S1A_IW_SLC__1SDV_20220118T101307_20220118T101334_041514_04EFD4_E72D', 'S1A_IW_SLC__1SDV_20220211T101306_20220211T101333_041864_04FBD3_0F7D']
['S1A_IW_SLC__1SDV_20230113T101313_20230113T101340_046764_059B32_3A13', 'S1A_IW_SLC__1SDV_20230125T101313_20230125T101340_046939_05A120_0A2F']
['S1A_IW_SLC__1SDV_20230817T101320_20230817T101346_049914_0600FF_6CC9', 'S1A_IW_SLC__1SDV_20230829T101320_20230829T101347_050089_060703_772E']
['S1A_IW_SLC__1SDV_20220211T101306_20220211T101333_041864_04FBD3_0F7D', 'S1A_IW_SLC__1SDV_20220223T101306_20220223T101333_042039_0501F3_1F84']
['S1A_IW_SLC__1SDV_20230101T101314_20230101T101341_046589_059552_3387', 'S1A_IW_SLC__1SDV_20230125T101313_20230125T101340_046939_05A120_0A2F']
['S1A_IW_SLC__1SDV_20230817T101320_20230817T101346_049914_0600FF_6CC9', 'S1A_IW_SLC__1SDV_20230910T101321_20230910T101348_050264_060CF4_D439']

[cirrusasf]

I picked up first 2 pairs and randomly 2 other pairs from above pair list. They are:
pair1
['S1A_IW_SLC__1SDV_20230724T101408_20230724T101435_049564_05F5BB_1F9A', 'S1A_IW_SLC__1SDV_20230817T101409_20230817T101436_049914_0600FF_214A']

pair2
['S1A_IW_SLC__1SDV_20231215T101409_20231215T101436_051664_063D15_7FE0', 'S1A_IW_SLC__1SDV_20231227T101408_20231227T101435_051839_064321_568B']

pair3
['S1A_IW_SLC__1SDV_20220729T101314_20220729T101340_044314_0549E2_58AC', 'S1A_IW_SLC__1SDV_20220822T101315_20220822T101342_044664_0554FB_BA66']

pair4
['S1A_IW_SLC__1SDV_20190919T101347_20190919T101414_029089_034D30_CC60', 'S1A_IW_SLC__1SDV_20191001T101348_20191001T101415_029264_035336_2AB9']

I submitted the InSAR jobs for these 4 pairs at https://hyp3-api.asf.alaska.edu/ui/. All jobs were successful and produced InSAR products. I am not able to catch the error mentioned in the issue.

[cirrusasf]

Compare jobs

Job1: 639aa2b8-c80d-4fcf-8185-4a8086a8085e (failed, by cnuwxc3)

      d0b9ce3e-305a-43e8-b9ba-a24fbc2acc47 (succeeded, by cirrusasf)

Job2: b0f2aca5-c2ac-4817-bffe-67f2d51b9a69 (failed, by cnuwxc3)

"cb45bbb3-4445-45e0-ab38-0e0245a3eea6" (succeeded, by cirrusasf)

I re-run hyp3-gamma against two pairs that were failed by cnuwxc3, with the same parameters, both ran successfully, and produced the products. I have no idea why other person can not ran successfully.

[cirrusasf]

Compare jobs

Job1: 639aa2b8-c80d-4fcf-8185-4a8086a8085e (failed, by cnuwxc3)

      d0b9ce3e-305a-43e8-b9ba-a24fbc2acc47 (succeeded, by cirrusasf)

Job2: b0f2aca5-c2ac-4817-bffe-67f2d51b9a69 (failed, by cnuwxc3)

"cb45bbb3-4445-45e0-ab38-0e0245a3eea6" (succeeded, by cirrusasf)

I re-run hyp3-gamma against two pairs that were failed by cnuwxc3, with the same parameters, both ran successfully, and produced the products. I have no idea why other person can not ran successfully.