10m RTC fails with segmentation fault (core dumped)

[jhkennedy]

These three granules all failed RTC processing at 10m (linked to the log file):

• S1B_IW_GRDH_1SDV_20190104T231244_20190104T231309_014351_01AB40_D0FA
• S1B_IW_GRDH_1SDV_20200111T231250_20200111T231315_019776_025643_788C
• S1B_IW_GRDH_1SDV_20210105T231256_20210105T231321_025026_02FA86_628D

with a segmentation fault (core dumped) during mk_geo_radcal2. Here's the relevant snippet from the logs:

09/08/2021 12:01:44 AM - INFO - Proc: ras_linear ./corrected.ls_map_rdc 32819 1 - 1 1 0 20 0 - ./corrected_0.ls_map_rdc.bmp
09/08/2021 12:01:44 AM - INFO - Proc: WARNING: BMP format image width out of valid range (1->32767): 32819
09/08/2021 12:01:44 AM - INFO - Proc: Segmentation fault (core dumped)
09/08/2021 12:01:44 AM - INFO - Proc: ERROR /usr/local/GAMMA_SOFTWARE-20191203/DIFF/scripts/mk_geo_radcal2: non-zero exit status: ras_linear ./corrected.ls_map_rdc 32819 1 - 1 1 0 20 0 - ./corrected_0.ls_map_rdc.bmp
09/08/2021 12:01:44 AM - INFO - Finished: mk_geo_radcal2 vv.mli vv.mli.par dem.image dem.par dem_seg dem_seg.par . corrected 10.0 0 -q
09/08/2021 12:01:44 AM - ERROR - Nonzero return value!
Traceback (most recent call last):
  File "/usr/local/bin/hyp3_gamma", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/__main__.py", line 32, in main
    load_entry_point('hyp3_gamma', 'console_scripts', args.process)()
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/__main__.py", line 73, in rtc
    dem_name=args.dem_name,
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/rtc/rtc_sentinel.py", line 350, in rtc_sentinel_gamma
    run(f'mk_geo_radcal2 {mli_image} {mli_par} {dem_image} {dem_par} dem_seg dem_seg.par . corrected '
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/rtc/rtc_sentinel.py", line 147, in run
    execute(cmd, uselogging=True)
  File "/usr/local/lib/python3.6/dist-packages/hyp3lib/execute.py", line 77, in execute
    raise ExecuteError(tool + ': ' + line)
hyp3lib.exceptions.ExecuteError: mk_geo_radcal2: ERROR /usr/local/GAMMA_SOFTWARE-20191203/DIFF/scripts/mk_geo_radcal2: non-zero exit status: ras_linear ./corrected.ls_map_rdc 32819 1 - 1 1 0 20 0 - ./corrected_0.ls_map_rdc.bmp

It's possible that this is just hitting the instance memory limit, and upping the machine size may solve it, but I have not investigated this yet.

---

Note: All jobs were requested with these options:

job_definition = {
    'job_type': 'RTC_GAMMA',
    'job_parameters': {
        'resolution': 10,
        'speckle_filter': True,
        'radiometry': 'gamma0',
        'scale': 'power',
        'dem_matching': True,
    }
}