OOI processing crashing

[Molkree]

First time I've seen this on September 19 2021.

INFO:Could not get data from 2021-09-19 12:10:00 to 2021-09-19 12:15:00
/home/runner/work/_temp/bd1bec46-4494-4803-8d13-2211f1bf5f52.sh: line 2:  2681 Killed                  python ooi_processing.py -o spectrograms --node PC01A -l 5.0
Error: Process completed with exit code 137.

This indicates that the process was killed by OS. First INFO line shows that there was no data in the 12:10-12:15 range and it failed on the NEXT time step (which will be 12:15-12:20).
When looking at the raw OOI files there are indeed no files up to ~12:16.

I've tested some more and looks like it is failing on the acquisition step using OOIPy. Specifying as little as one second of that first file fails execution:

import datetime

from ooipy.request import hydrophone_request

start_time_str = "2021-09-19T12-16-49"
end_time_str = "2021-09-19T12-16-50"
node = "PC01A"

start_time = datetime.datetime.strptime(start_time_str, "%Y-%m-%dT%H-%M-%S")
end_time = datetime.datetime.strptime(end_time_str, "%Y-%m-%dT%H-%M-%S")

hydrophone_data = hydrophone_request.get_acoustic_data(
    start_time, end_time, node, verbose=True
)

There are some huge allocations happening, around 10GB and then 10GB again.

This sure looks like the OOIPy bug, I will open the issue there.

[Molkree]

For what it's worth, looks like this short time period is the only problematic one (for that day), as after 12:20 everything works fine.