Relaxing the 1 minute ensemble limit to 30 seconds

[raphidoc]

As of now, the code doesn't allow for ensemble of less than one minute:

HyperCP/Source/ProcessL2.py

Lines 1235 to 1236 in 8ed6754

	if (esStopTime - esStartTime) < datetime.timedelta(seconds=60):
	msg = 'ProcessL2.ensemblesReflectance ensemble is less than 1 minute. Skipping.'

For me, this limit is too stringent. In the case of data acquired at my lab (HOCR .raw file of ~300 kB) each file is composed by less than one minute acquisition.

Nevertheless, by relaxing this 60 seconds limit to 45 or 30 seconds, I can process this data and get good results from it.
I propose to relax the ensemble time lower limit to 45 or 30 seconds to allow for this manual acquisition mode to be processed with HyperCP.

I believe that allowing the re-processing of older datasets is key to the adoption of the software. Once people adopt this software, it will likely be easier to convince them to change their data acquisition habits.

[oceancolorcoder]

This is a good point. I think that historically, many labs have collected very short bursts of data for some reason, and until now I have been resistant to accommodate it. The reason was that after quality controls applied in L1AQC and L1BQC, there are often only 5 or fewer spectra available to work with in one minute of data (theoretically this could be as high as ~30, but it never is). In the scenario of a 1 minute ensemble in L2, one would be faced with discarding (by default) all but perhaps one spectrum during glitter removal (brightest 90% of Lt). That does not allow for any estimation of standard error, and would be far less likely to be representative than an average of multiple spectra over time.

On the other hand, if your situation is simply that your files are broken up into 1 min interval, but run continuously (i.e., one minute per file), your raw files could simple be concatenated to one hour's worth of data and run the usual way.

[oceancolorcoder]

Another way of saying this is that in my experience (depending on the light field and dynamic integration time, of course), we want to go into L2 with enough data density that each ensemble starts out with ~100 spectra, yielding the average of the darkest 10 spectra. For typical QC settings in HCP, this tends to mean 5 minutes.

[oceancolorcoder]

That said, we can make special accommodation to be applied at the discretion of the end user. This lower limit on file size (ensemble size, really) could be a setting in the Configuration window. However, it should be strictly flagged in any data that might be submitted to ocean color archives. Beyond the simple attributes and cryptic header codes, we would want to establish a high-level data quality flag to catch the attention of data recipients (like me).

This could have another advantage in that context: implementing a high-level data quality flag -- not only for this but for wind, clouds, generic use of deglitching or NIR correction, etc. -- would be extremely valuable to those trying to decide whether the quality is sufficient for mission validation or not.

This latter point is probably worth opening a dedicated discussion thread, and chatting about with the team.

[raphidoc]

I just tested the concatenation of raw HOCR files, and it worked seamlessly:

 cat *.raw >> combined.raw

Thank you for the advice!

I understand that the objective is to produce validation-quality data. In regard of the various labs field practices I belive it is important to allow HyperCP to process those datasets that might not be validation quality. This creates a nice learning opportunity for people on how to improve their protocols to improve quality. The high-level data quality flag your talking about would be a good incentive.

[oceancolorcoder]

Agreed.