Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO #79

Open
meule opened this issue Nov 24, 2020 · 9 comments
Open

The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO #79

meule opened this issue Nov 24, 2020 · 9 comments

Comments

@meule
Copy link

meule commented Nov 24, 2020

Hey!

I'm processing T12UUB (Canada) and T47QLU (Myanmar) tiles for May-September 2020 and 2019 years with Maja 4.2.0 and I constantly get the error:

2020-11-24 21:37:56 [1655858] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_c37355cbe3907efb5a90a856e4e5d4e5/S2A_MSIL1C_20200616T035541_N0209_R004_T47QLU_20200616T070547.SAFE/GRANULE/L1C_T47QLU_A026027_20200616T040014/../../GRANULE/L1C_T47QLU_A026027_20200616T040014/QI_DATA/MSK_DETFOO_B02.gml !

How to fix that?
@olivierhagolle
Copy link
Contributor

Hi,
a few, rare, ill-formed L1C products lack an information needed by MAJA, to accurately compute the viewing angles, and MAJA is not able to process them. All we can recommend, when it happens, is to remove the product from the list of products to be processed and go on with the other ones.
Sorry for that,
Olivier

@meule
Copy link
Author

meule commented Nov 25, 2020

Thank you Olivier!
Is it possible, that all L1C products for a certain tile are invalid?
I'm trying to run python3 ./Start_maja.py -y --overwrite --type_dem srtm --nbackward 16 -f /home/ubuntu/d/maja_folders.txt -t 12UUB -s MM -d 2020-05-01 -e 2020-10-31
and python3 ./Start_maja.py -y --overwrite --type_dem srtm --nbackward 16 -f /home/ubuntu/d/maja_folders.txt -t 12UUB -s MM -d 2019-05-01 -e 2019-10-31
And I get error messages for all dates, like:

2020-11-24 17:31:57,686 [INFO ] Beginning workplan execution.                                                                                                                                      2020-11-24 17:31:57,686 [INFO ] Executing workplan #1/75
2020-11-24 17:31:59 [1074840] PROGRESS MAJA 4.2 initialize:476 Starting MAJA_L2_BACKWARD_CHAIN                                                                                                     2020-11-24 17:34:32 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2A_MSIL1C_20200607T182921_N0209_R027_T12UUB_20200607T2206
33.SAFE/GRANULE/L1C_T12UUB_A025907_20200607T183828/../../GRANULE/L1C_T12UUB_A025907_20200607T183828/QI_DATA/MSK_DETFOO_B02.gml !
...
2020-11-24 17:35:29 [1074840] PROGRESS MAJA 4.2 process_one_product:854 System infos (RAM;DISK;HOUR;MIN;SEC;LOAD) : 6326;10883;0;3;29;63                                                           2020-11-24 17:36:10 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a
constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2B_MSIL1C_20200605T183919_N0209_R070_T12UUB_20200605T221123.SAFE/GRANULE/L1C_T12UUB_A016970_20200605T184217/../../GRANULE/L1C_T12UUB_A016970_20200605T184217/QI_DATA/MSK_DETFOO_B03.gml !
...
2020-11-24 17:37:45 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2B_MSIL1C_20200602T182919_N0209_R027_T12UUB_20200602T220412.SAFE/GRANULE/L1C_T12UUB_A016927_20200602T183556/../../GRANULE/L1C_T12UUB_A016927_20200602T183556/QI_DATA/MSK_DETFOO_B03.gml !
...
2020-11-24 17:39:34 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a
constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2A_MSIL1C_20200531T183921_N0209_R070_T12UUB_20200531T221758.SAFE/GRANULE/L1C_T12UUB_A025807_20200531T184459/../../GRANULE/L1C_T12UUB_A025807_20200531T184459/QI_DATA/MSK_DETFOO_B03.gml !
...

and so on for all the dates between May and October.
And the same for tile 47QLU

@olivierhagolle
Copy link
Contributor

Glups, I never met such a case, up to now, we only met this rarely. We will have to do something about it
Sorry for that
Olivier

@c00kiemon5ter
Copy link

Hello, I'm trying out MAJA and seeing this, too. The following is the beginning of the process I run with the same error:

The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO

I am using MAJA v4.2.1.

2020-12-03 19:39:33,410 [INFO ] =============This is Start_Maja v4.2.0==============
2020-12-03 19:39:33,415 [WARNI] repCAMS is missing. Processing without CAMS
2020-12-03 19:39:33,418 [INFO ] Detecting input products...
2020-12-03 19:39:33,427 [INFO ] 16 L1C product(s) detected for tile 34SEH in /vagrant/34SEH/L1C/34SEH
2020-12-03 19:39:33,427 [WARNI] No L2A products detected for tile 34SEH in /vagrant/34SEH/L2A/34SEH
2020-12-03 19:39:33,427 [INFO ] Skipping CAMS file detection.
2020-12-03 19:39:33,427 [INFO ] Checking GIPP files
2020-12-03 19:39:33,428 [INFO ] Setting up GIPP folder: /vagrant/34SEH/foo/gipp
2020-12-03 19:39:33,445 [INFO ] Searching for DTM
2020-12-03 19:39:33,448 [INFO ] Found DTM: /vagrant/34SEH/foo/dtm/S2__TEST_AUX_REFDE2_34SEH_1001.HDR
2020-12-03 19:39:33,465 [INFO ] GIPP Creation succeeded for SENTINEL2_NATIF
2020-12-03 19:39:33,467 [INFO ] 13 workplan(s) successfully created:
               DATE |       TILE |     MODE |                                                             L1-PRODUCT | ADDITIONAL INFO
2019-02-08 09:21:41 |      34SEH | BACKWARD |      S2A_MSIL1C_20190208T092141_N0207_R093_T34SEH_20190208T112117.SAFE | Backward of 8 products
2019-02-18 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190218T092031_N0207_R093_T34SEH_20190218T095107.SAFE | L2 from previous
2019-02-28 09:20:21 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190228T092021_N0207_R093_T34SEH_20190228T105030.SAFE | L2 from previous
2019-03-05 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190305T092029_N0207_R093_T34SEH_20190305T132100.SAFE | L2 from previous
2019-03-10 09:20:21 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190310T092021_N0207_R093_T34SEH_20190310T112345.SAFE | L2 from previous
2019-03-15 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190315T092029_N0207_R093_T34SEH_20190315T120923.SAFE | L2 from previous
2019-03-25 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190325T092029_N0207_R093_T34SEH_20190325T121158.SAFE | L2 from previous
2019-03-30 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE | L2 from previous
2019-04-04 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190404T092029_N0207_R093_T34SEH_20190404T112331.SAFE | L2 from previous
2019-04-04 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190404T092029_N0207_R093_T34SEH_20190404T114927.SAFE | L2 from previous
2019-04-14 09:20:39 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190414T092039_N0207_R093_T34SEH_20190414T112240.SAFE | L2 from previous
2019-04-19 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190419T092031_N0207_R093_T34SEH_20190419T112226.SAFE | L2 from previous
2019-04-29 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190429T092031_N0207_R093_T34SEH_20190429T115143.SAFE | L2 from previous
Press Enter to continue...

2020-12-03 19:40:01,109 [INFO ] Beginning workplan execution.
2020-12-03 19:40:01,109 [INFO ] Executing workplan #1/13
2020-12-03 19:40:02 [19612] PROGRESS MAJA 4.2 initialize:476 Starting MAJA_L2_BACKWARD_CHAIN
2020-12-03 19:42:09 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B02.gml !
2020-12-03 19:42:21 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B03.gml !
2020-12-03 19:42:32 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B04.gml !
2020-12-03 19:42:43 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B08.gml !
2020-12-03 19:42:57 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B05.gml !
...

@jprankl
Copy link

jprankl commented Dec 4, 2020

Hi,
same for me. With MAJA 4.2 I get the index_shift error, which I did not get with MAJA 3.x

2020-12-04 11:14:43 [20938] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /data_nvme/tmp_work-test/Start_maja_eaef298c96a5c2596bf70447af4ac908/S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.SAFE/GRANULE/L1C_T33UWP_A023671_20200103T095402/../../GRANULE/L1C_T33UWP_A023671_20200103T095402/QI_DATA/MSK_DETFOO_B05.gml !

Products look somehow correct, (at least the cloud masks), but it seems that shadow masks are missing.
Bests,
Johann

@jerome-colin
Copy link
Collaborator

Hi Konstantin,
I tried to process T12UUB starting from May 2020 and indeed got some products skipped with 'DETFOO'-related 'WARNING', but strictly speaking, it doesn't end in error state, just moves up to the next product. At the time of writing, I got L2A results for May 1st, 16th, 26th and 28th, and it keeps running on June. Does it breaks in error state for you ?
Thanks,
Jerome

@jprankl
Copy link

jprankl commented Dec 4, 2020

Sorry, in my post before I wrote "index_shift error", but Jerome you are right, in my case, it is also a warning and it produces a product.
Bests,
Hannes

@olivierhagolle
Copy link
Contributor

Hi @jprankl, about your note on undetected cloud shadows, this can happen if the multi-temporal methods could not be used. For instance, if the initialization with backward mode had only one image, or at least one usable image (because of the index shift error or because of a very large cloud cover).
If you tell us more about the case where you observe that, we might help.
Olivier

@jprankl
Copy link

jprankl commented Dec 5, 2020

Hi Oliver, thanks, that might be the case. It was initialization in a cloudy season. Actually, the first product ended with an error (which I mentioned in #78), and for the next product (SENTINEL2A_20200103-095716-566_L2A_T33UWP_C_V1-0), MAJA computed an L2A product, but without a cloud mask and with get_detfoo_index_shift warnings.
S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.log
Thanks for your explanation!
Bests!
Johann

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@c00kiemon5ter @olivierhagolle @meule @jprankl @jerome-colin