Instability of LightGlue training

[LudvigDillen]

Hi there,

I have worked with LightGlue for some time now and unfortunately experienced quite severe stability issues.
If I run
python -m gluefactory.train sp+lg_megadepth --conf gluefactory/configs/superpoint+lightglue_megadepth.yaml --mp bfloat16

but with some small changes in the config, namely with
Before, After

data.batch_size: 32, 30
data.load_features.do: false, true
model.matcher.flash: false, true
model.matcher.weights: --, superpoint
train.log_every_iter: 100, 20
train.lr: 1e-4, 5e-5

The reason I am training from pre-trained weights here is because I want to add things on top of LightGlue.
Down below is some output that shows severe problems. This has been tested two times with similar results. Is this a known problem? I would very much like to solve it. The only time I got rid of this problem was when I replaced the sigmas with dustbins as in SuperGlue.