Use better performance metrics #5
Comments
|
To combat 'blurry' predictions, we could also try things like the Mean Gradient Error (https://arxiv.org/pdf/1911.09428.pdf) (unfortunately the github repo implementation doesn't exist anymore) that tries to make sure the model learns sharp edges. The authors in that paper combined it with MSE. |
|
It's possible that "blurry" predictions might still allow a downstream neural net to predict solar PV yield, and that the downstream neural net might learn to interpret the "blurry" predictions as low-confidence, which might be useful when the final PV yield prediction is probabilistic. So, yeah, "crisp" predictions would be awesome for human consumption; but I wouldn't worry toooo much if it's not possible to get super-crisp predictions :) Blurry predictions might still be great for predicting PV yield :) |
|
One possible metric is the Structural Similarity Metric (SSIM), there is a PyTorch differentiable version here that I'm going to try out https://github.com/VainF/pytorch-msssim |
|
Could also try using FID, which is correlated to human perception of visual quality, since we want these generated satellite images to look "natural" https://github.com/mseitzer/pytorch-fid |
|
Working on adding some things from this review paper: https://arxiv.org/pdf/2004.05214.pdf one idea that seems promising for creating a model that can be used as an ensemble, is for the generation to be probabalistic, make a few predictions, and then use the one that matches the ground truth the most for calculating the loss and training. Total variation loss has also been added now, and working on doing some more. |
|
LPIPS is another one too https://github.com/S-aiueo32/lpips-pytorch |
|
Try it with only calculating the loss on parts of the image stack where things have moved over a certain amount #88 (comment) |
* Add learnable query Needs more testing, currently does not work Also made slight change as PerceiverIO implementation changed as well in upstream repo * Update Perceiver IO with newer options Add in the decoder feedforward for the multi-pass version Continue working on the SinglePassPerceiver, output shape is wrong still, there is a 2 somewhere that I'm missing * Add decoder_ff as config option * Add total variation loss and Dynamic SSIM loss From https://arxiv.org/pdf/2004.05214.pdf mentioning that SSIM tends to regress to predicting the background, and MSE predicting blurry images, total variation loss is one way to combat that, and another is to only use SSIM on the parts of the image that changes * Add TODO * Add losses to get_loss * Add to assert * Add Gradient Difference Loss, relates to #5 * Remove MetNet and Nowcasting GAN files They are now in their own repos and importable with pip * Remove unused imports * Update to remove shadowing * Fix test * Skip two tests because of changes with Perceiver model * Update test * Remove some of MetNet code and Perceiver Encoders MetNet code is in the new metnet repo, same with the perceiver code in the perceiver-pytorch repo * Update to use OCF perceiver pytorch * Add encoder/decoders * Simplify MultiPerceiverSat using OCF perceiver * Update MultiPerceiverSat to use new perceiver * Only test on 3.8+ * Remove SinglePassPerceiver It can be accomplished with the perceiverio implementation fairly easily now, so once that's merged there can use it. * Remove MetNetPreprocessor Taken care of in the model repos * Switch to pip install * Switch pip name
|
Newer loss functions are being put in https://github.com/openclimatefix/nowcasting_utils to share between predict_pv_yield and SatFlow |
Just MAE or MSE or most normal performance metrics might show that optical flow performs better on average than an ML based approach, even if the ML model outperforms the optical flow in rarer situations or more complicated situations. So something like the MSE per hour, or something other than MSE over all examples might allow us to see where optical flow fails and ML models do well.
The text was updated successfully, but these errors were encountered: