Add Joint Model

[jacobbieker]

Detailed Description

We want a model that can predict both satellite imagery and pv yield.

Context

While we ultimately just care about the PV output of the model, predicting where the clouds are moving in the satellite imagery should be a good auxiliary task that might help the model learn what is important vs not.

Possible Implementation

Add another version of the Perceiver that takes two queries, one for the satellite imagery and one for PV output.

[JackKelly]

Sounds good!

Let's use this new issue to discuss the joint model. I'll close some older issues where we've discussed related ideas:

• ML architecture idea: Perceiver IO as an RNN predict_pv_yield#68
• Enable "RNN-like" behaviour for sequences: Each cross-attention gets a different input timestep perceiver-pytorch#1 (comment)

As I mumbled about on the meeting this afternoon, I think one of the first steps towards the 'joint model' is to get the position encoding right for the recent history of the PV power data. For each measurement of historical PV power, we definitely want to encode the position in time and space, probably using the same encoding as for the satellite data (so the model can associate historical PV data with historical clouds: it can see "ooh, look, when that cloud passed over that PV system half an hour ago, it reduced power output by 50%").

We may also want the PV "position" encoding to encode an embedding of the ID of the PV system ("ooh, I remember that PV system was unusually dirty, so I'll compensate for that")

(Maybe that "PV-position-encoding" code should go into nowcasting_utils, so predict_pv_yield can use it?)

[jacobbieker]

Yeah, I think breaking out the PV position encoding, and maybe the position encoding in general should probably go into nowcasting_utils I've opened one here https://github.com/openclimatefix/nowcasting_utils/issues/30