From d81366c26a6adb9798e41cf2b279726a8b972ec8 Mon Sep 17 00:00:00 2001 From: bovenzin Date: Tue, 3 Mar 2020 21:57:10 +0100 Subject: [PATCH 1/2] Update YourReview.md --- paper-to-review/YourReview.md | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/paper-to-review/YourReview.md b/paper-to-review/YourReview.md index 021b93e..4b03a78 100644 --- a/paper-to-review/YourReview.md +++ b/paper-to-review/YourReview.md @@ -1,22 +1,39 @@ # Your review -Selected paper: PUT THE NAME OF THE PAPER HERE +Selected paper: Improving Subseasonal Forecasting in the Western U.S. with Machine Learning ### Summary +In this paper the authors propose a machine-learning based system to predict the average temperature and +and total precipitation with two-to-four and four-to-six weeks advance. Their study responds to a year-long challenge issued by national institutions in order to improve the accuracy of midterm forecasts, that are complicated as they depend on both local weather and global climate variables. The contest asked participants to submit every two weeks their forecasts for each target variable (average temperature and total precipitation) for each of the two forecast horizons (3-4 and 5-6 weeks respectively). Upon definition of the ''anomaly'' as the difference between target variable and a longterm average of that same variable over a 30 year period, the accuracy of the forecast at a given target date is measured by the cosine similarity (also called "skill") between the predicted and observed anomaly vectors. The spatial region of interest was mapped into a grid of 514 points with a resolution of 1°x1° and each vector has as many components as the numer of grid points. Finally, for each target variable and horizon forecast, the average skill is calculated over the whole period of the challenge and the results are compared to those of other models. +The approach presented in the paper consists in combining two regression models, both of them based on "multitask feature selection", meaning that the variables for a target date are selected jointly - instead of selecting them independently for each grid point - while the regression coefficients are fit independently for each grid point. + +The first model is a Local Linear Regression with Multitask Feature Selection (MultiLLR), it has been trained on data from different sources that the authors collected in their SubseasonalRodeo Dataset, including lagged measurements of temperature, precipitation, humidity, pressure, wind, Madden-Julian oscillation. “Local” means that the training data is restricted to measurements within a range of 56 days around the target date. As the importance of the features depends on the target date (for instance, high humidity level can be a better predictor for precipitation in winter months than in summer), backward feature selection is performed for each target date. Starting with all thefeatures in the model at step zero, then each step the feature that influence the predictive measurement of the cosine similarity the least, is removed, until when further removals that change the outcome predictive measurement within a tolerance range are no more possible. + +The second model is a Multitask k-Nearest Neighbor Autoregression (AutoKNN). It uses as features only historical measurements of the target variables from l days prior to the target date, with l taking three fixed values, plus the constant intercept (with value equal 1 for all datapoints) and the observed anomalies of the k most similar neighbors to the target date. The most similar neighbor is identified as the candidate date, within a period of one year prior to the target date, whose anomaly vector is as similar as possible to the target date's anomaly vector. The authors include k=20 most similar neighbors when predicting temperatures, for a total of 24 features, while k=1 when predicting precipitations (5 features in total in total). This difference can be explained by the fact that when predicting precipitation, the top neighbors for a target date are shown to be from the same time of year as the target date, while for temperature the top neighbors are drawn from throughout the year, regardless of the month of the target date. The authors show corresponding plots. + +Finally, the two models are combined and the average of their l2-normalized predicted anomalies is defined as the ensemble anomaly. The authors motivate this choice by showing the mathematical proof that the so-defined ensemble anomaly is guaranteed to produce a better skill than the average of the skills of the individual models. + +The individual models and the esemble model are evaluated over the contest period and also over each year in the period 2011-2018, and they are shown to outperform both the benchmark models provided by the organizers of the challenge and the top competitor. + Put your summary of the paper here ### Strengths -Put strengths of the paper here +The study addresses the problem of forecasting on a time scale which has been defined “predictability desert” for the lack of models and methods that give good predictions. +The models used in this paper are conceptually simple as they use linear regression. +The authors have collected data from many different sources, realising a dataset for each target variable and each time range. + ### Weaknesses and limitations -Put weaknesses and limitations of the paper here +Besides considering the exact average of the two models, it would have been interesting to consider a weighted average of them when calculating the ensemble anomaly, and varying the relative weight to find the optimal ensemble skill. For instance, I believe that privileging the AutoKNN model would give a better mean skill for precipitation at 3-4 weeks. ### Implications +Societal implications: Improvement of subseasonal forecasts can lead to better management of water resources, agriculture processes, food prodcution, it helps to reduce the risks associated to extrem weather events + +Research implications: following this study, other multi-model combinations can be tried and the SubseasonalRodeo dataset can be used as a benchmark to train machine-learning models. -Put implications of the paper here ### From 0cdcd82cf92f84dd22d68f505031b2828322e039 Mon Sep 17 00:00:00 2001 From: bovenzin Date: Tue, 3 Mar 2020 23:41:34 +0100 Subject: [PATCH 2/2] Update YourReview.md --- paper-to-review/YourReview.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/paper-to-review/YourReview.md b/paper-to-review/YourReview.md index 4b03a78..494597e 100644 --- a/paper-to-review/YourReview.md +++ b/paper-to-review/YourReview.md @@ -4,25 +4,23 @@ Selected paper: Improving Subseasonal Forecasting in the Western U.S. with Machi ### Summary -In this paper the authors propose a machine-learning based system to predict the average temperature and -and total precipitation with two-to-four and four-to-six weeks advance. Their study responds to a year-long challenge issued by national institutions in order to improve the accuracy of midterm forecasts, that are complicated as they depend on both local weather and global climate variables. The contest asked participants to submit every two weeks their forecasts for each target variable (average temperature and total precipitation) for each of the two forecast horizons (3-4 and 5-6 weeks respectively). Upon definition of the ''anomaly'' as the difference between target variable and a longterm average of that same variable over a 30 year period, the accuracy of the forecast at a given target date is measured by the cosine similarity (also called "skill") between the predicted and observed anomaly vectors. The spatial region of interest was mapped into a grid of 514 points with a resolution of 1°x1° and each vector has as many components as the numer of grid points. Finally, for each target variable and horizon forecast, the average skill is calculated over the whole period of the challenge and the results are compared to those of other models. -The approach presented in the paper consists in combining two regression models, both of them based on "multitask feature selection", meaning that the variables for a target date are selected jointly - instead of selecting them independently for each grid point - while the regression coefficients are fit independently for each grid point. +In this paper the authors propose a machine-learning based subseasonal forecasts of average temperature and total precipitation in Western US. Their study responds to a year-long challenge issued by national institutions in order to improve the accuracy of midterm forecasts, that are complicated as they heavily depend on both local weather and global climate variables. The contest asked participants to submit every two weeks their forecasts for each target variable (average temperature and total precipitation) and two forecast horizons (3-4 and 5-6 weeks in advance). Upon definition of the ''anomaly'' as the difference between target variable and a longterm average of that same variable over a 30 year period, the accuracy of the forecast at a given target date is measured by the cosine similarity (also called "skill") between the predicted and observed anomaly vectors. The spatial region of interest is mapped into a grid of 514 points with a resolution of *1°x1°*: each anomaly vector has as many components as the number of grid points. Finally, for each target variable and horizon forecast, the average skill is calculated over the whole period of the challenge and the results are compared to those of other models. -The first model is a Local Linear Regression with Multitask Feature Selection (MultiLLR), it has been trained on data from different sources that the authors collected in their SubseasonalRodeo Dataset, including lagged measurements of temperature, precipitation, humidity, pressure, wind, Madden-Julian oscillation. “Local” means that the training data is restricted to measurements within a range of 56 days around the target date. As the importance of the features depends on the target date (for instance, high humidity level can be a better predictor for precipitation in winter months than in summer), backward feature selection is performed for each target date. Starting with all thefeatures in the model at step zero, then each step the feature that influence the predictive measurement of the cosine similarity the least, is removed, until when further removals that change the outcome predictive measurement within a tolerance range are no more possible. +The approach presented in the paper consists in combining two regression models, both of them based on "multitask feature selection", meaning that the variables for a target date are selected jointly - instead of selecting them independently for each grid point - while the regression coefficients are fit independently for each grid point. -The second model is a Multitask k-Nearest Neighbor Autoregression (AutoKNN). It uses as features only historical measurements of the target variables from l days prior to the target date, with l taking three fixed values, plus the constant intercept (with value equal 1 for all datapoints) and the observed anomalies of the k most similar neighbors to the target date. The most similar neighbor is identified as the candidate date, within a period of one year prior to the target date, whose anomaly vector is as similar as possible to the target date's anomaly vector. The authors include k=20 most similar neighbors when predicting temperatures, for a total of 24 features, while k=1 when predicting precipitations (5 features in total in total). This difference can be explained by the fact that when predicting precipitation, the top neighbors for a target date are shown to be from the same time of year as the target date, while for temperature the top neighbors are drawn from throughout the year, regardless of the month of the target date. The authors show corresponding plots. +The first model, the Local Linear Regression with Multitask Feature Selection (MultiLLR), has been trained on data from different sources that the authors collected in their SubseasonalRodeo Dataset, including lagged measurements of temperature, precipitation, humidity, pressure, wind, Madden-Julian oscillation. "Local" means that the training data is restricted to measurements within a range of *56* days around the target date. Since a feature can be more or less important depending on the target date (for instance, high humidity level can be a better predictor for precipitation in winter months than in summer), backward feature selection is performed for each target date. Starting with all features in the model at step zero, at each following step the feature that affect the cosine similarity the least, is removed, until when further removals are no more possible without changing the outcome of the predictive measurement beyond a tolerance value controlled by external input. -Finally, the two models are combined and the average of their l2-normalized predicted anomalies is defined as the ensemble anomaly. The authors motivate this choice by showing the mathematical proof that the so-defined ensemble anomaly is guaranteed to produce a better skill than the average of the skills of the individual models. +The second model, the Multitask k-Nearest Neighbor Autoregression (AutoKNN), uses as features only historical measurements of the target variables from *l* days prior to the target date, with *l* taking three fixed values, plus a constant feature and the observed anomalies of the *k* most similar neighbors to the target date. The most similar neighbor is identified as the candidate date, within a period of one year prior to the target date, whose anomaly vector is as similar as possible to the target date's anomaly vector. The authors include *k=20* most similar neighbors when predicting temperatures, for a total of 24 features, while *k=1* when predicting precipitations (5 features in total). This difference can be explained by the fact that when predicting precipitation, the top neighbors for a target date are shown to be from the same time of year as the target date, while for temperature the top neighbors are drawn from throughout the year, regardless of the month of the target date. The authors show this difference in plots. -The individual models and the esemble model are evaluated over the contest period and also over each year in the period 2011-2018, and they are shown to outperform both the benchmark models provided by the organizers of the challenge and the top competitor. +Finally, the two models are combined and the average of their *l2*-normalized predicted anomalies is defined as the ensemble anomaly. The authors motivate this choice by showing the mathematical proof that the so-defined ensemble anomaly is guaranteed to produce a better skill than the average of the skills of the individual models. -Put your summary of the paper here +The individual models and the ensemble model are evaluated over the contest period and also over each year in the period 2011-2018, and they are shown to outperform both the benchmark models provided by the organizers of the challenge and the top competitor. ### Strengths The study addresses the problem of forecasting on a time scale which has been defined “predictability desert” for the lack of models and methods that give good predictions. The models used in this paper are conceptually simple as they use linear regression. -The authors have collected data from many different sources, realising a dataset for each target variable and each time range. +The authors have collected data from many different sources, releasing a dataset for each target variable and each time range. ### Weaknesses and limitations @@ -30,7 +28,7 @@ The authors have collected data from many different sources, realising a dataset Besides considering the exact average of the two models, it would have been interesting to consider a weighted average of them when calculating the ensemble anomaly, and varying the relative weight to find the optimal ensemble skill. For instance, I believe that privileging the AutoKNN model would give a better mean skill for precipitation at 3-4 weeks. ### Implications -Societal implications: Improvement of subseasonal forecasts can lead to better management of water resources, agriculture processes, food prodcution, it helps to reduce the risks associated to extrem weather events +Societal implications: Improvement of subseasonal forecasts can lead to better management of water resources, agriculture processes, food production, reduction of the risks associated to extreme weather events. Research implications: following this study, other multi-model combinations can be tried and the SubseasonalRodeo dataset can be used as a benchmark to train machine-learning models.