We use a data stream recorded at SMEAR II station in Hyytiälä, Finland (61∘50′51′′ N, 24∘17′41′′ E; 181 ma.s.l.), measuring relationships between the forest ecosystem and atmosphere. We use data covering a period of 7 years (April 2005–April 2013), recorded at every 30 min from 37 observation sensors. The raw data coming from the station have on average 7 % of missing values. Missing values may occur due to the occasional failure of measuring sensors, wear and tear, or variations in electricity power supply. Some data are missing up to 50 % of the time. There is no single sensor that would provide non-interrupted readings over those 5 years; for any sensor from 1 % (about 4 days per year) up to 25 % (3 months per year) values are missing.

The task is to nowcast the current level of solar radiation from the meteorological sensor data, given in Table . The incoming radiation to Earth is constant with the accuracy we require at a given day and hour of the year. The only unknown is the absorption to the atmosphere and, more importantly, to the clouds and anthropogenic pollution plumes. Hence, an interesting variable to infer is the cloudiness, or, in other words, the deviation of the measured radiation from the theoretical maximum. In this schema other meteorological parameters could be used to estimate the cloudiness, and this can further be used to calculate the actual radiation, but the primary variable to nowcast is the difference between the theoretical and actual radiation.

This nowcasting task would be relevant to the stations where no radiation measures are available. The station SMEAR II, from which the input data originate, is able to measure solar radiation; hence, the true values are present for us for evaluation purposes. However, instrumentation for measuring solar radiation is not always available. Small meteorological observation stations may not be able to have solar radiation measured, but it may be interesting to nowcast radiation from meteorological data that is available anyway.

In this study our target variable is defined as the ratio of the actual radiation to the theoretical maximum radiation. This gives a value between 0 and 100 %, where 100 % indicates that all the theoretically possible radiation is actually incoming. The sensor Global RADIATION (Table ) is not used as an input into the nowcasting model; it is only used for evaluating the nowcasting accuracy. It indicates the actual radiation and is used in forming the target variable.

The theoretical maximum radiation is calculated using MIDC (Measurement and Instrumentation Data Center) SOLPOS (Solar Position and Intensity) Calculator

http://www.nrel.gov/midc/solpos/solpos.html

There are different ways to estimate the regression parameters . Ordinary least squares (OLS) is a simple and probably the most common estimator. It minimizes the sum of squared residuals giving the following solution: β^OLS=arg⁡min⁡β(y-Xβ)T(y-Xβ)=(XTX)-1XTy. Having estimated a regression model β^, nowcasting on new data xnew can be made as y^=xnewβ^.

For a linear regression model, it is possible to determine theoretically how many missing inputs can be tolerated before model outputs become obsolete. We can estimate the robustness of a linear regression model to potentially missing input data by using the deterioration index , which is defined as d=-βT(Σ-I)β, where β is a vector of the regression coefficients, assuming that the input variables have been standardized to zero mean and unit standard deviation; Σ is the covariance matrix of the input data; and I is the identity matrix. High values of the index d indicate low tolerance of the model to missing data. The prediction errors will increase quickly with the number of missing inputs. The smaller d, the more robust to missing data the model is. d may be negative; that is the best option.

Low d guarantees robustness to missing data, but the models with low d do not necessarily give good predictions when all the values are available. Hence, a tradeoff between accuracy and robustness needs to be found, and the following method can help to find it.

Suppose we get two models A and B, and we would like to select one for deployment. We can measure their prediction errors on a training data set using cross-validation: RMSE(A) and RMSE(B). We can also compute deterioration indices d(A) and d(B). Without loss of generality, assume that RMSE(A)≥RMSE(B); i.e., model B shows a better prediction accuracy when no data are missing. If d(A)≥d(B), then model B is also more robust. In such a case, model B is better (or at least as good as A) with regard to both characteristics, and hence B is preferred over A.

If, however, d(A)<d(B), then we can find out how many input readings can go missing before A becomes better than B. The number m∗ can be computed as m∗=(r-1)[RMSE(A)]2-[RMSE(B)]2d(B)-d(A), where r is the number of input sensors.

Firstly, we analyze in what way missing values occur in the case study data set. Figure a presents the distribution of missing sensors. We see that about half of the time nothing is missing and half of the time observation vectors are incomplete. Over 35 % of the time, 2–4 sensors are missing. The mean number of missing sensors over all the data set is 2.4. We observe from the data that up to 36 sensors (all the input sensors) may be missing at a time. From this analysis we conclude that the amount of missing data is at a massive scale and scope, and missing values needs to be taken into consideration when building nowcasting models on this data. The amount and frequency of missing data also indicates that a case deletion approach would not be suitable because there would be predictions missing continuously.

One may consider that removing from the data set one or two sensors with the largest amount of missing values could solve the problem. This could help if mostly the same sensors were missing all the time. In the following experiment we analyze in what way individual sensors are missing. First, we remove a sensor with the most missing values from the data set; this way the observation vectors at each 30 min time stamp become shorter, and they now include 35 sensors instead of 36. Given the updated observation vectors, we recalculate how many of those vectors contain at least one missing value. Then we remove the sensor lacking the next highest number of measurements and repeat the computation. Figure b presents the results. We see that removing a couple of largely missing sensors does not make the remaining observations complete. We would need to remove about half of the sensors in order to reach the stage where at least 95 % of the data are complete. The problem with such an approach is that the removed sensors may carry important information about the target, which then would be lost. To investigate this effect, Fig. c presents the relation between the missing-data rate in each sensor and the information about the target contained in it, measured as the absolute linear correlation with the target variable. We have removed the periods where the value of the target is equal to 0 (the dark periods when there is no solar radiation) from this analysis. We see that some sensors in the far right corner and upper center have a high missing-value rate but also high correlation with the target variable. This means that excluding sensors with high missing-value rates would lead to losses of valuable information about the target that would be useful for nowcasting.

One more issue with the data is that sensors do not produce missing values independently of each other. For example, if one temperature value is missing, then it is likely that the other temperature values are missing as well. It may be the case that sensors are missing together due to some common external reasons, for instance, electric power outages. This observation is illustrated by Fig. , which plots pairwise correlations between missing values for different sensors. Sensors that are often missing together are encoded in black (dark). We see that, in particular, temperatures (T), relative humidity (RH), visibility, and precipitation readings are often missing together. This means that we cannot rely on the redundancy of the sensors such that, if, say, a temperature reading is missing at 33 m, we can use the reading at 50 m. Both readings would often be missing together.

Finally, in many cases the average duration of missing values lasts for several hours. Figure  presents the average duration of missing values in the case study data set for each sensor. Since values may be missing for extended periods of time, we also cannot, from this perspective, simply discard data with missing values, since in such cases we would often not have model outputs for extended periods of time.

In summary, the amount of missing data is very large, and at this level data with missing sensors cannot be discarded without losing valuable information. Missing values are strongly correlated with each other; this makes it difficult and, in many cases, impossible to make use of sensor redundancy or impute missing data based on non-missing data. Removing sensors with the most missing data is also not feasible since missing values are not concentrated in several sensors but are distributed across all the sensors and the sensors with a lot of missing values at the same time carry relatively strong information about the target at times when the values are not missing. Hence, the most appropriate solution to the problem of missing values in this setting appears to be building models that are robust to missing data. This approach is free from any assumptions about the missing data and allows nowcasting even when all or nearly all the sensors are missing.

Next we experimentally analyze accuracies of several linear regression models and their robustness to missing values. The first experiment demonstrates how we can select the best model for deployment. The second experiment presents evidence about the performance on unseen data.

Table  presents the errors of the regression models ALL, rALL, SEL, rSEL, PCA, rPCA, and PLS, measured on the training set using a fivefold cross-validation and deterioration index estimated based on the training set. For PCA, rPCA, and PLS the number of components was fixed to k=18, which is a half of the original number of input sensors and explains 99 % of the variance. The cumulative percent variance method was used for selecting k, which is recommended as one of the most reliable methods in the literature . Figure  in the Appendix provides complementary information about the variance explained by PCA components. Later in this section we will present a sensitivity analysis to different values of k.

This analysis is performed from the perspective of an analyst, making a decision on which model to deploy. Cross-validation is used to avoid potential overfitting of the model parameters to the training data. Complementary information on the goodness of fit is presented in Appendix .

In the case of the analyst basing the decision only on the offline analysis of validation errors, he or she would select ALL for deployment since it gives the lowest error, while PCA and rPCA show nearly the highest error. However, the deterioration indexes computed for these models suggest the opposite: rPCA shows the best, while ALL shows the worst deterioration index value. The analyst can now theoretically compare the robustness of two models, for instance ALL and PCA, using the criteria from Eq. (), which gives m∗=(r-1)[RMSE(PCA)]2-[RMSE(ALL)]2d(ALL)-d(PCA)=(36-1)(21.8)2-(19.0)21 122 710-(-117)≈10-4. The result m∗=10-4 means that, if we expect at least one sensor reading to be missing in 10 000 observations, it is better to deploy rPCA than ALL. Recall that in the data about 2.4 sensors are missing on average in every observation. Hence, in this situation it is clearly worth deploying rPCA instead of a standard linear regression, even though the ordinary regression may be more accurate when no data is missing.

Let us consider the regularized version of ordinary regression rALL and rPCA. rALL shows better cross-validation accuracy than rPCA regarding the training data and not as bad a deterioration index as ALL. For rALL and rPCA, m∗=0.4, which means that it is still worth deploying rPCA.

The performance of PCA and rPCA seems very similar. For PCA and rPCA, m∗=13.1, which means that rPCA is expected to be more accurate than PCA if more than 13 sensors are missing; this would be quite pessimistic for our case study data, where the mean number of missing sensors is 2.4. Hence, the analysis suggests chosing PCA for deployment.

The following analysis simulates online operation after deployment. Regression models are trained on the training set, and then sequentially tested on the test set. Table  reports the testing results of the regression models ALL, rALL, SEL, rSEL, PCA, rPCA, and PLS (k=18).

The regularized principal component regression rPCA demonstrates the best performance on the test data (RMSE=19.49), closely followed by PCA without regularization (RMSE=19.52). The other regularized approaches, rSEL and rALL, perform notably worse (RMSE=20.43 and 20.28) but they still outperform the naive baseline NAI (RMSE=22.88). The unregularized approaches PLS, SEL, and ALL perform much worse than the baseline and illustrate well the dangers presented by massively missing values.

It is interesting to note that the analyzed strategy combining linear regression with mean replacement theoretically approaches NAI performance as more values go missing. If all the input values are missing, then the predictor turns into NAI automatically.

To analyze the performance further, we divide the test data into non-missing (44 %) and missing observations (56 %) and inspect the errors on these subsets separately. We see that the performance of all the models is similar when there is no missing data. The ordinary regression ALL has an advantage in accuracy, since it does not discard any information from the input data. However, the non-regularized models (ALL, SEL, and PLS) fail badly when there is missing data, while the regularized rSEL and rALL lose some accuracy but still remain competitive. Both non-regularized and regularized PCA remain nearly unaffected by missing data.

Figure  plots the distribution of absolute residuals for each approach. We can see that most of the errors (residuals) are concentrated around 10, which is a reasonably good result keeping in mind that the range of the target variable is from 0 to 100. It means that most of the predictions do not deviate too much from the true values. We can also see that NAI has less probability mass on the left-hand side, where the most accurate predictions are. As expected, intelligent predictors do better than NAI. Only the unregularized approaches ALL and SEL have any probability mass on the far right, which means that they occasionally produce predictions that may exceed the maximum of the true target. We can conclude from this investigation that predictions by most of the approaches are reasonably stable, and outliers in predictions do not pose any major threats.

Next, we analyze the sensitivity of the predictive performance to different parameter settings. So far we used a fixed number of components (k=18) for PCA, rPCA, and PLS and the same number of selected features for SEL and rSEL. Figure  shows the testing errors (RMSE) as a function of k.

An important observation can be made from this plot. The regularized approaches rSEL and rPCA perform reasonably well at all variants of the parameter k, while the non-regularized models SEL, PCA, and PLS perform poorly when a large number of components is retained. In such a case the resulting models are still similar to ALL, which uses all the available information. ALL, rALL, and NAI do not depend on the parameter k but are also included for comparison. We also observe that PLS becomes very effective at low k, but there is a risk of setting k incorrectly (e.g., around 25), in which case PLS gives the worst results. Therefore, we instead recommend using rPCA, which gives stable and accurate results even if k is suboptimal.

Finally, we visually analyze model outputs produced by the baseline approach ALL and a robust approach rPCA (k=18). Figure  plots four 3-day snapshots from the year 2012: 1–3 January, 1–3 April, 1–3 July, and 1–3 October. It is important to emphasize that, here, the plot shows raw outputs of the classifiers in order to better illustrate the effects of regularization, whereas, when calculating numerical errors, we postprocess all the model outputs to fall into the same interval as the original target ([0,100], where 0 means no irradiance is observed, and 100 (%) means all the theoretically possible irradiance is observed). That is, if the prediction is less than 0, we correct it to 0, and if the prediction is larger than 100, we correct it to 100. This postprocessing makes the baseline classifiers more competitive (and hence is a more prudent way of quantitative evaluation). We see from the figure that the baseline ALL sometimes fails very badly (particularly in the January and April plots), while the outputs of the regularized approach rPCA remain stable. In July there are only a couple situations when ALL shows very poor performance (we see a green inclination on day 2 and a green peak on day 3). In October both approaches perform similarly. Unlike ALL, rPCA perform in a stable manner and does not exhibit extreme failures.