Articles | Volume 7, issue 12
Atmos. Meas. Tech., 7, 4387–4399, 2014
https://doi.org/10.5194/amt-7-4387-2014
Atmos. Meas. Tech., 7, 4387–4399, 2014
https://doi.org/10.5194/amt-7-4387-2014

Research article 11 Dec 2014

Research article | 11 Dec 2014

Regression models tolerant to massively missing data: a case study in solar-radiation nowcasting

I. Žliobaitė1,2, J. Hollmén1,2, and H. Junninen3 I. Žliobaitė et al.
  • 1Aalto University, Department of Information and Computer Science, Espoo, Finland
  • 2Helsinki Institute for Information Technology (HIIT), Helsinki, Finland
  • 3Department of Physics, University of Helsinki, Helsinki, Finland

Abstract. Statistical models for environmental monitoring strongly rely on automatic data acquisition systems that use various physical sensors. Often, sensor readings are missing for extended periods of time, while model outputs need to be continuously available in real time. With a case study in solar-radiation nowcasting, we investigate how to deal with massively missing data (around 50% of the time some data are unavailable) in such situations. Our goal is to analyze characteristics of missing data and recommend a strategy for deploying regression models which would be robust to missing data in situations where data are massively missing. We are after one model that performs well at all times, with and without data gaps. Due to the need to provide instantaneous outputs with minimum energy consumption for computing in the data streaming setting, we dismiss computationally demanding data imputation methods and resort to a mean replacement, accompanied with a robust regression model. We use an established strategy for assessing different regression models and for determining how many missing sensor readings can be tolerated before model outputs become obsolete. We experimentally analyze the accuracies and robustness to missing data of seven linear regression models. We recommend using the regularized PCA regression with our established guideline in training regression models, which themselves are robust to missing data.

Download
Short summary
We present a case study in solar/radiation nowcasting using environmental sensor measurements as inputs. While some sensor readings may oftentimes be missing, predictions need to be output continuously in near real time. We are after linear regression models that would be robust to missing data, i.e., that would perform well with or without data gaps. We recommend using regularized a PCA regression with our established guidelines for building robust regression models.