A lack of high-resolution observational data is one of the obstacles that limits the advance of numerical weather prediction (Bauer et al., 2015). This limitation can be extended to all areas in atmospheric research. In recent years, many new observational technologies have emerged, including built-in smartphone sensors, such as those for pressure, temperature, humidity and aerosols (Overeem et al., 2013; Snik et al., 2014; Muller et al., 2015; Droste et al., 2017; Meier et al., 2017; Zheng et al., 2018). With over 2.7 billion people in possession of smartphones (Bankmycell, 2019) and an increasing trend in equipping smartphones with atmospheric measurement sensors, smartphone data can potentially be an auxiliary resource for global, high-density observations capable of resolving convective-scale features with a resolution lower than 2 km (Mass and Madaus, 2014).

The smartphone sensors monitor atmospheric parameters and convert them into electrical signals which can then be collected by different platforms, such as mobile weather applications. Low-cost smartphone sensor data have been used in several atmospheric research studies. Overeem et al. (2013) and Droste et al. (2017) used smartphone battery data to study air temperature and their application to urban heat islands. Snik et al. (2014) mapped atmospheric aerosols using smartphone spectropolarimeters. Surface pressure is one of the most useful variables because it can reflect information about the whole atmospheric column and is less sensitive to the observational background (e.g., indoors/outdoors or the influence of the underlying surface versus other variables like temperature and wind; Mass and Madaus, 2014; Hanson, 2016); therefore, smartphone pressure data have received considerable attention from researchers. In addition to applications in weather forecasting (Mass and Madaus, 2014; Madaus and Mass, 2017; McNicholas and Mass, 2018b; Hintz et al., 2019), smartphone pressure data can be used to monitor atmospheric tides (Price et al., 2018).

While smartphone pressure data may have potential value, they require validation and quality control before use. Price et al. (2018) and Hintz et al. (2019) showed that, although the variability between smartphone pressure data and meteorological-station observations is highly correlated, there exists noticeable bias. Price et al. (2018) calibrated the long-term stable bias using a one-point calibration method, while Hintz et al. (2019) developed screening methods to reduce observational noise. Machine learning has also been applied to correct atmospheric pressure data (Kim et al., 2015, 2016; McNicholas and Mass, 2018a). Most previous publications on smartphone data calibration adopted a user-based approach which required the identification of each unique user and personal information. However, this raises privacy and ethical issues that pose a concern to the public. As highlighted by Muller et al. (2015) and Mooney et al. (2017), collecting as little personal information as possible and keeping raw data private are guiding principles of privacy preservation. Moreover, without a stable data collocation platform, performing user-based calibration can be time and resource consuming, especially for densely populated regions. It is therefore imperative to develop a new method that can efficiently calibrate smartphone pressure bias while protecting user privacy. It is worth noting that the need for such an effort has been recognized by other researchers, and similar efforts are being undertaken (McNicholas, 2020).

China has one of the world's most densely distributed smartphone user bases (Bankmycell, 2019) which can potentially produce highly dense observations. In this paper, we present, for the first time, a year-long dense and extensive smartphone dataset collected by the Moji Weather app, which is developed and operated by the internet environmental meteorological corporation Moji. The Moji Weather app is a popular smartphone weather app used in many countries with a 53.90 % market share and more than 500 million users, as well as over 100 million weather queries made every day (Moji, 2019a, b). In the present study, we use the Moji smartphone pressure data for all of 2016 to show the spatial and temporal distribution of the dataset. With this highly dense network, we demonstrate the feasibility of a new machine-learning bias-correction method that does not require users' private information, thereby ameliorating ethical issues. The dense network also makes it possible to study the detailed structure of atmospheric convection, which is demonstrated in this study by applying the bias-corrected data to the fine-scale analysis of a hailstorm that occurred in Beijing.

This paper is organized as follows. Sect. 2 describes the data and methods used in our research. The statistical characteristics of this dataset, bias-correction results and its application to a hailstorm case are presented in Sects. 3, 4 and 5, respectively. Conclusions and discussion are given in the final section.

Three neighborhood-based bias-correction experiments, each using one of the aforementioned machine-learning methods, were conducted on a domain covering Beijing and its surrounding area from May to August 2016. The machine-learning bias correction was performed in each of the subdomains in Fig. 8c using surface observations as the truth and the smartphone input parameters listed in Table 1. The region was affected by the 10 June 2016 hailstorm and had a high density of smartphone pressure observations (Fig. 7a–d).

Constrained by the requirements of adequate data samples and reasonable computation cost, we chose 16 subdomains of 0.25∘ (longitude) × 0.20∘ (latitude) in size. The pressure time series from two representative stations in Fig. 8a and b show that, although the trend in the weather station and smartphone is consistent, bias is clearly present, which is consistent with the results of Price et al. (2018) and Hintz et al. (2019).

Figure 9 shows the mean absolute error (MAE) and computation time at different training regions for the three methods; it is evident that the RF method is more accurate and time saving. The computation times for subdomain 2 and subdomain 6 using the SMO method are more than 9 h. From this comparison, we have found that the RF algorithm is more suitable for the neighborhood-based bias correction of smartphone observations without requiring users' personal information. Furthermore, we discovered that the random data separation into training set and test set can cause random errors in the bias-corrected data; hence, in order to eliminate these errors, the correction procedure was repeated for 50 times to generate an ensemble result.

Collecting smartphone data through a weather app is convenient and common; however, the approach relies on the loyalty of users. Calibrating smartphone pressure individually can be only applied to data from long-term users, but it cannot be used for recently added users. In contrast, performing data correction for the aggregated data in a 0.0001∘ × 0.0001∘ grid box in a subdomain makes it possible to collect data from both user groups. In order to evaluate the applicability of our method on data from both types of users, we define the data sites appearing in both the training set and test set as stable sites and those only appearing in the test set as additional sites. To quantify the performance of bias correction, the domain-average MAE and standard deviation of ensemble mean for the 16 subdomains are displayed in Fig. 10 for the raw and bias-corrected data from both the stable sites and the additional sites. The MAE was calculated using data from the smartphone sites for each subdomain. Comparing the MAEs between the raw (Fig. 10a) and bias-corrected data (Fig. 10c), it is evident that the neighborhood-based bias-correction method is capable of substantially reducing the MAE not only for the stable sites but also for the additional sites with slightly more reduction for the stable sites (from 5.95 to 0.53 hPa) than for the additional sites (from 5.90 to 0.99 hPa). It is also shown that the method reduces the MAE spread by 78 % for the stable sites and by 16 % for the additional sites (Fig. 10b, d). A lower MAE and spread reduction for the additional sites is not surprising because they are newly added data with shorter data history and hence have fewer data samples (Fig. 11a, b). Encouragingly, our results suggest that the neighborhood-based method can partially mitigate the difficulty related to recently added data with shorter data history. In comparison with the bias-correction method based on a single site, the neighborhood-based method resulted in a substantially smaller MAE (see Fig. 11c, d).

High-density pressure observations can potentially help identify small-scale surface pressure patterns beneath a thunderstorm (Johnson and Hamilton, 1988). Although the quality-controlled gridded smartphone pressure data reduce the number of data points, they are still adequate to represent the fine-scale pressure patterns. In this section, we first show what small-scale information the quality-controlled high-density pressure data at the smartphone sites (with a spatial resolution of 0.0001∘ or approximately 10 m) can provide and then demonstrate the impact of the smartphone data on the gridded 1 km pressure analysis that is obtained using the objective analysis method described in Sect. 2.4.

Figure 12 shows a composite plot of radar reflectivity, pressure changes calculated from surface weather station observations and from smartphone data, and wind and equivalent potential temperature from the station observations. To be consistent with the time interval of the radar volume scan, the smartphone QC-data averaged every 6 min were used to generate the 6 min pressure tendency. Further, because the weather station data are at a 5 min interval, the pressure change and temperature from these data are shown at times closest to those of the radar volume scan. Since there are only 15 weather stations providing pressure observations in this region, they are unable to locate the leading edge of the cold pool. In contrast, the smartphone pressure observations are much denser and hence are able to capture the fine-scale pressure change associated with the cold pool, as depicted in Fig. 12 by the “×” symbol representing the 6 min change in perturbation pressure (i.e., domain mean subtracted) greater than 0.52 mb. Compared with the cold pool leading edge identified by θe, following Schlemmer and Hohenegger (2014), from the analysis of surface observations, the leading edge of the cold pool based on the smartphone pressure change is about 10 km ahead at 15:06 LST (Fig. 12b) and quite close at 15:24 LST (Fig. 12c). At 14:54 LST (Fig. 12a), the pressure change is largely negative ahead of the cold pool, whereas Fig. 12d mainly shows negative pressure changes after the leading edge has passed the area; both are consistent with the surface station observations but are more detailed.

We conducted three objective analysis experiments using the method described in Sect. 2.4 to demonstrate the potential benefit of using smartphone observations along with surface weather station observations to improve surface pressure analyses, i.e., the station observation experiment (SFC) using only weather station pressure observations, smartphone experiment (SP) using only smartphone data, and SFC + SP using both the station and smartphone data. The analysis grid spacing is 1 km. Figure 13 shows the domain for surface analysis and the locations of the Beijing radar and surface stations.

The analyses of perturbation pressure (i.e., relative to domain mean) from the experiments SFC (Fig. 14a, c, e) and SFC + SP (Fig. 14b, d, f) are compared at 15:00, 15:06 and 15:12 LST in Fig. 14. To illustrate the coupling between pressure and wind in the storm region, the wind field at 150 m from VDRAS (Variational Doppler Radar Analysis System) and the composite reflectivity observation are overlaid. VDRAS is a rapid update analysis system based on the variational technique that blends radar radial velocity and surface wind observations to produce 3D wind analyses (Sun and Crook, 1997, 1998). We first note that the perturbation pressure analysis from SFC + SP (right column) displays small-scale features in and around the storm that are absent in SFC (left column). The high center of pressure perturbation is nearly collocated with the center of the outflow near the northwest flank of the main body of the storm system (Fig. 14b, d, f). The vertical cross sections shown in Fig. 15 through the line A–B (see Fig. 14) indicate that the high-pressure perturbation corresponds to the rear-flank downdraft aloft behind the intense radar echoes of the southeastward moving convective system. Although the relatively low-pressure regions are seen in front of the convective system in both experiments, only the SFC + SP experiment captures the relatively low-pressure region northwest of the system. The overall distribution pattern of pressure perturbation in SFC + SP is consistent with the conceptual model of Markowski and Richardson (2010), but the current analysis reveals that the surface high-pressure region and low-level divergence center slightly lag behind the center of the intense reflectivity echoes rather than right beneath it, as in their conceptual model. We believe the difference results from the higher resolution of the smartphone data applied in this study, but further studies are needed to draw a definite conclusion. Furthermore, the pressure analysis from SFC + SP provides more detailed information about storm evolution than what is shown in SFC. As the storm moves southeastward, the cell in the southwest, denoted as cell 2 in Fig. 14, separates into two (Fig. 14b), and the northern one merges into cell 1 (Fig. 14d, f). During the merging process, the high-pressure region behind cell 1 becomes stronger and wider, which may indicate the enhancement of cell 1 in correspondence with the increased downdraft and updraft, as shown in Fig. 15c.

Analysis accuracy for the two experiments was verified against the 15 weather station pressure measurements in the domain. In order to avoid dependence between the analysis and verification, both experiments were repeated 15 times; each alternately excludes the measurement from the specific station to be verified against. The temporal distributions of MAE between model analysis and observation at different surface stations are shown in Fig. 16. The results confirm that the experiment SFC + SP reduces the analysis error at most stations, even at those around which there are relatively fewer smartphone observations, such as the stations Xianghe (XH) and Langfang (LF). Although at the stations where there are much fewer smartphone observations, such as Guan (GA) and Zhuozhou (ZZ), the analysis with smartphone pressure data alone in the experiment SP results in a larger error than in the experiment SFC; adding the station observations in SFC + SP results in reduced analysis error (Fig. 16n, o). The correlation between the smartphone data density and the analysis accuracy is more clearly illustrated by Fig. 17, which shows that the MAE is less than 0.20 hPa as long as there are more than three smartphone sites around the verifying weather station measurement.

In summary, our quantitative verification results demonstrate that the high-resolution smartphone data generally improve surface pressure analysis in comparison with the weather station data; combining these two datasets results in a further improvement, especially at the locations where the smartphone data are sparse.

This study focused on smartphone pressure data acquired from the Moji Weather app in 2016 and showed their characteristics for the first time. A neighborhood-based bias-correction method applying machine-learning techniques was developed without any privacy information needed. The bias-corrected data were employed to explore the potential value of these data for improving atmospheric analysis through the case of a hailstorm in Beijing, China.

Since these data are produced by citizens at large, their spatial and temporal distributions are affected by human behavior. It was shown that the data are mostly distributed around urban areas, and data volume peaks during summer. There is also a diurnal cycle in which the data volume is higher during the day than at night, with two peaks appearing at 07:00 and 18:00 LST. Our case study showed an anomalous increase in data volume when the hailstorm occurred, suggesting that public concern increases in anticipation of high-impact weather situations, which means the data can be useful for disaster prevention.

We proposed and demonstrated a neighborhood-based bias-correction method that can address user privacy issues. Despite growing concern from the public regarding personal privacy, few studies have addressed how to circumvent the problem. Since Moji protects data privacy during the collection and processing stages, no private information was included in the raw data that we received, and the bias-correction method proposed in this study does not require such information. Our results showed that the MAE and MAE spread can be successfully reduced not only for long-term stable sites but also for recently added sites that present a challenge using the traditional user-based bias-correction method.

With this feasible and effective bias-correction method, the potential utility of the high-resolution smartphone data (approximately 10 m horizontal resolution) is shown using a hailstorm case. We have found that the 6 min pressure change can provide convective-scale information such as cold pool leading edge, especially in megacities where the data are most dense. Using a modified Barnes objective analysis method on a 1 km grid, we also showed that the data can be used in conjunction with weather station data to improve surface pressure analysis. The analysis is capable of depicting the high pressure associated with the rear-flank downdraft of the hailstorm and temporal variation in pressure perturbation related to the splitting and merging process within the convective system.

Through the current study, we have gained an understanding of the smartphone pressure data characteristics, developed a practical and effective quality-control and bias-correction method, and demonstrated the value of the data in surface objective analysis; our next step is to explore whether the data can be useful in improving convective weather forecasting through data assimilation. Previous data assimilation research with smartphone pressure data mainly focused on assessing whether the data have a positive impact on regions where weather stations are not available (McNicholas and Mass, 2018b; Hintz et al., 2019). However, it may present a greater challenge to demonstrate that the smartphone data can yield additional benefits to the existing weather station network mainly because of the uneven distribution of the smartphone data across the globe. Efforts are needed to develop data assimilation approaches that can make best use of the smartphone data in numerical weather prediction models by taking into account the characteristics of these data. The current study also points to the need of an improved smartphone data collection mechanism. The data volume collected by a weather app relies heavily on the popularity of the application that serves as the data-collection platform (Kim et al., 2015; Hintz et al., 2019). As such, the data distribution relies heavily on the severity of local weather. Thus, a more stable and widely used platform is needed to provide useful high-resolution global observations without a correlation to local weather. Additionally, the smartphone information included in our research is limited; additional auxiliary information, such as smartphone models, sensor types and the altitude at which smartphone data were measured, would be conducive to the bias-correction procedure and subsequent analysis.