Contemporary numerical weather prediction (NWP) models equipped with state-of-the-art data assimilation techniques have advanced to a level considered indispensable for many research applications. In addition to their importance in weather analysis and forecasting, the NWP models are crucial for climate studies. For instance, the atmospheric reanalysis projects led by NWP centers (e.g., Kalnay et al., 1996; Kanamitsu et al., 2002; Uppala et al., 2005; Onogi et al., 2007; Saha et al., 2010; Dee et al., 2011; Ebita et al., 2011; Compo et al., 2011) provide data sets useful for a broad range of applications by synthesizing observations from diverse sources and a priori knowledge through data assimilation techniques. The reanalysis products are particularly valuable where observations are insufficient in number and accuracy to provide a good estimate of atmospheric states. However, reanalysis products are susceptible to deficiencies of the observations, showing in some cases very obvious and unphysical time-varying biases (Trenberth et al., 2001; Marshall, 2003; Bengtsson et al. 2004; Sterl, 2004; Renwick, 2004; Karl et al., 2006; Graversen et al., 2008; Reichler and Kim, 2008; Thorne and Vose, 2010; Screen and Simmonds, 2011).

Despite their importance, NWP data are far from perfect. Today, operational NWP centers apply bias corrections to satellite data judging against the model's own state at the time of assimilation (Dee and Uppala, 2009). Because not all satellite platforms possess obvious systematic errors that are discrete and exceed the model's uncertainty, the bias detection may at times become ambiguous. The model-based bias correction is challenging because of the strong feedback among observations and model states. That is to say, biased observations lead to a biased analysis, which in turn favors the observations that are biased similarly to the model states. NWP and reanalysis products are also known to have greater uncertainties in regions where satellite observations predominate, compared to regions with plentiful radiosonde observations (e.g., Langland et al., 2008). This underscores the importance of bias-free observations that can counteract the model's systematic error and act as anchor points for the bias correction.

Besides the bias leaking from observations, NWP models themselves produce considerable systematic errors due to shortcomings in the governing equations, numerics, surface forcing, and parameterizations of unresolved physical processes (Saha, 1992; Larson et al., 2001; Trenberth and Stepaniak, 2002; Danforth et al., 2007; Mass et al., 2008; Wee et al., 2012). The presence of large biases in the assimilating model causes spurious shifts and other artifacts in the analysis, even if all assimilated observations are unbiased and correctly represented by the assimilation system (Kobayashi et al., 2009). Unless adequately addressed, the systematic error in NWP data curtails the effectiveness of the bias correction, leading to an under-utilization of observations. It also negatively affects model-based homogenization of climate data records and has the risk of misrepresenting the climate change signal that the observations are bearing. In addition, lessening systematic NWP error is crucial because NWP models and reanalyses drive changes in the climate models that are going to be used for future projections of climate change (Folland et al., 2001). However, identifying and rectifying NWP errors is demanding. This is mainly due to the lack of accurate and independent observations that can be used for the verification because NWP centers are so eager to assimilate all good-quality observations.

Observations underpin all areas of numerical modeling, weather analysis and forecasting, and climate monitoring and projections that are closely relevant to each other. Thus, the availability of high-quality observations is of the utmost importance, carrying broad socioeconomic implications. Recently, Global Positioning System (GPS) radio occultation (RO) (Melbourne et al., 1994; Ware et al., 1996; Kursinski et al., 1997; Anthes et al., 2008) has been receiving a great deal of attention as a promising source of data for both weather and climate applications. The primary observable of RO is the phase path of GPS signals received by an accurate receiver onboard a low Earth orbiting (LEO) satellite. By analyzing the time–frequency content in the occulted signals, a profile of the ray's bending angle and subsequent profiles of atmospheric refractivity, pressure, and temperature can be derived. In past decades, numerous studies have demonstrated the unique strengths of GPS RO, which include high accuracy and vertical resolution, global coverage, all-weather capability, and self-calibration aptitude (e.g., Kursinski et al., 1997; Hajj et al., 2002; Wickert et al., 2004; Kuo et al., 2004). The data are accepted as an operationally reliable source of information by NWP centers worldwide (Poli et al., 2010), and have shown clear positive impacts on weather forecasting (e.g., Healy, 2008; Buontempo et al., 2008; Cucurull and Derber, 2008; Aparicio et al., 2009; Rennie, 2010) and merits in atmospheric reanalysis projects (Saha et al., 2010; Dee et al., 2011). In particular, RO data are assimilated without any bias correction. RO data offer a great potential for weather and climate research (e.g., Kursinski et al., 1997; Anthes et al., 2000; Hajj et al., 2000; Ladstädter et al., 2011), and are recognized as a promising contribution to the climate data record (GCOS, 2010, 2011).

Numerous studies besides the aforementioned confirm that RO data are of high quality. However, RO data possess retrieval errors (occurring in the course of imperfect atmospheric data processing) as well as measurement errors (those in the primary observable). A number of studies analyzed the error sources and propagation of the errors through the retrieval process (e.g., Kursinski et al., 1997; Feng and Herman, 1999; Syndergaard, 1999; Rieder and Kirchengast, 2001; Hajj et al., 2002; Kuo et al., 2004; Steiner and Kirchengast, 2005). Some of the error sources are particularly hard to tackle including ionospheric residuals (e.g, Syndergaard, 2000), the influence of a priori error on the statistical optimization of bending angle (e.g., Wee and Kuo, 2014), horizontal inhomogeneity in the atmosphere (e.g., Poli and Joiner, 2004), abnormal propagations of radio waves (e.g., Sokolovskiy, 2003), strong multipath effects in the lower troposphere (e.g., Gorbunov et al., 2006), and the ambiguity in separating moisture and dry air density from the bending angle or the refractivity (e.g., Kursinski et al., 1997). Different approaches that aim to reduce the errors are proposed in the literature; however, they yield slightly different results. This in turn leads to the structural uncertainty (Thorne et al., 2005) in the RO data. Intercomparisons of RO data sets processed by different centers worldwide (von Engeln, 2006; Ho et al., 2009, 2012; Steiner et al., 2013) showed that inter-center differences increase rapidly with height above 25 km especially for more derived variables. The elevated disparity in high altitudes is attributed to different noise regularizations employed by the centers (Steiner et al., 2013). Needless to say, differences in other parts of the retrieval process also contribute to the uncertainty.

GPS RO offers a hierarchy of data products through the chain of atmospheric data processing, ranging from the measurement of carrier phase to derived parameters such as the temperature. Data close to raw measurement are relatively simple in the error structure, but are difficult to model or interpret. While derived RO data have close geophysical relevance, they are complicated in the error characteristics due to retrieval errors and the propagation of errors through the retrieval process. In particular, Abel inversion and hydrostatic integration are strong error propagators, causing the errors in the derived RO data to be widely correlated. In recent years, the RO technique has been evolving rapidly and in the future could provide solutions that can substantially reduce the above-mentioned retrieval errors. Therefore, retrieval errors can be considered to be largely temporary and partially independent of the intrinsic error of RO measurements. Nonetheless, the intricate sources of retrieval error remain challenging at the present moment. Meanwhile, a precise characterization of data quality is helpful for RO data users to build strong confidence in the technique (GCOS, 2007; Hartmann et al., 2013). It is, however, not easy to assess the quality of derived RO data because of their complicated error characteristics (e.g., Wee et al., 2010; Gorbunov et al., 2011; Wee and Kuo, 2014). A way to overcome the difficulty is to evaluate the quality of minimally processed data rather than those of derived RO data. By doing so, the uncertainty in the derived RO data due to imperfections in the retrieval process can be avoided. Provided that retrieval errors are additive, the uncertainty in RO data increases as the retrieval process proceeds. Therefore, the minimally processed data may give insights into the intact competence of the RO technique, possessing the minimal data uncertainty that is attainable for derived RO data with improved (if not perfect) retrieval algorithms yet to come.

The atmospheric contribution to measured phase path (i.e., excess phase) is the primary observable of the RO technique and is customarily considered as the starting point in the sequence of atmospheric data processing. It is therefore appropriate to regard the phase path as the minimally processed data (hereafter simply referred to as measurement, in contrast to derived RO data or retrievals) amongst all data types for which geophysical interpretation is possible. The error of the phase path (measurement error hereafter) depends on many factors such as thermal noise, clock instability, local multipath, receiver performance, observational geometry, and atmospheric condition (Kursinski et al., 1997; Hajj et al., 2002). Depending on the standpoint, errors in orbit ephemerides and those in ionospheric correction can be also considered as a part of the measurement error.

In studies to date, the quality of phase measurements for RO is conceived through theoretic considerations or assessed in instrument-level perspectives (e.g., Kursinski et al., 1997; Hajj et al., 2002). Some sources of the measurement error are highly dynamic (e.g., the local multipath, observational geometry, and atmospheric conditions), changing significantly from one occultation event to another. So, theoretical approaches focusing on few scenarios of probable conditions (e.g., Kursinski et al., 1997) might be insufficient for representing the delicate dynamics. This compels studies on actual phase measurements that are sampled under diverse atmospheric conditions. Previous studies showed that random components of the measurement error can be estimated dynamically based on a signal's spectral contents (Hocke et al., 1999; Gorbunov et al., 2006) or in terms of measured signal-to-noise ratio (SNR) (Lohmann, 2007; Wee and Kuo, 2013). However, typical quality indicators relating to inter-sample variations, e.g., SNR and Allan deviation (Allan, 1966), give not much information about systematic errors. Therefore, quality assessments on real-world phase measurements by comparing them with independent, correlative data sets will be complementary to the theoretical approaches and dynamic error estimations. Once available, the comparison-based estimate of measurement error will be valuable for practical RO applications, such as data assimilations and uncertainty-based retrieval schemes.

In this study, measured phase paths are compared to their model counterparts. The essential prerequisite to do so is establishing a realistic modeling environment that includes effective forward observation operators and accurate specification of input atmospheric parameters. In order to draw meaningful conclusions that are valid for diverse atmospheric conditions, a large number of actual occultation events are used. By looking into the primary observable directly, our approach has the advantage of avoiding most complications and uncertainties pertinent to atmospheric data processing. Reliable detection of NWP errors is known to be challenging. The focus of our study is thus to explore whether measured phase paths are accurate enough to discern deficiencies in NWP data. This may also offer another angle to grasp the factual capability and limitation of the RO technique. Knowledge of the quality of phase paths is essential for understanding the behavior of retrievals and interpreting their comparison with correlative data. In the followings, the methodology used in this study is described, and then key findings are presented, and finally a summary and concluding remarks are given.

The GPS RO data used in this study are obtained from CHAllenging Minisatellite Payload (CHAMP) and Satélite de Aplicaciones Cientificas-C (SAC-C) missions, and processed by the Data Analysis and Archive Center (CDAAC) of the Constellation Observing System for Meteorology, Ionosphere, and Climate (COSMIC) at the University Corporation for Atmospheric Research (UCAR). The CDAAC data-processing algorithms and procedures are described by Kuo et al. (2004) and Schreiner et al. (2011). The data type used here is the excess phase path measured at two GPS L-band frequencies, f1=1.57542 GHz (L1) and f2=1.2276 GHz (L2). The excess phase ΔΦ can be modeled as follows: ΔΦ+ζ=∫GPSLEOnds=∫GPSLEO1+10-6k1pT+k2pwT2-k3nef2ds, where ζ is the range between the transmitter (denoted as GPS) and the receiver (LEO); n is the refractive index in the atmosphere, along the ray path ds, including both neutral atmospheric and ionospheric contributions; T is temperature in K; p is (total) pressure in hPa; pw is water vapor pressure in hPa; ne is electron number per cubic meter; f is GPS carrier frequency in Hz; and, k1=77.6 hPa K-1, k2=3.73×105 K2 hPa-1, and k3=4.03×107 m3 s-2 are coefficients. The neutral atmospheric refractivity used in this study is after Smith and Weintraub (1953). Some studies considered different refractivity formulae (Aparicio et al., 2009; Cucurull, 2010; Healy, 2011). The difference among them (e.g., see Rüeger (2002), and Aparicio and Laroche (2011)) is, however, considerably smaller than the typical size of NWP error. The contribution of electron density to the refractivity shown in Eq. () is the first-order expansion of the Appleton–Hartree formula (Budden, 1985) and the remaining higher-order dispersion terms contribute less than 0.1 % in general (Seeber, 1993; Bassiri and Hajj, 1993; Fritsche et al., 2005). The effect of omitted higher-order terms is larger under solar-maximum daytime conditions. Nonetheless, it is much smaller than the uncertainty of a priori electron density used in this study.

Provided a proper field of the refractive index, appropriate observation operators permit the modeling of phase paths. The inverse operators of standard RO data-processing algorithms (i.e., the methods of geometrical optics and wave optics) are well-suited for the purpose. However, they are based on the assumption of spherical symmetry in the refractive index and thus are unable to take into account horizontal inhomogeneity in the atmosphere, which would be a major source of error in modeling the measurement. Ray tracing (e.g., Høeg et al., 1995; Kirchengast, 1998; Healy, 2001; Gorbunov and Kornblueh, 2003; Liu and Zou, 2003; Poli and Joiner, 2004; Wee et al., 2010; Foelsche et al., 2011) on the other hand has the potential to properly model bending angles or phase paths accounting for horizontal variations in the atmosphere. The main obstacle in using ray tracers is computational cost, especially when rays are traced along the full range of GPS–LEO radio links. Ray tracing proceeds by recursively determining ray's direction along the ray path. This requires repetitive evaluations of refractivity gradient along the path. If an exact analytic model of the refractivity exists, a smaller step size for the ray integration will reduce the error in the estimated refractivity gradient and eventually lead to an improved solution of the ray tracing. In practice, however, no such comprehensive analytic model encompassing the whole atmosphere, including the ionosphere, is available for RO applications. Typically, global NWP models provide the refractivity on their grids that are discrete and limited in the resolution. Since the error due to repeated interpolations across the resolution-limited grid points while evaluating the refractivity gradient is non-negligible, a step size considerably smaller than the models' resolution does not necessarily yield a more precise solution for the ray tracing.

In this study, the curved ray tracer (CRT) developed by Wee et al. (2010) is used. The CRT is a rigorous ray tracing method that cuts down the computational cost drastically, without compromising the quality of the solution. The essence of CRT is the parameterization of curved ray paths with a series of osculating circles, based on the observation that the ray's curvature varies slowly and smoothly in the atmosphere, not exceeding the curvature of the Earth unless super-refraction occurs. Our comparison shows for a given accuracy tolerance that CRT allows considerably larger step sizes compared to a conventional ray tracer that is implemented by the authors. Another strength of CRT is that its solution is less sensitive to the step size, meaning that the parameterized curves follow actual ray paths very closely, even for a larger step size. The advantage of CRT is the most notable around the tangent point, where a ray experiences the greatest refractivity gradient. This is where conventional ray tracers suffer the most, requiring a very small step size. On the contrary, the ray's curvature there does not change substantially, allowing CRT to use a larger step size (Wee et al., 2010). The effectiveness of CRT in turn allowed us to perform ray tracings for a large number of occultation events. In this study, a convergence tolerance of 1 mm is used for ray shootings. The ray shooting is the multiple iterative end-to-end tracings of a ray for each epoch to realize the observed GPS–LEO link. The importance of ray shooting must be addressed because it is what makes the modeling of phase path particularly meaningful. Only through the ray shooting for the complete link between the transmitter and the receiver can the measured full phase path be closely replicated. Otherwise, the modeling of the phase path may not bring much benefit additional to the modeling of bending angle. Correct determination of the ray's direction along the ray path is a prerequisite for a successful ray tracing. Therefore, a modeled bending angle (i.e., the net change in the ray's direction between GPS and LEO) is readily available once the ray tracing is completed. On the other hand, extra modeling (e.g., a representation of the complete ray path by linking the locations and directions of the ray that are settled at individual integration steps, and a path-following integration of the refractive index) is necessary for the ray tracer to provide an estimate of the phase path.

Ray tracing requires the information about spatial variations in the atmosphere along the ray path. In this study, the refractive index from the Earth's surface up to the height of GPS satellites (∼ 20 200 km) is provided. The operational analysis and the 45-year reanalysis (known as ERA40) (Uppala et al., 2005) of the European Centre for Medium-Range Weather Forecasts (ECMWF) (OP and RA hereafter) furnish the refractive index for the lower neutral atmosphere. The OP used here is a reduced-resolution version, T106 in spherical harmonics (1.125∘ at the equator), of the original data (T511) on 26 constant pressure levels from the surface to 1 hPa (∼ 48 km). The resolution of RA is T159 (0.75∘) on 60 model levels with its top at 0.1 hPa (∼ 65 km). Both OP and RA are available every 6 h. The neutral atmosphere above the top of the ECMWF data and up to 200 km is extended with an empirical model, MSIS (Mass spectrometer and incoherent scatter radar) (Hedin, 1991; Picone et al., 2002). The International Reference Ionosphere (IRI) (Bilitza, 2001) and the Russian Standard Model of Ionosphere (SMI) (Chasovitin et al., 1998) are used to provide the electron density in the ionosphere and in the plasmasphere, respectively. The SMI is forced to fit the IRI-produced peak height (hmF2) and peak electron density (NmF2), and so the electron density is determined by the IRI below the hmF2 and by the SMI above. Although the electron density is lower in the plasmasphere than in the ionosphere, GPS signals travel a much longer distance through the plasmasphere. Therefore, the plasmasphere makes a considerable contribution to the total electron content (TEC) perceived at the receiver and in turn to the phase path. Observed historical values of sunspot number (Rz), geomagnetic amplitude index (Ap), and 10.7 cm solar radio flux (F10.7) are provided to the ionospheric models and to MSIS as input parameters. The combination of atmospheric and ionospheric models gives a complete description of the refractive index all along the entire GPS–LEO link. Readers are referred to Wee et al. (2010) for more details.

The RO events observed by CHAMP and SAC-C during a 4-month period (May–August 2002) are simulated with CRT. In doing so, the L1 and L2 phases of 50 Hz sampling rate are individually modeled. After applying a quality control that discarded some faulty data (e.g., corrupted by unfixable cycle slips, too noisy with low SNR, physically unrealistic or highly suspicious in the quality when compared to the model, or too short in the height range), 42 409 occultation events (23 563 CHAMP and 18 846 SAC-C) are successfully modeled. In this study, measured and modeled RO data are compared in the neutral atmospheric excess phase. To do so, the modeled excess phases are obtained by subtracting the range between GPS and LEO satellites, provided by an algorithm of precise orbit determination (POD), from full phase paths. After that, first-order ionospheric effects are eliminated from both measured and modeled excess phases via a frequency-weighted linear combination: ΔΦc=f12ΔΦ1-f22ΔΦ2f12-f22, where ΔΦc is the ionosphere-corrected excess phase. Depending on the condition of space weather, ionospheric residual error due to the omitted higher-order terms can be non-negligible (Bassiri and Hajj, 1993; Hajj et al., 2002; Danzer et al., 2013). Another complication in the ionospheric correction is that the ionosphere exerts influence on L1 and L2 frequencies differently, leading to slightly different ray paths. This in turn limits the validity of the first-order correction. In addition to the inter-frequency difference, the existence of the ionosphere itself also poses a difficulty. In GPS RO, an ideal ionospheric correction is expected to produce the neutral atmospheric effect without the influence of the ionosphere. However, as shown by Wee et al. (2010), ΔΦc differs from ΔΦIF, which is the hypothetical excess phase observable in the ionosphere-free atmosphere. That is to say, the ionosphere raises or lowers the tangent point, and hence the neutral atmospheric effect experienced following the perturbed ray path is different from what could have been observed along the ray path in the electron-free atmosphere.

The error due to perturbed ray paths is different from higher-order residuals. A significant portion of ionospheric correction error relates to the first-order term and can be removed with the knowledge of ray paths (e.g, Syndergaard, 2000). Fortunately, ray tracing offers a realistic simulation of ray paths. Wee et al. (2010) showed that measured and modeled ΔΦc agree very well, although ΔΦc is suggestively different from ΔΦIF. That is because the path-related errors in measured and modeled ΔΦc cancel each other out to a large degree, as long as the modeled ray paths are realistic. This is one of the reasons that ionospheric models are used in this study even though the modeled ionospheric effects are eventually removed and the comparison is made in the neutral atmospheric phase. The benefit of dual-frequency modeling depends on the actuality of ionospheric and atmospheric models. Especially the empirical ionospheric models might be unreliable at times of higher solar activity or at altitudes where sharp refractivity gradients exist (e.g., around D and E layers). Also, the solar activity in 2002 was quite high. This increases the ionospheric residual error in the measured ΔΦc. In this study, higher-order ionospheric effects are disregarded, assuming that the path-related first-order correction error is dominant. Higher-order dispersion terms deserve more research efforts and taking their effects into account will undoubtedly improve the accuracy of modeled phase path. It must be stressed though that the ionospheric correction error is larger for the phase measurement whose tangent point is within the ionosphere. This is due to the direct impact of local electron density around the tangent point. The ionospheric residual errors are especially alarming when they are vertically accumulated in the standard RO processing. In our approach, the larger residual errors in high altitudes do not propagate downward. In summary of the data processing carried out in this study, measured phases are kept intact except for the ionospheric correction so as not to cause adverse error propagations. Instead, model variables are brought into the measurement space for their comparison with the measurement through ray tracing.

There are a few reasons that the particular period, May–August 2002, was chosen for this study. First of all, the ECMWF started operationally, assimilating GPS RO data on 12 December 2006 (Healy, 2007), and ERA40 did not make use of RO data (Uppala et al., 2005). Therefore, RO data were independent of the ECMWF data during that period. At that time, RO was a relatively new technique and thus had a short span as a climate record. The RO technique has been improving rapidly over time in both instrumental and algorithmic standpoints, and hence the data collected during the pioneering days might be less reliable, containing a higher level of noise. Also, the number of occultation events observed during the early period, before the launch of the six-satellite COSMIC in 2006, was small. In 2002, two RO missions, CHAMP and SAC-C, were in operation. These two missions provided an excellent opportunity to cross validate the RO data (e.g., Hajj et al., 2004). That year is also scientifically significant because the stratospheric sudden warming (SSW) that occurs about every other year in the Northern Hemisphere had been observed only once in the Southern Hemisphere in September 2002 (Gerber et al., 2012). The extreme flow conditions in the austral stratosphere during the period exposed a computational instability of the ECMWF forecast model that had not been seen previously in either tests or operational use (Simmons et al., 2005). The analysis period of ERA40 also ended in September 2002. The unusual atmospheric conditions that preceded and perhaps preconditioned the SSW offer a good testing environment for the NWP system. For instance, as early as the beginning of 2002, ERA40 forecasts already showed a distinct degradation in the fit to radiosonde data, e.g., at 200 hPa (Uppala et al., 2005). Manney et al. (2005) found that during the period some global analyses could differ by about 20 K in the temperature from radiosonde observations.

Figure 1 compares measured ΔΦc at 25 km with modeled ΔΦc for which neutral atmospheric refractivity is derived mainly from either OP or RA. For the plots shown in Fig. 1a–b, a 2-month sub-period (June–July, 2002) is used for graphical convenience. The comparisons presented in the rest of this paper make use of all data available during May–August, 2002. A practical issue deserving of clarification is the so-called initial phase ambiguity, which is the unknown number of cycles or turns of phase when a receiver locks onto the signal carrier of a GPS satellite for the first time. Unless properly resolved, it causes a bias in the measured phase. The neutral atmospheric excess phase decreases rapidly with height and becomes smaller than the typical size of measurement noise in ΔΦc (roughly 3–5 mm in the stratosphere) at 70–80 km. Based on the fact, the initial ambiguity is resolved in this study by setting the measured ΔΦc equal to the modeled ΔΦc at the top of the occultation, which is usually higher than 120 km. It should also be mentioned that although the result is depicted in relation to the tangent height for the purpose of offering a geophysical description, the actual comparison is made in the time domain. That is because the tangent heights of measured phase paths are unknown. Specifically, the difference between a measurement sample and its model counterpart is assigned to the model's tangent height. While the impact parameter varies along a ray path in the heterogeneous atmosphere and is not a ray-invariant, the tangent height (i.e., the geometric height of the tangent point) is always computable and unique on model side.

The departure of OP from the observation (M–O) (Fig. 1a) shows a distinct pattern of negative differences over the interior of East Antarctica, whereas RA (Fig. 1b) shows structured positive differences over the large area around the Ross Ice Shelf. The latitudinal scatter plots (Fig. 1c and d) show that OP and RA differ significantly in the M–O, especially over the high southern latitudes, indicating that at least one of the NWP data are biased. The wavelengths of GPS signals (∼ 19 cm for L1 and ∼ 24 cm for L2) are too long to interfere with cloud droplets, hydrometeors, and aerosols. In addition, RO is not affected by the thermal radiation from the Earth's surface, as opposed to other satellite sensing techniques. Therefore, RO has no particular reason to cause such regional-scale systematic differences.

As described earlier, OP is lower than RA in the model top. The difference between MSIS and RA in the range of 48–65 km thus introduces some level of uncertainty into the comparison at lower heights. One way to quantify the uncertainty is comparing a suppositional RA whose top is assumed to be 48 km (RA48) to the actual RA in the excess phase. To do so, the temperature for RA48 is linearly transitioned from RA into MSIS starting from 45 km and is completely replaced by MSIS at 48 km. The difference between RA and RA48 in the excess phase is found to be very small both in the mean and in the standard deviation, at least for the particular period used in this study (Fig. 2). The comparison is made in the Southern Hemisphere, southward of 30∘ S. The difference above 65 km is induced by an upward hydrostatic reconstruction of the pressure. Therefore, the difference between OP and RA in the M–O at 25 km is hardly attributable to their difference in the top height. In addition, the comparison indicates that the downward propagation of forward modeling error is quite limited in the extent.

NWP data are known to possess organized large-scale systematic errors, e.g., the climatic error of operational ECMWF forecasts, as shown by Jung et al. (2005). Langland et al. (2008) related the regional error to the irregular distribution of in situ and satellite observations. Studies also found that global analyses differ the most in data-void areas, signaling their uncertainty there (Newman et al., 2000; Marshall, 2002; Sterl, 2004; Bromwich and Fogt, 2004; Betts et al., 2006). In this regard, a noteworthy detail shown in Fig. 1c and d is that the large departures (M–O) near the South Pole (> 2 %) are opposite in their sign, despite both OP and RA being produced by the same NWP center (i.e., ECMWF) using virtually identical observations. The disparity may relate to the difference in the assimilation method. For instance, RA uses a three-dimensional variational scheme (3DVAR), whereas OP uses a 4DVAR. They are also different in the data usage (Uppala et al., 2005). Nonetheless, this is a good example of the uncertainties in NWP data. The difference between OP and RA does not itself inform which is at fault. It is thus worthwhile to see if the phase measurement is accurate enough to serve as a reference against which their relative trustworthiness can be reconciled.

In order to confirm that the discrepancy between OP and RA is not caused by RO, we separated RO data into the missions, and then compared their departures from a common model. As shown in the scatter plots of O–M for OP at 25 km (Fig. 3a and b), the two missions are quite similar in the zonal mean and standard deviation of O–M. Further analysis is carried out using collocated RO pairs. Over the Southern Hemisphere, 819 closely distanced pairs (within 2 h in time and 300 km in distance) are found. The paired deviations from RA are highly correlated, where the correlation coefficients are 0.89 at both 25 and 12 km (Fig. 3c and d). The deviations aggregate densely at small-magnitude ends (i.e., near the origin of the coordinates) and along the line of y=x. The root-mean-square distance perpendicular to y=x is very small: 0.296 and 0.087 % at 25 and 12 km, respectively. If the high correlation among the collocated pairs is caused by correlated observation errors (i.e., systematic measurement error), the OP and RA in Fig. 1c–d are also expected to show a positive correlation in M–O over the region. On the contrary, they exhibit a strong negative correlation. Thus, the high positive correlation among the collocated pairs reflects the systematic NWP error common to the pairs. Indeed, this is a well-accepted way of characterizing spatial error correlation in NWP data (Hollingsworth and Lönnberg, 1986; Kuo et al., 2004; Desroziers et al., 2005).

Figure 4a and b compare OP and RA in the systematic difference and standard deviation from measured phase paths. The statistics are stratified into three latitude bands: Northern Hemisphere (NH), Southern Hemisphere (SH), and tropics (TR). The TR is defined as the area between 30∘ S and 30∘ N. In tropospheric heights, the analyses are close to RO and each other in the mean. However, their systematic differences from RO increase rapidly above 15 km and diverge into opposite directions. At the height of 40 km, OP and RA are very different from each other in all latitudes. As backed by the results presented thus far, both OP and RA are significantly biased, and RO data are able to quantify their systematic errors. In the data-rich NH, OP and RA deviate less from each other and RA stays almost unbiased up to 30 km. This indicates that the data assimilation systems rely heavily on conventional observations, at least in defining the model's mean states.

In the standard deviation, OP generally agrees better with RO than RA does, in particular in the stratosphere of the SH. The exception is the lower altitude below 22 km of the SH, where the vertical resolution of the specific OP used in this study seems too coarse to properly represent the strong thermal gradient relating to the intense polar vortex during the period. In the NH and TR, both analyses show remarkable agreements with RO in the standard deviation, less than 0.2 % at 8 km and increases to 0.6–0.7 % at 30 km. This suggests that ECMWF analyses as well as RO data are proper in this measure. Errors of both RO and NWP are responsible for the larger standard deviations in the upper stratosphere. In the tropical lower troposphere, measurement error that arises from difficulties in reliably tracking GPS signals passing through the optically complex atmosphere also contributes to the increasing standard deviation. No comparison is made in the lowest 2 km because the voltage SNR drops below a prescribed threshold (50 V/V) in most cases. Needless to say, it is also challenging for NWP models to properly represent the complex atmospheric structure. The phase measurements are accurate and clearly discern the growth of prediction error with the forecast range (Fig. 4c–e). The standard deviation of RA forecasts from RO data increases monotonically with the lead-time in all latitudes and is most pronounced in the SH. The forecast error increases rapidly in the upper troposphere and lower stratosphere. These areas, SH and high altitudes, are where the ECMWF data need greater improvement.

Another noteworthy feature in Fig. 4a is large-scale undulations in the stratosphere. In the SH, the standard deviation of RA also shows a trace of the oscillation. Previous studies have reported the oscillatory vertical structure over the Antarctica in the ECMWF temperature (e.g., Randel et al., 2004; Uppala et al., 2005; Manney et al., 2005). RO-derived temperature also captured similar features (e.g., Gobiet et al., 2005; Foelsche et al., 2008). Although both the phase measurement and the derived RO temperature are able to detect the oscillations, the important distinction between them is that the phase path offers a higher level of transparency when the focus is on identifying NWP errors. A primary concern about derived temperature here is the influence of a priori information in the stratosphere. Advanced algorithms (e.g., Gorbunov, 2002; Wee and Kuo, 2013, 2014) may alleviate the problem but they are unexpected to completely resolve the issue. Readers are referred to a review on the problem provided by Wee and Kuo (2014).

In order to evaluate the advantage of the phase path over derived RO data in the NWP verification, we compared CDAAC's derived RO data with the ECMWF data over the SH. CDAAC recently released a new version of CHAMP data. The RO data set used in this study is an older version (0007.0004). The ECMWF data are interpolated to the exact location of individual RO profiles. In doing so, the horizontal drift of tangent points with height is fully considered. Figure 5a shows the difference in the bending angle before and after the statistical optimization (SO). The SO is a regularization of measurement noise in the stratosphere by means of blending observed bending angle with a priori accounting for their relative error. The CDAAC's SO makes an obvious change in the mean in altitudes above 30 km during May–August 2002. In the SO, the weighting given to a priori increases with height, as can be inferred from the increasing standard deviation with height. The mean difference decreases with height above 34 km because the observed and a priori bending angles (over the SH and during the period) differ less from each other in higher altitudes. This may relate to the vertical extent of the SSW during the period. The propagation of a priori error is clearly visible in the refractivity (Fig. 5b) and derived temperature (Fig. 5c). The SO shifts the mean of M–O significantly for both OP and RA, to a degree greater than the difference between them above 25 km. Consequently, OP and RA now show the same sign in M–O throughout the height range of 15–40 km.

CDAAC's SO uses a long-term climatology as a priori that does not account for interannual variations. Accordingly, when atmospheric conditions deviate considerably from the long-term climatology (e.g., the SSW event in this study), the SO can result in a sizeable error in the bending angle that is in turn carried forward into subsequent data products. The errors due to SO are particularly problematic because they are systematic and thus accumulate (instead of canceling each other out) through Abel inversion and hydrostatic integration, and propagate well down to 10 km. In the worst-case scenario, a priori error can supersede actual climate anomalies contained in RO data, hiding them from detection (Wee and Kuo, 2013). Although they agree fairly well with the ECMWF temperatures both in the mean and standard deviation below 25 km, RO-derived temperatures possess measurement error and retrieval errors that are mingled together, and thus necessitate meticulous care for interpreting their comparison with NWP data. In contrast, the phase path does not undergo the atmospheric data processing and is thus free from the retrieval errors. On the NWP side, the phase path can be modeled very precisely using the CRT, meaning that the modeling does not substantially increase the NWP error. Our study also suggests that the vertical propagation of NWP error due to the modeling is quite limited in the extent. Being minimally processed, the phase path has the minimal data uncertainty and hence offers a more reliable quantification of the NWP error.

The quality of derived RO data is difficult to properly characterize because of the propagation and interaction of errors through the data-processing chain. Among the errors arising from various sources, retrieval error depends on the algorithms used for the data processing and has very complicated characteristics. In order to overcome the difficulties arising from the retrieval error, we propose that the minimally processed RO data (i.e., the phase path) should be used for studies aiming at understanding the basic limitations and aptitudes of the RO technique, which is vital for a sound application of RO data. Without being hindered by the retrieval error, the use of the phase path is envisaged to provide uncomplicated insights into the factual quality of RO data. In this study, phase measurements are modeled with an effective ray tracer in conjunction with a precise ray shooting, where atmospheric refractive indices are modeled with ECMWF data, and with empirical ionospheric and plasmaspheric models.

This study confirmed the high quality of phase paths. It is concluded that the measured phase path is accurate enough to differentiate ECMWF's operational analysis and the 45-year reanalysis in view of their systematic error, even though they were produced by the same center using virtually the same set of data. In the southern hemispheric stratosphere, in particular, the two analyses were significantly biased in opposite directions with the phase measurement in between. The phase path showed a good agreement with ECMWF analyses in the standard deviation and lucidly revealed the error growth of short-term ERA40 forecasts. As our comparison showed, the quality of ECMWF data during the period chosen for this study seems particularly unsatisfactory. The quality of ECMWF data has been rapidly improving in recent years, especially in traditionally data-sparse regions. Therefore, application of our approach to recent data will be useful if the dependency of NWP data on RO data can be properly addressed.