Spatial scaling in the variability of atmospheric properties such as water vapour (q) can be characterised via structure functions, with the nth-order structure function of a field f(x), Sn defined as 1Sn(r)=Efx-fx+rn, where E[] is the expected value, x a location, and r a separation between points. Fields of temperature (T), q, and wind speed are commonly well modelled by a power law: 2Snr∝rζn, such that ζn is the log–log gradient of Sn as a function of r. There is strong motivation to quantify and understand these exponents and the ranges Δr within which they are valid, and here we specify second-order structure functions S2 with exponent ζ2, describing variance scaling. This is related to the commonly referenced Fourier power spectrum exponent β: 3β=-ζ2+1. One motivation for obtaining these exponents is that climate model sub-grid variability in q is strongly linked to cloud formation (Golaz et al., 2002; Perraud et al., 2011; Sommeria and Deardorff, 1977). In principle, sub-grid variance can be tuned for each model set-up, but scale-aware variance relationships allow a smooth and consistent transition between low-resolution (order ∼ hundreds of kilometres) and high-resolution (order ∼ km) models (Arakawa et al., 2011; Schemann et al., 2013).

At scales larger than model grid cells, observational estimates of variance scaling can also be used to assess model performance, as has been done using estimates of temperature (T) and q from Atmospheric Infrared Sounder (AIRS) and airborne campaign data (Kahn et al., 2011).

Furthermore, the scaling exponents are related to the physical processes that generate the cascade of turbulent eddies in the atmosphere. For example, while mean-scale statistics retrieved by AIRS are approximately isotropic in the horizontal, there are differences in scaling between the horizontal and vertical (Pressel and Collins, 2012). Scaling following ζ2=2/3 is predicted for a passive tracer in the inertial range of 3-D locally isotropic turbulence following Kolmogorov theory. Accounting for the buoyancy effects can strongly modify that scaling (Bolgiano, 1959; Obukhov, 1959), with different exponents expected for the velocity (ζ2=6/5–7/5) and scalars (ζ2=2/5), as shown in e.g. Kunnen et al. (2008), Wroblewski et al. (2010), or Boffetta et al. (2012). Exponents of 7/5 have been commonly measured for vertical wind profiles from dropsondes (Lovejoy et al., 2007), and values typically near ζ2 of 6/5 for horizontal water vapour in non-convective areas and ζ2=0.72 in convective areas have been determined from airborne lidar retrievals over horizontal ranges of up to 100 km (Fischer et al., 2012, 2013); meanwhile, the sub-kilometre regime remains undermeasured.

Of particular interest for modellers is the existence of “scale breaks”, distances at which the exponents change such that a smooth transition between model resolutions may not be possible. These have been calculated to occur at distances over a broad range of 10–1000 km (Bacmeister et al., 1996; Gage and Nastrom, 1985; Kahn et al., 2011; Pinel et al., 2012), and the range of scales has been speculated to be related to the size of convective systems (Dorrestijn et al., 2018) or changes in the nature of turbulence (Kurowski et al., 2015; Skamarock et al., 2014).

This study focusses on the estimation of horizontal scaling of clear-sky TCWV at sub-kilometre scales and specifically on the integrated PBL water vapour which we refer to as the partial column water vapour (PCWVPBL). Recent work using airborne data (Thompson et al., 2021) and large eddy simulation (LES) output (Richardson et al., 2021a) has provided evidence that sub-kilometre horizontal variability in total column water vapour (TCWV) is almost perfectly correlated with PCWVPBL variability, such that high-spatial-resolution retrievals of TCWV from visible and shortwave infrared (VSWIR) imaging spectrometers can provide unique information about PBL q variability. This is not a statement that all water vapour is inside the PBL but rather that on sub-kilometre spatial scales, the variability in low-altitude water vapour dominates the horizontal column variability. It remains a challenge to disentangle variability from different heights within the PBL such as the sub-cloud layer or a conditionally unstable cloud layer, which may have different scaling properties.

In particular, the PBL depth is typically 1–2.5 km in these simulations, while we obtain variability statistics over horizontal ranges of under 1 km. The vertical averaging over a scale larger than the horizontal calculation means that the interpretation of the physical meaning of the derived exponents is challenging and may not be directly related to the theoretically derived exponents discussed above. This study aims only to determine whether the measurement problem of the solar path can be overcome and leaves the physical interpretation beyond its scope. This point will be revisited in Sect. 4.

Several modern and upcoming missions obtain or will obtain VSWIR spectra that could allow retrieval of TCWV at horizontal resolutions from 20 to 100 m. Current examples include the Multi-Spectral Imager (MSI) on Sentinel-2 (Drusch et al., 2012), the PRecursore IperSpettrale della Missione Applicativa (PRISMA, Candela et al., 2016), and the DLR Earth Sensing Imaging Spectrometer (DESIS, Krutz et al., 2019). Upcoming missions such as NASA's Earth Surface Mineral Dust Source Investigation (EMIT; Green and Thompson, 2020) and ESA's Copernicus Hyperspectral Imaging Mission for the Environment (CHIME; e.g. Rast et al., 2019) will obtain a horizontal resolution of order 30–80 m, and this study will assess performance assuming footprint sizes of 40–50 m, i.e. at the mid-point of that range.

This footprint selection was made based on the resolution of available LES output, and the analysis method is based on the output of retrievals developed specifically for EMIT, which primarily retrieves TCWV to allow atmospheric correction for its surface reflectance target observable. Other instruments such as PRISMA or MSI may have different error characteristics, but we treat retrieval errors in a general manner that could be expanded to these other instruments.

This study attempts to determine whether EMIT will be able to obtain ζ2 over 0.5–1 km after accounting for (1) random retrieval error, (2) systematic biases in retrieval mean and sensitivity, and (3) solar zenith angle.

The PBL is of particular interest since it is the location where reflective low clouds form, and in the Dorrestijn et al. (2018) analysis, ζ2 varied more in the 850 hPa layer than the 300 or 500 hPa layers. However, previous analyses have generally been restricted to far larger spatial ranges, with Dorrestijn et al. (2018) referring to 55–165 km scale variance as occurring at the “tiny scale”. Other examples at higher resolution generally use airborne measurements, with few calculations for separations under 1 km using onboard sensors (Cho et al., 1999), using lidar at 5–100 km (Fischer et al., 2013), or evaluating simulations with lidar for separations > 11 km (Selz et al., 2017).

This study uses LES outputs and does not make any statements about the realism or cause of the LES output ζ2 values but instead aims solely to identify and quantify retrieval biases and errors. Its greatest contribution is to demonstrate that directional calculation strategies can remove biases in estimates of ζ2 introduced by the direct-beam component of the solar path through the atmosphere.

We show that, in a set of 23 LES snapshots, the non-vertical direct-beam path prevents accurate retrieval of sub-kilometre ζ2 when standard methods are naïvely applied. However, errors in ζ2 depend on the direction in which it is calculated relative to the solar azimuth, and selecting the correct solar-aware direction eliminates the bias and therefore overcomes a fundamental barrier to VSWIR estimation of sub-kilometre q scaling. Diffuse sunlight is handled through a plane-parallel radiative transfer approximation, which means that complex 3-D radiative effects are neglected. In clear-sky areas near clouds, 3-D effects can brighten observed spectra (Várnai and Marshak, 2009), with induced biases of order ∼ 0.25 % for VSWIR column CO2 retrievals (Massie et al., 2021). The consequences for hyperspectral TCWV retrievals at 30–80 m horizontal resolution are not currently known, although the effect on retrieved ζ2 would depend on the spatial scaling of these TCWV biases. In Sect. 4 we propose a field experiment design to test our conclusions. Spaceborne spectrometers measure light that has passed along some path, r↓↑, consisting of downward (Sun-to-surface, r↓) and upward (surface-to-sensor, r↑) components. Retrievals will respond to the path-integrated water vapour (PIWV) between the surface and top of atmosphere (TOA): 4PIWV=∫TOAsurfaceq(r↓)dr+∫surfaceTOAq(r↑)dr. Meanwhile the commonly desired value is the vertically integrated water vapour, TCWV: 5TCWV=∫surfaceTOAq(z)dz. For a horizontally uniform q field (equivalent to “plane-parallel” in radiative transfer parlance) there is a simple geometric relationship between the two: 6TCWV=PIWVuniform/1μ+1μ0, where μ is the cosine of the sensor-viewing zenith angle and μ0 the cosine of the solar zenith angle. Despite the fact that real q fields vary horizontally such that Eq. (6) is not strictly true, the VSWIR community commonly assumes a plane-parallel atmosphere and reports the retrieved value as being TCWV (Carbajal Henken et al., 2015; Diedrich et al., 2015; Grossi et al., 2015; Nelson et al., 2016; Noël et al., 2004; Preusker et al., 2021). Here we calculate PIWV by tracing solar rays through 3-D LES output, but for consistency with VSWIR literature terminology, we relate it to an effective TCWVeff: 7TCWVeff=PIWV/1μ+1μ0. Our retrieved values, TCWVret, refer to estimates of this property. Figure 1 compares the true TCWV with TCWVeff derived from ray tracing through 3-D LES output when solar zenith angle SZA is 45∘ and solar azimuth is at 0∘, meaning that the horizontal component of r↓ is in the negative y direction. There is an apparent “smearing” from Fig. 1a to b in the y direction, so we refer to these solar-geometry-induced changes in TCWVeff due to the horizontal variability in the q field as the “solar smearing” effect (for a simplified illustration of the physical principles behind why our strategy is anticipated to reduce biases in ζ2, see Figs. S1–S3 in the Supplement).

This consequence of solar and view geometry is well known; for example, Thompson et al. (2021) only used flight lines with very low SZA to minimise its effect in a study of ζ2. However, this severely limits viable data, so here we suggest a new method that exploits the directionality of the solar-smearing effect in order to perform such calculations across a wider range of conditions. This paper uses the solar-path-traced outputs of Richardson et al. (2021a) to demonstrate that by calculating the structure function in a direction perpendicular to the solar azimuth, the bias in ζ2 is removed within the 23 LES snapshots considered. To our knowledge, this is the first quantification of such a technique to mitigate solar-geometry-induced errors in spatial statistics of retrieved atmospheric properties.

If this result can be extended to the real world, then upcoming high-spatial-resolution spaceborne VSWIR spectrometers will provide a breakthrough for analysis of moisture scaling across unprecedented spatial scales. Our retrievals are restricted to clear-sky areas over land, but this still represents a substantial advance in the current capacities of other instrument types. Lidar measurements avoid the solar-path issue and can retrieve vertical information including above clouds, but current and anticipated spaceborne lidars do not offer VSWIR's fine spatial resolution or broad spatial coverage from a wide swath. Sounders such as infrared can profile the atmosphere but have footprints that are too large for sub-kilometre exploration.

Even airborne measurements, which offer far less coverage than spaceborne sensors, may suffer from their own challenges. Evidence suggests that the tendency of flights to follow isobars rather than maintain altitude can introduce height variations that blend vertical variation into horizontal calculations and result in exponents similar to those predicted due to buoyancy's vertical effect (Lovejoy et al., 2004; Pinel et al., 2012).

The VSWIR sampling technique introduced here could greatly expand the range of conditions under which spatial scaling of water vapour can be quantified. In Sect. 2 we describe the LES output, simulated retrievals, and how retrieval errors and solar path are accounted for in calculation of ζ2. Section 3 presents the results and Sect. 4 discusses and concludes.

We have shown that for the q fields simulated in five LES runs of shallow convective PBLs, a novel strategy accounting for solar azimuth eliminates the SZA-induced bias in calculated ζ2 over r of 0.5–1 km from high-spatial-resolution VSWIR retrievals. This substantially increases the range of applicable geometries for which VSWIR retrievals can be used to estimate spatial scaling statistics. For example, in the airborne case studies of Thompson et al. (2021), only flight lines with SZA < 15∘ could be used, while our method promises unbiased estimates of sub-kilometre ζ2 for SZA up to 60∘. Removal of the bias is particularly important for applications, since q-scaling analyses using spaceborne data typically group or average over many sets of measurements (e.g. Kahn and Teixeira, 2009), and this averaging will not reduce bias in the way that it reduces RMSE. For the first time it also allows calculation of high-resolution ζ2 statistics over mid-latitude and polar areas of the globe that do not experience low solar zenith angles.

This approach should be applicable to any instrument that obtains TCWV from VSWIR with a horizontal resolution approaching 50 m, not just EMIT. Operationally, this requires sufficiently long, continuous sampling perpendicular to the solar azimuth. We have not analysed length requirements here but note that for airborne campaigns this is easily addressed on a flight-to-flight basis. For spaceborne instruments, the sampling will depend on the swath size and orbital geometry. EMIT's approximately 75 km swath (Bradley et al., 2020) spans distances larger than those considered here, and so we expect sampling to be sufficient regardless of orbital configuration. Similarly, ESA's CHIME contractor notes a 128 km swath, and similar capacities are expected for the missions that address NASA's Surface Biology and Geology (SBG) and Aerosols, Clouds, Convection and Precipitation (ACCP) designated observables. However, we note that it will be necessary to understand the implications of non-uniform footprint size- and footprint-dependent errors determined from the on-orbit instrument performance in order to increase the confidence in estimates of ζ2 from these spaceborne sensors.

Our analysis is based on the retrieval outputs of Richardson et al. (2021a), which showed how random retrieval error σε can be identified and removed from TCWVret fields by exploiting the properties of TCWV S2 at small scales. It also showed that errors in the T and q profiles assumed in the retrieval could affect retrieval sensitivity a1=dTCWVret/dTCWVeff. We have shown here that to obtain ζ2 it is critical to remove the effect of σε but that while errors in a1 are the largest error source for estimates of spatial standard deviation, they do not greatly affect the derived ζ2.

Our results were only determined for areas where the surface type is composed either of mixed vegetation or mixed urban-mineral surfaces, so either a simultaneous surface classification (as provided by ISOFIT) or ancillary surface information would be required. We also identified that horizontal gradients of 0.3 AOD km-1 or less have only a small effect on derived ζ2 in most cases, and since ISOFIT simultaneously retrieves AOD, it seems likely that cases where this is not true could be identified. However, we highlight further investigation of the effect of realistic AOD structure as a useful future investigation. A major limitation of this study is that 1-D radiative transfer was used to derive the relationship between TCWV and TCWVret, while clouds can affect nearby clear-sky scenes through 3-D radiative effects. Future work could address this by using a 3-D radiative transfer code (e.g. Evans, 1998; Emde et al., 2011) as the forward model in an improved OSSE.

Ideally this sampling strategy could be field tested, and a strategy to do so would be to perform flights both parallel and perpendicular to the solar azimuth with collocated retrievals of TCWV from VSWIR and an independent instrument that is not affected by SZA, such as a differential absorption lidar (DIAL) or passive sounding instrument. The non-VSWIR instrument would allow calculation of ζ2 that is not affected by sunlight path, and the conclusions of this study would be supported if the VSWIR and non-VSWIR estimates showed good agreement for the perpendicular but not parallel flight paths. We highlight the High Altitude Lidar Observatory (Bedka et al., 2021) as a candidate sensor for such an experiment.

For science applications, it would also be necessary to identify which scenes are likely to satisfy our requirements: for example, conditions proximate to deep convection may have more substantial above-PBL variability at < 1 km scales and result in weaker correspondence between TCWV and PCWVPBL. In addition, it may be necessary to identify PBL height, which would require independent information from weather forecast models or reanalysis, in situ measurements, or other instruments. Furthermore, users would also need to consider the effect of sampling biases that may depend on cloud fraction or on the presence of non-isotropic variability in features such as horizontal convective rolls (e.g. as explored in Carbajal Henken et al., 2015). If locations commonly experience meteorological features with a preferential orientation, for example due to orography or a coastline, then this method may not adequately capture its full structure.

Finally, the interpretation of these exponents has not been considered in detail here, but future work could proceed either observationally or theoretically. For an observational study, the relationship between retrieved exponents and other properties, such as later convective initiation, could be investigated. A theoretical study would need to apply current understanding of turbulent physics to address precisely the problem of the horizontal variability of vertically averaged profiles. Further in the future, perhaps multi-angle imaging spectroscopy could provide profiling from VSWIR measurements via computed tomography, which has been demonstrated to retrieve aerosol profiles in some conditions using the Multi-Angle Imaging SpectroRadiometer (MISR) on Terra (Garay et al., 2016), and upper-tropospheric water vapour profiles from airborne measurements with the Gimballed Limb Observer for Radiance Imaging of the Atmosphere (GLORIA, Ungermann et al., 2015). We are not aware of any likely short-term space missions that would allow water vapour tomography from VSWIR retrievals at EMIT-like horizontal resolution, with the upcoming Multiangle Imager for Aerosols (MAIA, Diner et al., 2018) having a horizontal resolution similar to that of MODIS. If such an instrument were launched, high-spatial-resolution tomography would also be subject to the smearing effect from the non-vertical solar path and so may benefit from our proposal, although a detailed study of the particular instrument and retrieval set-up would be required.

Our main conclusion is that a novel sampling strategy can allow a breakthrough in space-based measurement of PBL water vapour scaling and that while individual uncertainties and geographical sampling may vary with the mission, this principle will apply to the array of upcoming VSWIR instruments whose spectra will allow column water vapour retrievals at resolutions from 30 to 80 m or better. The resulting statistics offer a new check on the validity of high-resolution atmospheric models and inform sub-grid parameterisations of coarser ESMs.