Assuming the sensor's FOV does not include the Sun, the zenith spectral radiance consists of solar radiation scattered by the molecules, aerosols, and cloud water droplets in the FOV, which may include radiation that has been scattered multiple times from the atmosphere and the terrain. The VNIR spectral band (Fig. 3), at a moderate spectral resolution of 2 nm, shows a broad baseline with multiple narrow absorption features. Many of these are due to water vapor, as well as Fraunhofer lines. The spectral radiance at 440 nm is in a region relatively free from atmospheric gaseous absorption and is thus suitable as a COD proxy. We chose 440 nm for TWST radiance-to-COD lookup because that is a wavelength used by AERONET cloud-mode sensors, which serve as a source for comparative validation.

The model used for generating 440 nm radiance-to-COD lookup tables is the MODerate resolution atmospheric TRANsmission (MODTRAN) atmospheric radiative transfer code (Berk et al., 2006). MODTRAN5 incorporates the Discrete Ordinates Radiative Transfer (DISORT) code (Stamnes et al., 1988) for plane-parallel stratified media, i.e., idealized one-dimensional radiative transfer (1DRT). Calculations are done for a typical water stratus cloud above a stated ground albedo, for a stated nominal aerosol profile, over a grid of COD and SZAs. Figure 4 is a graphical depiction of sample tables. For any SZA, there is a “bright-point” radiance where the idealized 1DRT cloud radiance reaches a maximum, typically occurring for a COD between 2 and 8, as seen in Fig. 4. Real clouds manifest three-dimensional radiative transfer (3DRT) effects, including radiances exceeding the idealized 1DRT bright-point radiances (Marshak et al., 2000). When faced with such exceedances, the TWST retrieval algorithm reports the COD corresponding to the bright-point radiance but flags an out-of-bounds (“3-D cloud”) condition.

The TWST algorithm currently operates without any information on the droplet size distribution or the cloud base height, and with a prior estimate of the ground albedo and aerosol loading profile. We do not consider deviation of the actual from nominal aerosol profile, as such perturbation from the baseline aerosol optical depth (AOD) is typically a small contributor to reported COD. We performed some initial sensitivity studies on these remaining parameters. The albedo sensitivity findings below will prove of value in helping to explain the minor disagreement bias between coincident TWST and AERONET COD observations (Sect. 4.2.2).

Because our implementation of the TWST algorithm uses a radiance database generated with the MODTRAN model, we studied 440 nm radiances from four different cloud types parameterized within MODTRAN, which assume Mie scattering, lognormal droplet size distribution, and liquid water refractive index. These types have effective radii of 12.0 (cumulus), 7.2 (altostratus), 8.3 (stratus), and 6.7 µm (stratocumulus). Water cloud drop-size distributions typically vary from an effective radius of 1–20 µm (see, e.g., Chiu et al., 2006). We modeled clouds with fixed base height of 0.5 km and fixed physical thickness of 0.5 km. For each cloud type we varied the LWP enough to achieve 550 nm CODs between 0 and 100; LWP was used because it is an input to MODTRAN. COD values were estimated from LWP using the Wood and Hartmann (2006) modification to the Stephens (1978) formula as described in Chiu et al. (2012). Figure 5 shows the computed 440 nm vertical radiances plotted vs. LWP and COD. The different size distributions lead to different radiances for the same LWP as expected. However, in the COD vs. radiance plot the curves overlay closely, at least for COD ≤ 20. We verified this intuitive result using a simple standalone two-stream computation as an alternative to the DISORT algorithm included within MODTRAN (Stamnes et al., 1988). These results are corroborated by the more extensive sensitivity results of McBride et al. (2011). This relative insensitivity of COD with effective radius gives us some confidence in reporting COD values in the face of the variety of water clouds.

Inter-reflections between the ground and a thick cloud can be significant unless the Earth albedo is low. At the 440 nm lookup table wavelength most Earth cover types have albedo of 0.2 or less as shown in Fig. 6 with samples of the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) database (Baldridge et al., 2009); notable exceptions are for white sand, fresh snow, and ocean ice. To characterize the sensitivity of retrieved COD due to albedo uncertainty, we used MODTRAN to compute 440 nm zenith radiances for a low-altitude stratus cloud for albedos of 0.0, 0.1, 0.2, and 0.5, over a range of solar zeniths and cloud optical thicknesses. Figure 7 presents a first-order indication of sensitivity and plots 440 nm radiances vs. COD for albedo bounds of 0.0 and 0.5. The thin–thick ambiguity, the strong variation with solar zenith, and the weaker variation with ground albedo are evident in these plotted results.

The plot in Fig. 8a further explores this sensitivity and shows the signed change in retrieved COD value for an unexpected increase in the ground albedo from 0.1 (for which the radiance-to-COD lookup tables are computed) to 0.2. Each curve, for either thick or thin cloud, pertains to some fixed percentage of the aforementioned 1DRT bright-point radiance “Lbrt” (which varies with SZA; cf. Fig. 4). These curves show that a higher-than-expected albedo implies retrieval of a lower (higher) COD in the thick (thin) regime. The largest change we found was ΔCOD of 5, but that occurred for very thick clouds, such that the relative change was only 10 %. Near the bright point (red curves in Fig. 8a) the COD vs. radiance is quite nonlinear (Fig. 4), and thus the red curves, approximated by linear interpolations for this study, are less precise and jagged.

To provide analytic support to these albedo sensitivity findings, we performed calculations employing asymptotic radiative transfer (ART) theory relations as elucidated by King (1987) and Melnikova et al. (2000). Both MODTRAN and our ART calculations compute the 440 nm radiance and optical thickness for cloud with phase function asymmetry parameter g=0.86 and single-scattering albedo ω0 > 0.9999997 (i.e., conservative scattering). ART-computed sensitivities are plotted in Fig. 8b, and compare well in both trend and magnitude against the thick regime curves of Fig. 8a, to which ART theory pertains (here COD > 9). These ART sensitivities are processed from the more directly obtained ART calculations plotted in Fig. 9. The connection between Figs. 9 and 8b is depicted by the dotted path shown in Fig. 9. The reader is directed to graphically determine the error in retrieved COD value by first starting with a given true COD value, tracing rightward from that y intercept parallel to the x axis, and reaching a curve pair for a given SZA. The right curve of the pair registers the actual radiance measured for the unexpected 0.2 albedo, but the left curve is radiance-to-COD lookup table computed for the expected 0.1 albedo. So at the intersection with the right curve, the reader traces down parallel to the y axis to intercept the left curve and then traces leftward back to the y axis and reads out the lower COD value. The solid curves of Fig. 8b are not exactly comparable to those of Fig. 8a in that the former are for constant COD whereas the latter are for constant relative radiance (with respect to the 1DRT bright-point radiance). To corroborate that the COD error in fact decreases with SZA, the dashed curves of Fig. 8b better (but not exactly) correspond to constant relative radiance. These dashed curves have COD decrease with decreasing SZA, which referring to Fig. 4 yields a more stationary relative brightness (downward-sloping line marker) than for COD constant with SZA (horizontal line marker).

At this point it is important to note that the spectral radiance chosen at some other wavelength than 440 nm could be used for the radiance-to-COD lookup. In principle the choice of wavelength depends upon freedom from atmospheric gaseous absorption and on the ground albedo of the measurement site. One wants a wavelength with the lowest absolute albedo uncertainty to minimize errors in the COD due to errors in the assumed albedo for the MODTRAN5 computations. This flexibility in the choice of wavelength is basis of the term “spectrally agile” within the TWST acronym.

For low-altitude water clouds, uncertainty in the cloud base height has a negligible impact on COD retrieval. We ran MODTRAN for cloud base heights of 500 m and 2 km, iterating over 10 COD and 11 solar zenith angles for each. The 440 nm radiances were nearly identical, as shown in the scatterplot of Fig. 10.

Various lookup table algorithms were investigated to reach a reasonable tradeoff between accuracy and speed. The MODTRAN5 tables are preprocessed as follows. Referring to Fig. 4, for each SZAi entry, a cubic spline curve SR440(COD;SZAi) is fit to its SR440-vs-COD table for both optical thickness regimes. A bright-point radiance vs. SZA spline curve Lbrt(SZA), depicted by the dotted black curve in Fig. 4, is fit through the bright-point radiances across the SZAi entries. During operation, the algorithm identifies the tabulated SZAj closest to the current solar zenith angle SZAobs. Then a working copy of its spline curve SR440(COD;SZAj) is linearly scaled in radiance so that its bright point matches Lbrt(SZAobs). This scaled curve is then used to look up the COD value for the measured SR440.

The well-known oxygen A-band centered near 760 nm (Mulliken, 1928; Wark and Mercer, 1965) has been used for many years to study the atmosphere from satellite and ground-based sensors. Pfeilsticker et al. (1998) first used well-resolved A-band spectra to study the probability density function of geometrical path lengths for skylight transmitted from clear and cloudy skies to the ground, following the suggestion of Pfeilsticker et al. (1996) and Harrison and Min (1997). The A-band is virtually free from absorption by other atmospheric constituents (Pfeilsticker et al., 1998) except for aerosol and cloud continua extinction plus a very small amount of line absorption by water vapor (Fig. 11). Thus its continua-normalized (Sect. 3.3.2) spectral-average quantity, termed the equivalent width (EQW), provides a direct measurement of the average amount of oxygen-density-weighted photon path length from the Sun to the sensor. Since oxygen is uniformly mixed in the atmosphere, this is related to the photons' physical path lengths. Therefore the EQW supplies useful information about whether a zenith radiance measurement is in the optically thin or optically thick regime. A virtue of EQW is that it may be stably computed from low-resolution spectral data such as from the TWST sensor, as detailed in Sect. 3.3.2.

Of course, other factors cause EQW to change besides COD. Changes in the SZA produce decreases in EQW with time during the morning and increases in the afternoon. Changes in the density-weighted average cloud thickness and cloud altitude also affect the EQW independent of the COD.

The equivalent width is computed from the spectral radiances between 750 and 785 nm by fitting a straight line to the continuum baseline from 750 to 760 and from 770 to 785 nm (Fig. 11), then dividing each measured spectral radiance by the corresponding linear fit baseline to produce a transmittance value, and then summing these values across the absorption band. This calculation normalizes away the continuum transmittance. The veracity of these calculations depends on accurate spectrometer dark-current calibration and subtraction (discussed in Sect. 2.3). Otherwise, a dark bias of the spectral radiance would falsely alter the computed transmittances and EQW value.

COD is a two-valued function of up-looking spectral radiance, while oxygen equivalent width is a monotonic function of COD for COD ∼ > 1. By plotting SR440 vs. EQW as COD increases from no cloud to thick clouds, one traces out a “nose-like” shape (Fig. 12). For the very lowest COD values, EQW decreases with increasing COD and the slope is negative. Beyond about COD = 1, the lower portion of the nose where the slope is positive corresponds to the optically thin regime; the upper portion where the slope is negative corresponds to the optically thick regime. Within a span of several seconds, passing clouds most often do not trace out a complete nose but only a small segment of it as the cloud evolves and drifts in the wind over the sensor. Notwithstanding the lowest COD values discussed further below, whether the cloud changes involve increases or decreases of the COD, the slope of the corresponding segment indicates the cloud's optical thickness regime. This is the TWST basis for resolving the COD ambiguity.

The nose plot in Fig. 12 includes a smooth curve, based on MODTRAN5 computations but elastically stretched to fit the depicted data points over a 4 min measurement where the COD varied strongly between the indicated blue-sky, thin, and thick regimes, and points deviating well away from the ideal 1DRT smooth curve. These deviating points are classified as either 3-D cloud based on their SR440 exceedance of the 1DRT bright-point radiance value (Sect. 3.2.1) or as “mixed” points attributed to heterogeneous cloud structure within the field of view, itself a 3DRT effect. The classification of the remaining data points into optically thin, thick, or blue-sky regimes was corroborated against coincident all-sky camera video. Although the MODTRAN5 computed nose plot curve supports these regime classifications, it is important to note that the TWST algorithm does not employ model-generated nose plot curves to guide its thickness regime determination. Indeed, the particular shape and slope of a computed nose plot curve varies, as it should, with the unknown physical cloud thickness. Instead, the algorithm exploits the aforementioned positive slope (thin) and negative slope (thick) generic properties of the nose plot.

The cloud optical thickness regime determination operates in two distinct radiance domains. When the COD is very low, e.g., COD ≤ 1, the amount of radiation in the NIR is very small and the SNR of the EQW is low (e.g., “clear sky” in Fig. 3), and thus the nose plot slope SNR is too low to resolve the thickness regime. The ratio of SR440 to SR870, termed the color index, has a much higher SNR and is more reliable in this regime. This, of course, is a simple consequence of the wavelength dependence of Rayleigh scattering. For low–moderate-altitude water clouds, small–moderate SZA, and typical 440 nm ground albedos less than 0.2 (Fig. 6), our data analyses have found a hard threshold of 4 to be a sure indicator of optically very thin clouds (e.g., clear sky in Fig. 3) and a soft threshold of 2 < index < 4 to be a strong indicator (e.g., “COD < 1” in Fig. 3). When the color index is less than 2, the cloud's optical thickness is not well correlated with the index, and the algorithm must rely on the nose plot slope.

Figure 13 re-depicts the nose plot data points of Fig. 12, this time connecting a subset of points with line segments to indicate adjacent samples in a 2 min time series. If measured nose plots followed an idealized 1DRT curve as indicated in Fig. 12, the determination of thickness regime would be nearly trivial. A linear regression over a short time segment would suffice. Clearly more complex logic is required, yet it is visually evident that local coherence could be exploited. The qualitative reduction in ambiguity afforded by examining a sufficient time record suggests the use of a filter with memory. For example, for passing or evolving clouds spatially well-resolved within a narrow field of view, the thickness regime should not switch rapidly between thick and thin except possibly near the bright point (thick–thin regime boundary) where a switch is inconsequential to the retrieved COD. We implemented a time-varying hysteresis filter to effectively avoid this unwanted switching. The hysteresis action is achieved by keeping track of the maximum and minimum values of equivalent width over a predetermined time interval, typically about 3 min. In order to drive the filter toward a different thickness regime, the hysteresis limits must be exceeded. The output of the hysteresis filter is discrete ternary: -1, 0, or 1, corresponding to thick, indeterminate, or thin. Finally, this ternary variable is input to a linear, single-pole autoregressive (AR(1)) filter to afford additional smoothing. The output of this filter is thresholded and used as the thickness regime estimate.

Figure 13 includes time series plots of SR440 and algorithm retrieved COD corresponding to the connected data points in the nose plot. The dark- and medium-blue points indicate thick and thin cloud, respectively, while light-cyan points indicate clear sky. Yellow points are for SR440 values greater than the 1DRT bright-point value and are indicative of 3-D cloud effects. The green aforementioned mixed points (Sect. 3.3.3) were determined manually and are those where the values deviated strongly from the overall nose curve locus; red line segments serve to indicate where along the time series those deviations start and end. Using an instrument with small field of view and fast time response, we expect to see good temporal coherence in the data, and in fact the radiance time plot shows that the cloud optical thickness regime does not change randomly. One can see that the green-labeled points are always transitions between thin and thick cloud which did not follow the idealized nose curve through the bright point (at a radiance of about 23). The EQW values of those points are reasonable, but the radiances are lower than expected. Our supposition at this time is that some of these are due to further 3-D cloud structure effects, but leading to darker radiances rather than the bright radiances of the yellow points, while others points may be due to spatial averaging over the field of view.

This required careful time synchronization between the AERONET and TWST data times. A linear drift of 0.27 s day-1 was determined by least squares fits to the individual days with a 4 s difference between the high gain (×8) A and low gain (×1) K measurements from AERONET. The result (Fig. 14) shows that both sensors were reporting SR440 values in very good agreement. The rms difference was 0.63 (µW cm-2 sr-1 nm-1). A simple linear fit without a constant yielded a slope of 1.003 (0.0004). TWST values at high spectral radiance showed some evidence of nonlinear response.

Several conclusions follow from the very good agreement among TWST and AERONET spectral radiances. The first is the expectation of a COD comparison not influenced by TWST spectral radiance errors. As a corollary, the COD comparison should not be unduly influenced by different fields of view (1.2∘ for AERONET vs. 0.5∘ for TWST) and zenith pointing (robotic control for AERONET vs. fixed tripod with bubble level for TWST), given the close agreement over many different cloud conditions. The sensors were laterally displaced by about 3 m, and for a 1 km cloud base altitude their field-of-view footprints are 20 and 8 m. Of course, the agreement only proves consistency, not accuracy, for either sensor. The second is the radiometric stability of TWST during its TCAP deployment. This is corroborated by the stability of the four pre- and one post-test radiometric calibrations, with the photoresponsivity coefficient at 440 nm for 9 July being 98.1 % of that for 17 May.

A comparison of COD values between TWST and AERONET cloud-mode must recognize the time-sampling differences between them. For these comparisons only the 90 s average COD was available for AERONET cloud mode, which is a form of trimmed mean based on up to 10 instantaneous COD measurements during each measurement period (see Chiu et al., 2010; Sect. 2.3).

A time series comparison of COD is shown in Fig. 15. The agreement indicates that the TWST thickness regime filter is able to track the rapidly changing COD. The ensemble comparison of the COD values (Fig. 16) shows evidence of the two different types of errors in the TWST and AERONET cloud-mode algorithms: errors in cloud thickness regime and errors in numerical COD. To attempt a comparison, each plot data point represents the average of the 90 instantaneous COD measurements produced by TWST during that same 90 s period for AERONET.

Of the 244 overlapping COD values, 235 (96 %) showed the same cloud thickness regime. Some analysis was done in an effort to determine whether TWST or AERONET cloud mode was probably correct. For the nine cases where AERONET and TWST disagreed on the thickness regime, detailed nose plots were generated to see if we could visually extract more than the simple slope information used in the automated algorithm. Four of the cases produced close to the ideal nose shape, indicating that the TWST thickness regime was probably correct. For the other five cases, the nose plot was too distorted to determine the thickness regime, indicating that the TWST thickness regime was probably incorrect and should have been assigned the “unknown” label.

A linear fit of TWST to AERONET cloud-mode COD, for the 235 cases of thickness regime agreement, for a fixed zero intercept, found a slope of 0.843 (TWST reporting higher COD values) with an rms difference of COD 3.2. This was repeated while dropping the two high COD value outlier points (Fig. 16), but the slope only changed by 1 %. No evidence of a constant offset between TWST and AERONET cloud mode was found. However, the sparsity of such evidence is due to the relatively few optically thin COD cases available from AERONET, due to the secondary mission status of its cloud mode. (When skies are largely clear, AERONET executes its primary mission of aerosol optical depth and microphysical property retrieval measurements.) Therefore, another linear fit, this time with a free intercept, found a slope of 0.905 and constant offset of -2.1.

The two primary candidates for causing the observed disagreements are differences in the TWST and AERONET cloud-mode lookup tables and effects from the trimmed mean process. There may also be some residual effects due to FOV and pointing differences, although these are not expected to be large due to the very good spectral radiance agreement (Sect. 4.2.1). A partial explanation centering on the lookup tables is the difference in assumed ground albedos between the sensors. The TWST SR440-to-COD lookup table generated from MODTRAN used a weighted average of water, deciduous vegetation, dead pine, and sand albedos, resulting in an Earth albedo at 440 nm of 0.078545. On the other hand, AERONET updates its ground albedo episodically every few days from MODIS data products or a (rolling) 16-day average MODIS historical database (Chiu et al., 2012). For this dataset, the AERONET-employed albedos were lower than that assumed for TWST, varying between 0.02 and 0.04, with an average of 0.03. Most of the sample points are in the optically thick regime, and according to our albedo sensitivity discussion (Sect. 3.2.2), a lower-than-expected albedo implies TWST retrieval of higher COD values in the thick regime, consistent with the linear fits. Figure 8 depicts an approximately constant relative COD retrieval error of about 10 % per 0.1 albedo increment. The 0.05 average difference in assumed albedo therefore explains about half (0.05) of the difference between a slope of unity and the fitted slope (0.905).

It must always be kept in mind that the COD values determined by TWST and AERONET cloud mode are only equivalent 1DRT quantities. Violations of 1-D assumptions are present in nearly all our measurements to some extent. This includes (1) cases where the observed spectral radiance is greater than that possible for any 1-D cloud, (2)  cases where deviations from the ideal nose plot are too high for any 1-D cloud, and (3) the many cases where the rapid variation in spectral radiance is too high for 1-D clouds.