The data used in this study are from the Baseline Surface Radiation Network (BSRN), which is widely regarded as a reference-quality archive of ground-based radiation measurements . The observations are available from the official BSRN FTP server and from PANGAEA, which is an open-access data publisher for earth and environmental science. This study uses shortwave radiation quantities (Gh, Bn, and Dh) from 75 stations, and the corresponding station metadata are listed in Table . These stations span all five continents and islands in all four oceans, providing broad coverage of global radiation regimes. To balance representativeness and data volume, the most recent continuous 4-year period is selected for each station. If a station does not meet this criterion, all available records are used.

Because solar radiation quantities exhibit both annual and diurnal cycles, it is customary to use normalized indices (or k indices) in data analysis. There are multiple normalization options, and the simplest is to convert irradiances to transmittances. For Gh, Bn, and Dh, the corresponding transmittances (kt, kb, and kd) are computed as follows: 8kt=GhE0,9kb=BnE0n,10kd=DhE0, where E0=E0nμ0 is the extraterrestrial GHI; the notation for the transmittances follows the widely accepted convention in solar energy meteorology . In many references, kt is referred to as the clearness index . Similar to the three radiation quantities, the transmittances also satisfy the closure relationship, 11kt=kb+kd. In the clustering analysis below, the kt, kb, and kd time series are used. Furthermore, all nighttime data points (defined here as Z>85°) are removed. In addition, a filter requiring kt, kb, and kd to be greater than zero is applied to exclude low-sun observations, which often exhibit high noise because of radiometer design limitations .

In this study, radiation climatological features are selected to group sites with similar radiation regimes, so that QC coefficients can be developed at the group level rather than for individual sites. Defining radiation regimes has been attempted in numerous studies and is referred to as radiation zoning or radiation climate classification . In most studies, radiation regimes are defined by clustering monthly or daily mean radiation variables or the clearness index e.g.,. However, as data resolution increases, the clustering dimensionality – where each measurement can be treated as one dimension – quickly becomes computationally infeasible, even with dimension-reduction techniques. Therefore, this study uses radiation climatological features to represent high-resolution radiation data.

Three feature classes are considered: (1) zenith-angle range, (2) harmonic-analysis features, and (3) probability-density-function (PDF) features. The zenith-angle range is included because it is directly involved in QC (Eqs. –), where climatological limits are configured as a function of Z. Intuitively, using the same QC envelope (cf. Fig. ) for one site with a minimum zenith angle of 50° and another with a minimum zenith angle of 0° is inadequate. Harmonic analysis describes seasonal patterns (not limited to annual and diurnal cycles) in radiation time series, which are related to the range and shape of the scatter points in Fig. . Harmonic analysis has also long been used as a stand-alone tool for radiation climate classification . In addition, the PDFs of solar-radiation transmittances (kt, kb, and kd) are known a priori to be tied to prevailing local sky conditions. For example, arid sites are expected to show higher probability concentrations in the high-kt range. Thus, careful PDF analysis can extract statistical descriptors of radiation regimes. Because the zenith-angle range is straightforward to obtain, the extraction of the other two feature classes is described below.

With the extracted radiation climatological features, the next step involves dimensionality reduction and clustering. This process ensures that stations from locations worldwide are allocated into sensible groups, allowing for the configuration of group-specific QC limits. In this work, we employ PCA followed by Ward's hierarchical clustering. This combination is chosen for its ability to handle feature redundancy while providing a structured, multi-level grouping of stations based on their variance profiles.

PCA transforms correlated variables into a new set of orthogonal principal components. These components are linear combinations of the original features, ordered by explained variance. To avoid errors in distance calculations during clustering, rows containing missing values are removed before analysis. Following standard practice, a 90 % variance-retention threshold is applied. In this implementation, the first seven principal components are retained because they jointly satisfy this threshold.

These seven principal components are then used as inputs to hierarchical clustering with Ward's minimum-variance criterion. Unlike k-means, which requires the number of clusters to be predefined, hierarchical clustering builds a nested tree structure (dendrogram) from Euclidean distances. To determine the optimal number of clusters (k), we conduct silhouette analysis for k=4 to 10. The final value of k is selected by maximizing the mean silhouette width, which balances high intra-cluster cohesion and strong inter-cluster separation. This procedure yields a cluster configuration that effectively categorizes stations by their geographic and climatological characteristics.

To define radiation-regime-specific climatological limits for QC, two complementary outlier-detection methods are used: the ERL test and the iForest algorithm. Both methods produce binary outlier–inlier classifications, and their agreement is quantified using the F1 score. Because the ERL parameters directly determine the QC outcome, they are configured by maximizing F1. This maximization problem is solved with L-BFGS-B. In other words, the ERL parameter set that yields the highest F1 score is adopted as the optimal climatological limit for each climate category. The detailed implementation is presented in the following sections, and the complete QC-configuration workflow is shown in Fig. .

The limited-memory BFGS (L-BFGS) algorithm is a quasi-Newton optimization method designed for large-scale problems. It approximates the inverse Hessian to determine the search direction, while using a compact history of gradients and parameter updates rather than storing a dense Hessian approximation. This limited-memory strategy requires storing only an m×n matrix, where n is the number of variables and m (typically m<10) is a small number of historical updates. As a result, memory usage is substantially lower than in standard BFGS. L-BFGS is therefore well suited to high-dimensional parameter-estimation tasks. The L-BFGS-B algorithm extends L-BFGS to include simple box constraints. At each iteration, it identifies fixed and free variables (from gradient information), applies L-BFGS updates only to the free variables, and iterates until convergence .

Using the unsupervised learning framework described in Sect.  and , the 75 BSRN stations are grouped into seven distinct regimes. Figure  shows a heatmap of pairwise Euclidean distances (denoted as Ward distance, “WD”) among the PCA-transformed station features, using principal components that explain ≥ 90 % of the total variance. The color gradient represents distance magnitude, with darker shades indicating higher multivariate similarity. Notably, the diagonal blocks are much darker than the off-diagonal regions, indicating high intra-cluster cohesion and low inter-cluster similarity and thus confirming the effectiveness of the hierarchical clustering. This result is further quantified in Table , which displays the mean within-cluster and inter-cluster distances in the form of a 7×7 matrix. In Fig. , Dendrograms along the top and left margins illustrate the nested grouping structure produced by Ward's minimum-variance method. Adjacent discrete color blocks mark the final regime assignments, partitioned according to the optimal cluster count identified by silhouette analysis. This robust regionalization provides the basis for configuring regime-specific ERL tests.

To visualize the spatial distribution of the identified radiation regimes, Fig.  maps the BSRN stations and colors each site by its final cluster assignment. The resulting pattern shows a physically meaningful regionalization. This coherence is especially evident for the eight polar stations, which form a distinct, unified cluster due to their shared extreme annual irradiance cycles (i.e., polar day and night). In addition, geographically distant stations exposed to similar atmospheric conditions are consistently assigned to the same regime, as illustrated by the arid stations at Desert Rock (DRA) in the United States and Alice Springs (ASP) in Australia. This agreement further supports the physical relevance of the extracted distributional features. Considering both the spatial organization and the intrinsic properties of these features, the seven regimes can be interpreted as distinct global climatological zones. Table  summarizes the regimes, including their abbreviations, representative atmosphere/surface conditions, and defining radiation traits. A detailed discussion of each regime's physical drivers and climatological context is provided below.

Regime 1 (tropical and subtropical savannas, TSS) comprises 18 stations with a mean absolute latitude of 20.8° and primarily represents global savanna and monsoon belts located between equatorial rainforests and subtropical deserts. The defining atmospheric feature of this regime is a pronounced wet–dry seasonal bifurcation driven by migration of the intertropical convergence zone and associated monsoon dynamics. Because these stations are at low latitudes, they maintain high solar elevation angles throughout the year, yielding exceptionally high theoretical maxima for Gh. However, the surface radiation budget shows strong seasonal variability. During the dry season, stable subsiding air linked to subtropical high-pressure systems allows extended periods of intense, largely unattenuated Bn. By contrast, wet-season onset brings deep convection and heavy precipitation, shifting the radiative balance so that Dh becomes dominant because of enhanced cloud scattering. Visual evidence and additional discussion are provided in Fig.  in Appendix .

Regime 2 (polar and sub-polar, PSP) includes eight stations in extreme high-latitude environments, with a mean absolute latitude of 78.2°. This cluster is rooted in polar and sub-polar regions characterized by cold, dry air masses and persistent snow/ice cover. The key physical control is the extreme annual irradiance cycle – polar day and polar night – which keeps solar elevation angles low even during summer (Fig.  in Appendix ). As a result, the surface radiation budget is strongly modulated by the high albedo of frozen surfaces. Although Bn is strongly suppressed by long atmospheric optical paths and low sun angles, Dh often dominates the radiation profile. This dominance is further reinforced by multiple scattering between the reflective surface and frequent boundary-layer ice crystals or polar stratiform clouds. It should be noted that although BAR and TIK are high-latitude Arctic stations, they are assigned to another cluster because the clustering is based on multivariate features, not on latitude alone. In fact, BAR and TIK exhibit a cloud-dominated, diffuse-heavy, beam-suppressed signature that aligns with Regime 4 (see below) rather than with the drier, clearer-beam polar sites in Regime 2.

Regime 3 (arid and semi-arid deserts, ASD) comprises eight stations located mainly in subtropical high-pressure zones, with a mean absolute latitude of 26.8°. This group includes geographically distant but climatologically similar arid and semi-arid environments, such as DRA and ASP. Atmospheric conditions are controlled by persistent subsidence from the descending branches of the Hadley circulation. This large-scale subsidence maintains stable, dry air masses, minimal precipitation, and predominantly clear skies throughout the year (Fig.  in Appendix ). As a result, the radiation profile shows weak cloud attenuation and low atmospheric moisture. These factors produce high atmospheric transmittance, yielding consistently high Bn and Gh and very low diffuse fractions. The statistical prevalence of clear-sky days makes ASD one of the most stable and radiatively intense regimes in the surface solar radiation budget.

Regime 4 (maritime cloudy mid-latitudes, MCM) consists of eight stations across mid-to-high latitudes, with a mean absolute latitude of 51.7°. These maritime and coastal environments are strongly influenced by oceanic moisture transport and frequent subpolar low-pressure passages. Consequently, atmospheric conditions are dominated by persistent marine stratocumulus and extensive frontal cloud cover. This near-continuous cloudiness efficiently scatters incoming solar radiation, producing a substantially higher baseline diffuse fraction than in continental regions at similar latitudes (Fig.  in Appendix ). Although seasonal cycles in daily irradiance remain evident, persistent attenuation lowers overall Gh and strongly suppresses Bn, making Dh a defining and consistently important component of the local surface radiation budget.

Regime 5 (temperate mid-latitudes, TML) is the largest cluster in the BSRN dataset, containing 27 stations with a mean absolute latitude of 42.7°. Its broad distribution places these sites in the temperate mid-latitudes, where the Westerlies and frequent synoptic weather systems dominate atmospheric variability. Cloud and surface conditions are therefore highly transient, alternating between clear-sky anticyclonic periods and overcast conditions linked to passing cold and warm fronts. The TML radiation profile is characterized by a clear seasonal cycle in daily irradiance superimposed on strong day-to-day variability. Although summer solar elevations permit substantial Gh, the relative contributions of Bn and Dh shift markedly with synoptic state, making TML one of the most dynamically variable regimes in terms of short-term surface radiation behavior. Additional details are shown in Fig.  in Appendix .

Regime 6 (heavy aerosol attenuation, HAA) is a highly specialized cluster containing only two stations, Gandhinagar (GAN) and Gurgaon (GUR), both in the Indian subcontinent. Unlike regimes shaped mainly by broad latitudinal gradients or synoptic variability, this local environment is defined by exceptionally high aerosol optical depth from severe anthropogenic pollution and pre-monsoon dust loading. As a result, even under nominally clear skies, strong atmospheric scattering produces an unusually high baseline Dh and pronounced attenuation of Bn (Fig.  in Appendix ). During summer, this aerosol-driven attenuation is further intensified by the arrival of the South Asian monsoon, which brings persistent cloud cover and deep convection that strongly suppress Gh. The algorithm's separation of these two stations into an independent regime – using radiation observations alone and without explicit aerosol inputs – demonstrates that the extracted climatological features robustly capture extreme scattering and attenuation processes.

Regime 7 (low-latitude anomalous, LLA) consists of a small, diverse cluster of four stations – Nauru Island (NAU), Observatory of Huancayo (OHY), São Martinho da Serra (SMS), and Tiruvallur (TIR) – representing highly localized tropical and subtropical microclimates. Unlike broader zonally organized regimes, this group is controlled by strong site-specific geographic forcings rather than latitude alone (see Fig.  in Appendix ). For example, grouping an equatorial marine site near sea level (NAU, 7 m) with a high-altitude Andean station (OHY, 3314 m) highlights the algorithm's sensitivity to anomalous radiation signatures. Atmospheric conditions at these sites are shaped by localized processes, such as persistent deep oceanic convection or strong orographic lifting, which produce highly variable insolation profiles. Their surface radiation budgets therefore deviate from typical low-latitude behavior, with large and irregular shifts between clear-sky Bn and strongly attenuated Dh. Identification of this anomalous regime further supports the need for a data-driven regionalization framework.

Before we move forward to discuss the configuration results, it should be noted that the seven radiation regimes are identified unsupervised from the BSRN data. Once this classification is fixed, it can be applied to new stations or locations as a supervised assignment step: extract the same features from 1 min Gh, Dh, and Bn, standardize and project them into the PCA space, and assign the site to the nearest regime. In the future, the same unsupervised classification framework could be extended to satellite-derived irradiance, enabling a gridded global regime map and spatially varying QC limits; this extension is not discussed further in this work.

Optimization of regime-specific climatological limits is performed using a multi-stage data-driven framework designed to separate physically plausible upper boundaries from meteorological noise. To handle the large volume of 1 min observations while targeting theoretical upper limits, we adopt a stratified sampling strategy. The dataset is first partitioned into 1° zenith-angle bins. Within each bin, the top 5 % of irradiance values for Gh, Bn, and Dh are extracted to represent the upper envelope. To preserve typical observational states and the broader distributional structure, this envelope-focused sample is supplemented with a 10 % random sample from each bin. An iForest – configured with 100 trees in a three-dimensional feature space comprising kt, kd, and kb – is deployed to assign binary classifications to each data point using a kernel-density-estimated threshold. Structural limit curves, defined by I=aE0nμ0b+const, are fitted to these boundaries. The L-BFGS-B optimizer is used to estimate coefficients (a,b) by maximizing the F1 score relative to the iForest labels. The initial values for both a and b, for each component and cluster, are set to 1; the algorithm converged for all cases.

Table  reports the optimized coefficients and corresponding F1 scores for all seven radiation regimes. The consistent F1 range of 0.98–0.99 across all three components (F1,G,F1,B,F1,D) provides strong quantitative evidence that optimization successfully identifies the physical boundaries isolated by the unsupervised framework. Coefficient patterns also reflect distinct physical regimes. For Gh, the amplitude factor aG remains relatively constrained (0.81–1.10), and the zenith-response exponent bG is stable across clusters (1.05–1.25).

The Bn coefficients further reflect atmospheric attenuation effects. Most pristine and temperate regimes have aB values in the range 0.88–0.99, whereas Regime 6 drops markedly to aB=0.72. This quantitatively captures the large beam-radiation suppression associated with high aerosol loading over the Indo-Gangetic Plain and demonstrates the method's ability to regionalize anthropogenic attenuation. In addition, bB is systematically lower than bG (0.11–0.35), consistent with the stronger zenith-angle sensitivity and faster decay of direct-beam irradiance at high solar zenith angles.

Finally, Dh limits show substantial regional variability, especially in the amplitude factor aD, which controls the theoretical upper envelope for diffuse scattering. The optimized results indicate that the upper boundary of diffuse irradiance is strongly regime dependent. For example, the stable, clear-sky conditions in Regime 3 yield a comparatively low envelope (aD=0.63). In contrast, strong-scattering environments require much higher limits to avoid falsely flagging valid data; Regime 2 is a clear example, with aD=0.92 and a steeper zenith-response exponent bD=1.11. These values quantitatively represent diffuse dominance under high-albedo, multiple-scattering conditions over polar snow and ice. Overall, the coefficient contrasts confirm that a single global QC limit cannot represent the diverse scattering physics across global radiation climates.

To evaluate the performance of the proposed climatological limits relative to the baseline BSRN ERL limits, QC outcomes are compared across all identified radiation regimes. Figure  presents a 7×3 matrix of scatter plots showing measured irradiance components versus solar zenith angle for all seven regimes. The superimposed “eyebrows” indicate the boundary differences between the static ERL and the configured climatological limits. For Gh, ERL thresholds are consistently broader than the proposed limits in all regimes, confirming the known tendency of ERL to be overly permissive. For Bn, the two limit sets are generally similar, except in Regime 6, where the configured limits are markedly more restrictive and better adapted to high-aerosol, low-beam conditions. For Dh, the comparison is regime dependent: ERL limits can be higher, comparable, or lower than the configured limits, which motivates further quantitative analysis.

Table  translates these visual differences into operational QC impacts by comparing rejection rates under the baseline ERL and the proposed climatological limits. Under standard ERL settings, QC is generally too permissive across most regions, with near-zero rejection rates in many regimes. By reducing the excess buffer above the observed physical scatter, the proposed limits tighten constraints and raise rejection rates to more informative ranges. The value of regionalization is especially clear in two regimes: polar/sub-polar (PSP) and heavy aerosol attenuation (HAA). In PSP, ERL is overly strict for the diffuse component, flagging 1.40 % (10 380 points) of Dh samples. The customized limit relaxes this boundary and reduces the Dh rejection rate to 0.04 %. In contrast, in the polluted HAA regime, global ERL fails to provide meaningful upper-bound control, rejecting none of the Gh and Dh samples because aerosol-suppressed observations remain below the ERL ceiling. The clustered limits correct this blind spot, restoring sensitivity and increasing the Gh and Dh rejection rates to 0.09 % and 0.02 %, respectively.

To further explain why ERL frequently discards data in Regime 2, we conduct a false-positive analysis for the Alert (ALE) station in the Lincoln Sea (Fig. ). An automated scan identifies a continuous 24 h period (30 May 2010) with an exceptionally high ERL rejection rate but excellent radiometric closure (Gh≈Dh+μ0Bn). On that day, the static ERL test flags 1147 of 1440 points as erroneous. However, physical validation via the closure equation shows a mean closure error of only 1.10 %, with a maximum error of 3.78 % over the full diurnal cycle. This near-perfect agreement indicates that the sensors were operating correctly and that the elevated diffuse scattering was a real atmospheric signal rather than an instrumental artifact. Thus, while standard ERL fails to accommodate intense polar scattering and causes substantial data loss, the proposed climatological limits correctly envelop these valid enhancements, reducing false positives while retaining physically justified upper bounds.