StarDICE represents one of the initiatives focused on creating a measurement process that bridges the gap between laboratory flux standards (such as silicon photodiodes calibrated by NIST; see ) and the stars found in the CALSPEC library of spectrophotometric references . Since type-Ia supernovae (SNe Ia) and most astronomical surveys rely on the calibration of these standard stars for their measurements , successfully establishing this connection with high precision effectively addresses the calibration challenge associated with the Hubble diagram for cosmology and the study of dark energy driving the accelerated expansion of the universe (see , for a review of SNe Ia in cosmology).

The StarDICE proposal relies on the near-field calibration of a stable light source . It serves as a distant in situ reference for a compact astronomical telescope. One of the largest remaining sources of systematic uncertainty when observing stellar sources from the ground is the Earth's atmospheric transmission . It is dependent on many environmental conditions and processes, including absorption and scattering by molecular constituents (O₂, O₃, and others), absorption by PWV, scattering by aerosols, and shadowing by larger ice crystals and water droplets in clouds that is independent of wavelength and responsible for gray extinction . Current atmospheric transmission or extinction models do not integrate the possible impact of clouds. Indeed, the formation of thin clouds through the condensation of water droplets and ice can result in clouds that are extremely faint and that cannot be perceived in the visible spectrum with the naked eye. These clouds often exhibit complex spatial structures, as demonstrated in . Clouds passing through the photometric instrument's FOV result in an attenuation of stellar flux. Previously, this issue was addressed by incorporating a gray extinction correction, involving the fitting of an empirical normalization parameter for each observation. Nevertheless, as highlighted by , this approach has been proven to be insufficient due to calibration limitations arising from the dynamic and evolving nature of cloud cover conditions. To tackle this challenge in the StarDICE experiment, our solution involves employing an infrared thermal camera. This specialized equipment offers high-sensitivity radiometric measurements, capturing the sky radiance within the atmosphere's transparency window (10–12 µm). With the help of additional cloud spatial structure identification analytical capabilities, this instrument may be the key to assessing photometric observation quality and label science images with superior state-of-the-art accuracy. The primary initial objective is to generate a catalog of optical exposures from the telescope that are suitable for extracting stellar photometric flux and conducting subsequent analysis.

In recent years, numerous cloud sky–cloud segmentation algorithms have been introduced, along with the increased development of all-sky ground-based cloud monitoring stations . Indeed, cloud segmentation is a big challenge for remote sensing applications as clouds come in various shapes and forms. Most modern approaches aim to use computer vision algorithms and train them based on very specific publicly available cloud image databases such as SWIMSEG , SWINSEG , SWINySEG , WSISEG , HYTA , and TLCDD . Many proposed solutions are focused on visible RGB images. CloudSegNet is a lightweight deep-learning encoder–decoder network that detects clouds in daytime and nighttime visible color images. CloudU-Net modifies CloudSegNet architecture by adding dilated convolution, skip connection, and fully connected conditional random field (CRF; see ) layers to demonstrate better segmentation performance overall. It uses the powerful U-Net architecture originally applied to medical image segmentation. CloudU-Netv2 replaces the upsampling in CloudU-Net with bilinear upsampling, improves the discrimination ability of feature representation, and uses the rectified Adam optimizer (RAadm is a variant of the Adam stochastic optimizer that introduces a term to rectify the variance of the adaptive learning rate; see ). SegCloud has been trained based on 400 images and possesses a symmetric encoder–decoder structure and outputs low- and high-level cloud feature maps at the same resolution as input images. TransCloudSeg addresses the loss of global information due to the limited receptive field size of the filters in CNNs by proposing a hybrid model containing both the CNN and a transformer as the encoders to obtain different features. CloudDeepLabV3+ designs a lightweight ground-based cloud image adaptive segmentation method that integrates multi-scale feature aggregation and multi-level attention feature enhancement. ACLNet uses EfficientNet-B0 as the backbone, atrous spatial pyramid pooling (ASPP; see ) to learn at multiple receptive fields, and a global attention module (GAM; see ) to extract fine-grained details from the image. It provides a lower error rate, higher recall, and higher F1 score than state-of-the-art cloud segmentation models. DeepCloud uses the method of Fisher vector encoding, which is applied to execute the spatial feature aggregation and high-dimensional feature mapping of the raw deep convolutional features. CloudRAEDnet proposes a residual attention-based encoder–decoder network and trains it based on the SWINySEG dataset.

The majority of these models are typically structured using an encoder–decoder architecture, which is the primary innovation brought forth by incorporating CNNs . The encoder is tailored to acquire representational features, facilitating the extraction of semantic information while the decoder reconstructs these representational features into the segmentation mask, allowing for pixel-level classification .

Others have proposed solutions for all-sky infrared image classification. apply pre-processing steps (smoothing noise reduction, enhancement through top-hat transformation and high-pass filtering, and edge detection) before extracting features that are useful for distinguishing cirriform, cumuliform, and waveform clouds. A simple rectangle method as a supervised classifier is applied. They find a 90 % agreement between a priori classification carried out manually by visual inspection and their algorithm based on 277 images. suggested (i) a method for determining clear-sky radiance thresholds, (ii) a cloud identification method made up of a combined threshold and texture method, and (iii) an algorithm to retrieve cloud base height from downwelling infrared radiance. They showed that structural features are better than texture features in classifying clouds. proposed a three-step process: (i) pre-processing, (ii) feature extraction, and (iii) a classification method to group images into five cloud categories (stratiform, cumuliform, waveform, cirriform, and clear) based on manifold and texture features using a support vector machine (SVM; see ). Their experimental results demonstrate a higher recognition rate, with an increase of 2 %–10 % based on ground-based infrared image datasets. These methods classify clouds into separate categories based on their typology. Until now, all the previously examined approaches, while effective within their specific domains, proved to be unsuccessful when applied to our particular use case. Therefore, we propose a new deep-learning framework based on a linear classifier and U-Net architectures to identify cloud images and to detect cloud structures in real time.

Our instrument is an infrared thermal camera similar to the device – specifically the FLIR Tau2 – which operates in the LWIR band, covering the 8–14 µm range. It features a focal plane array (FPA) consisting of 640 × 512 uncooled micro-bolometers, capturing images at a frame rate of 8.33 Hz. The camera is paired with a 60 mm lens at f#1.25 aperture, resulting in a 10.4 × 8.3 degree² imaging area. The primary purpose of deploying this instrument on the equatorial mount adjacent to the StarDICE photometric telescope is to continuously assess the atmospheric conditions (specifically gray extinction) within the line of sight of the visible CCD camera during observations. The IR instrument FOV is chosen to be larger than the telescope FOV (0.5 × 0.5 degree²) to anticipate the movement of clouds in the smaller FOV of interest. Through meticulous calibration, radiative transfer calculations, and data analysis using simulations, we can extract valuable information about the sky to monitor real-time atmospheric conditions. In Figs.  and , we show the instrument mounted on a prototype equatorial mount with the necessary command and control equipment and one sample gray-scale image, respectively. We also monitor the surrounding and internal temperatures of the camera in real time to correct for temperature-related variations in sensor response. The device is controlled and commanded via the ThermalCapture Grabber USB 2.0 interface, which grants access to full 14-bit radiometric raw data. We have developed an open-access Python program, available on GitHub (https://github.com/Kelian98/tau2_thermalcapture​​​​​​​, last access: 1 January 2024; 10.5281/zenodo.15311830, ), to control the camera's functions and capture images. These images are saved in FITS format . In this study, we only consider raw analog-to-digital unit (ADU) images for simplification purposes, but the method would be identical with radiometrically calibrated images.

A substantial quantity of images is essential for the effective training and testing of both the classifier and segmentation algorithms. Our dataset comprises LWIR sky images that we captured ourselves. It encompasses a total of 3400 cloudy and clear-sky images for the classifier and 4445 sky images with cloudy-only images for the segmentation algorithm and their associated ground-truth masks. By ground truth, we mean the empirically and manually generated masks. To speed up computations and minimize memory consumption, we downsampled the original-sized images into 160 × 128 resolution. Cloudy-sky images were collected during a two-night period in early 2023 at Prades-le-Lez (43°41^′51^′′ N, 3°51^′53^′′ E) during highly variable weather conditions. Conversely, cloud-free images were obtained over a shorter period a month later.

To compensate for the lack of cloud-free images and to prevent potential biases in training due to data imbalance, we generated synthetic cloud-free images to create a composite dataset containing as many images as the cloudy dataset. These synthetic images replicate realistic observations by simulating 2D horizontal gradients, mimicking the increase in sky downwelling radiance as the camera's field of view tilts toward high zenith angles (i.e., low-elevation angles). Realistic sources of noises affecting uncooled infrared thermal cameras are introduced, including read noise, fixed-pattern noise, sky noise, and the narcissus effect. This addition ensures that the spatial noise in the synthetic images closely resembles that of actual cloud-free images. Figure  illustrates a typical cloud-free image alongside a synthetically generated one, with the spatial noise indicated for each. It is worth noting that the absolute ADU value has no impact as the data are normalized before training.

All images and masks are visually inspected. Samples presenting artifacts such as tree branches from surroundings or buildings in the FOV corners are discarded. As the camera acquisition frame rate enables us to get ∼ 8 images per second, the pre-processing algorithm included constraints on consecutive image selection based on their time series. Selected frames are taken with at least 2 s between each other to introduce a wider range of displayed clouds.

Ground-truth masks identifying cloud structure in cloud images were manually created through multiple distinct steps of non-linear stretching procedures using Astropy methods for each image in the dataset. They consist of a Boolean 2D array of the same image size, where “true” identified pixels represent cloud pixels, and “false” identified pixels represent clear-sky areas. This step has been partially automated. Binary masks that do not capture the cloud structure sufficiently have been kept aside to test segmentation model performance. Figure  depicts three raw images with their associated manually generated ground-truth cloud masks for training purposes.

Furthermore, we performed multiple random augmentations (e.g, flip, shear, rotate, shift, and zoom) on each original image to artificially enlarge the size of each dataset and to reduce overfitting . All augmented images are produced through the random sequential applications of these five distinct operations to initial images. These operations are executed with a random varying degree of intensity contained in specific ranges. Random rotations are applied within an amplitude ranging from -45 to +45°. Shear is introduced with a random magnitude ranging from -0.2 to +0.2. Shifting operations are applied to between 0 and 50 pixels in terms of both width and height to avoid the generation of unrealistic symmetric structures. Zoom operation is applied within the range of 1 to 3. No other transformation, such as histogram equalization or contrast enhancement, is applied to prevent any bias or alteration in the segmentation performance. After the selection and augmentation procedures, we conducted a visual examination of all the created sky or cloud images to ensure that they appeared to be realistic. Since all the parameters in the image augmentation process undergo controlled adjustments, our generated images closely mirror authentic sky and/or cloud scenes. Datasets for both models are split into training and validation subsets, with ratios of 80 % and 20 %, respectively.

In this section, we outline the architectural designs of two distinct deep-learning models tailored to automate classification and segmentation tasks. On the one hand, we implement a linear classifier for image classification, whose specific goal is to discriminate between cloud-free (photometric) and cloudy images (non-photometric). On the other hand, the segmentation for cloud structure detection is performed via an optimized U-Net model based on pre-classified cloudy images. The output probability map can later be thresholded according to the user's needs to produce the desired predicted binary pixel segmentation map and to allow us to obtain finer details regarding the photometric state of the field at the pixel level. Figure  illustrates the proposed deep-learning framework compared to conventional segmentation algorithms.

For our image classification model, we used a RidgeClassifier from scikit-learn to classify images as pure sky (clear) or cloudy. The RidgeClassifier is a linear classification model that employs ridge regression , a technique that introduces a regularization term. This regularization helps in addressing multicollinearity, improving the model's stability and robustness, particularly in scenarios with high-dimensional data or collinear predictors. By balancing the trade-off between bias and variance, the RidgeClassifier effectively minimizes overfitting, making it a suitable choice for our image classification task. The training process for the model involves using a dataset consisting of 2720 images, where half of them are cloud infrared images and the other half are cloud-free infrared images, each paired with appropriate ground-truth labels. The datasets are deliberately balanced to avoid any biases in the model that might favor one class over the other.

For cloud structure identification, we adopted the U-Net architecture due to its proven efficiency in semantic segmentation tasks . The U-Net model comprises an encoder and a decoder, which facilitate the capturing of context-rich features and precise delineation of cloud structures. The encoder employs convolutions and max-pooling layers to progressively downsample the input image, thereby capturing high-level features. These features are then decoded using up-convolutions and skip connections, enabling the accurate reconstruction of the segmented cloud structures. Figure  illustrates some examples of infrared cloud images alongside their corresponding ground-truth masks and predictions. Figure  depicts the architecture of the segmentation model.

The training process comprises two distinct phases, addressing the classifier model and the U-Net segmentation model. The loss function employed for training is the binary cross-entropy, which quantifies the difference between predicted probabilities and actual binary class labels for each instance in the dataset. Mathematically, given an instance's true binary label y (0 or 1) and the predicted probability p of it belonging to class 1, the binary cross-entropy loss L is calculated as follows: 1L=-1N∑iyi⋅log⁡fw(xi)+(1-yi)⋅log⁡1-fw(xi), where L is the binary cross-entropy loss. N is the total number of instances in the dataset, i is the index representing an individual instance, yi is the ith true binary label (0 or 1), and fw(xi) is the predicted probability that belongs to class 1 based on the model with parameters w. The goal of training is to minimize this loss function by adjusting the model parameter weights w to better align the predicted probabilities fw(xi) with the true labels yi.

Both the classifier and the segmentation algorithms are implemented using the Python programming language, with the aid of the Flax package , a neural network library that is part of the JAX ecosystem . Training is conducted on the GPU cluster infrastructure of the MESO@LR (https://meso-lr.umontpellier.fr/, last access: 1 January 2024) high-performance computing center, utilizing an NVIDIA Quadro RTX 6000. To expedite computations and to encapsulate the global trend, images are normalized and downsampled to the fixed resolution of 160 × 128. The models are trained using the Adam optimizer , with a batch size of 64 images. The learning rate is initiated at λ=10-3 and decreases with a cosine learning rate decay function . To prevent overfitting and expedite the training process, an early stopping mechanism is employed, which halts the training if the loss value does not exhibit a decline below a certain threshold after 15 epochs.

During the training of the U-Net model, the hyperparameter-tuning process is carried out to identify the optimal architecture configuration. This process is facilitated by the Optuna framework , which employs a sampling strategy algorithm to explore various configurations. The configurations tested cover a range of different filter numbers and sizes in the convolutional layers of the U-Net. Among the numerous configurations tested, the architecture with channels specified as 128, 64, 32, and 16 for the encoder and decoder blocks achieved the lowest loss, indicating superior performance in segmenting cloud structures.

Furthermore, a pruning strategy is integrated within the Optuna framework to curtail the exploration of sub-optimal configurations early in the training process, thereby significantly reducing the computational resources and time required for the hyperparameter-tuning process. This strategy employs a Median Pruner, which ceases the training of trials exhibiting performance lower than the median performance of completed trials . The results of the hyperparameter-tuning process reveal that the architecture with the specified channels of 128, 64, 32, and 16 outperforms others in terms of loss minimization.

In order to evaluate the performance of the proposed models, we adopt several metrics: accuracy (A), precision (P), recall (R), F1 score (F1), the area under the curve (AUC), and the binary cross-entropy loss L defined in Eq. (). All of these metrics are defined in the equations below. 2A=TP+TNTP+TN+FP+FN3P=TPTP+FP4R=TPTP+FN5F1=2⋅P⋅RP+R

6AUC=∫01RFPR-1(t)dt In the above, true positive (TP) denotes the number of correctly classified positive instances, false positive (FP) denotes the number of negative instances that were incorrectly classified as positive, false negative (FN) denotes the number of positive instances that were incorrectly classified as negative, and true negative (TN) denotes the number of correctly classified negative instances; false-positive rate (FPR) measures the model's ability to incorrectly identify negative instances as positive among all actual negatives and is calculated as FPR = FP / (FP + TN).

In our methodological framework, the initial step involves the application of Otsu's thresholding to transform infrared sky images into a binary format. This effectively segregates the cloud features from the background, providing a simplified representation where clouds are distinctly highlighted. Following this, the connected-component labeling technique, as implemented by the function skimage.measure.label, is employed. This function discerns connected regions within the binary array, where connectivity is defined by the presence of adjacent pixels sharing the same value.

This labeling process assigns a unique identifier to each contiguous cloud region, thus enabling an accurate enumeration of individual cloud formations. By comparing this automated count to visual assessments, our analysis revealed a consistent accuracy in the segmented-image counts. Figure  shows examples of cloud counting for different images along with the original, ground-truth and probabilistic segmentation images. The segmented images frequently provided counts that closely matched visual estimations, surpassing the performance of raw and binary-mask-derived counts, particularly in scenarios where images suffered from high noise levels. Such robustness underscores the advantage of our segmentation approach in providing reliable cloud quantification.

Comparing machine-learning models optimized for different types of input data can be meaningful in certain contexts, but it requires careful consideration. The most critical factor is the nature of the data. State-of-the-art models presented in Sect.  are optimized for different types of data (three-channel RGB images). It may seem incoherent to compare the performance with single-channel gray-scale infrared images. Indeed, these data types have distinct characteristics, and models may perform differently based on these differences. Some models have been proposed to target infrared images with categorization tasks. Considering whether the models can be adapted or fine-tuned to work with both RGB and infrared data is challenging. This might involve multi-modal learning approaches or transfer-learning techniques , which are not within the intended purpose of this work.

Nevertheless, we attempt to evaluate the robustness of our segmentation model by testing its ability to generalize to other datasets, including SWIMSEG , SWINSEG , and WSISEG .

We transform RGB images into gray-scaled images with the OpenCV color conversion method COLOR_RGB2GRAY, defined by the following equation: 7RGB→gray=0.299⋅R+0.587⋅G+0.114⋅B, where R, G, and B are, respectively, red, green, and blue channels of the input color image. Metrics for each dataset are summarized in Table .

While the results indicate accurate recognition of most cloud structures in the images by our models, applying a model trained on our dataset directly to another dataset yields less satisfactory performance due to the sub-optimal transformation of color images to gray scale. The method's efficiency is hindered by the strong blue color channels resulting from Rayleigh scattering , particularly affecting its performance based on publicly available datasets transformed in this manner. Further efforts are required to enhance the conversion of RGB color to gray-scale images, aiming for a comparable contrast to infrared thermal images. Still, our framework demonstrates satisfactory results when exclusively trained on the modified images (FS in Table ) as opposed to our original gray-scale images.

The segmentation model occasionally exhibited superior performance compared to the ground-truth masks, as evidenced by examples in Fig. . Notably, in this instance, the ground-truth mask failed to identify certain sky patches on the left side of the image. Conversely, the segmentation model's prediction demonstrated a non-zero probability of the existence of sky patches in those areas. This discrepancy arises from the model's utilization of a non-linear mapping technique employing a multi-layer perceptron with ReLU activations. This mapping aims to transform a normalized continuous pixel array into a binary pixel array. Consequently, regions containing sky pixels (denoted by low pixel values) possess the potential to be assigned a low probability value for cloud presence, even when the ground-truth mask assigns certainty (a value of 1) to those patches as cloud-covered areas.

To validate its effectiveness, the segmentation model is evaluated against the conventional Otsu algorithm with the validation subset from our own LWIRISEG dataset. Otsu's method consists of an adaptive thresholding algorithm that automatically computes the threshold from the image histogram distribution without parameters, supervision, or any prior information . Figure  depicts some typical comparison results between the two methods and the ground-truth masks given to the deep-learning model for training. The metrics defined in Sect.  are computed with Otsu's algorithm and are presented in Table . This demonstrates that the proposed deep-learning modified U-Net model performs significantly better than Otsu's algorithm, with mean pixel accuracies being 95.17 % and 59.16 %, respectively. Perfect precision of 100 % means that Otsu's algorithm does not produce any false positives, implying that it is overly conservative in making positive predictions. As noted by , the primary reason for the sub-par performance of Otsu's algorithm is its reliance on pixels of the same class having similar gray values, which contradicts the characteristics exhibited by clouds. These experimental findings validate the effectiveness of the segmentation model, highlighting its practical significance for upcoming observations.

In our analysis of infrared sky images, we assessed the suitability of linear classifiers against more complex deep-learning models. Employing linear classifiers such as SVM, logistic regression, perceptron, and ridge regression, optimized via stochastic gradient descent (SGD) with l2 regularizations, we aimed to prevent overfitting and maintain model simplicity. Among these, ridge regression emerged as the top performer.

Dimensionality reduction techniques supported these findings. Principal component analysis (PCA) demonstrated that two principal components could explain a significant portion of the variance (96.1 %), implying that the data are almost linearly separable. However, non-linear dimensionality reduction through uniform manifold approximation and projection (UMAP) perfectly segregated cloudy from cloud-free images, indicating that, while the data are nearly linearly separable, non-linear methods provide strong separation, as shown in Fig. .

In summary, for the task at hand, deep learning models such as ResNet may seem excessive. A well-tuned linear model, particularly ridge regression, is equally effective, if not more so, due to its interpretability and simplicity. The near-linear nature of the data suggests that simpler models could suffice for such classification challenges.

The framework established in this paper is one subpart of the StarDICE data processing operations. This will serve the rest of the analysis by identifying and classifying the quality of photometric exposures performed in parallel.

The work undertaken in this paper will be used and the associated module will include the following operations for analysis: (i) classifying infrared sky images obtained by the imaging system in real time, (ii) analyzing cloud-labeled sky images and deriving the corresponding cloud structure and cover using the segmentation algorithm, and (iii) generating alerts or flags in accordance with the results.

As mentioned in Sect. , we were able to train a model based on our gray-scale images and another model based on RGB images, and both models produced great results. However, using a model trained on gray-scale images to predict masks for RGB images or even RGB images transformed to gray scale yielded poor results. Additionally, training a model on all the data resulted in sub-optimal performance. As a future endeavor, we can explore the usage of a multi-modal deep-learning model that can work with both RGB and gray-scale images.

Improving the accuracy and robustness of our framework could involve further training on a larger dataset in more variable conditions. With the upcoming remote-operation capabilities of the telescope system expected to yield a substantial volume of data next year, we anticipate capturing a broader spectrum of sky atmospheric conditions. In this case, a single network could predict two types of outputs (e.g, pixel segmentation map and a metric describing image quality). Still, additional effort will be necessary to categorize the images based on the varying cloud coverage types.

Finally, the standard U-Net model lacks temporal correlation. No information about the displacement of the cloud is taken into account. We could gain in model accuracy by incorporating the temporal information using recurrent neural network (RNN)-based algorithms or the temporal U-Net to effectively model temporal information in sequences .