Author: Joerg Sintermann
16.02.2016, Zurich, Switzerland
***********************************************************************************************************

This is a brief description of how to apply the Swiss miniDOAS data evaluation "miniDOAS_Revaluation". I take no responsibilities for code maintainance and correctness. Please acknowledge this work as assosicated with Sintermann et al. (2016), doi:10.5194/amt-2015-360, if you directly or indirectly make use of the approaches presented with this code. 

For details see Sintermann et al. (2016), doi:10.5194/amt-2015-360, and Volten et al. (2012), doi:10.5194/amt-5-413-2012.
***********************************************************************************************************

This data evaluation requires:

* miniDOAS raw spectral data in a certain format (see example file; spectra have been recorded with custom LabView, NI software and intensities have been summed up over the given period, here 1 minute), recorded with Swiss miniDOAS as in Sintermann et al. (2016)
* R (https://cran.r-project.org)
* installed R packages as in the script "miniDOAS_packages.R"
* a folder structure as suggested in the example

The evaluation uses several R-scripts:

* 1_miniDOAS_average_refspecs.R: calculates averaged reference spectra *.txt files from raw spectra (you'll have modify some lines of code !), required for the DOAS evaluation
* 2_miniDOAS_do_data_eval_here.R: executes the DOAS evaluation (you'll have modify some lines of code !), requires results from "1_miniDOAS_average_refspecs.R"
* miniDOAS_functions.R: function collection
* miniDOAS_evaluation_param.R: main parameters for data analysis are determined here, such as high-pass filter type and parameters
* miniDOAS_rawdata_structure.R: determines the structure of the daily *.csv file containing raw spectral information
* miniDOAS_packages.R: loads required R-packages
* miniDOAS_spectrometer_param.R: specifies the spectrometer parameters, such as linearity correction coefficients
* miniDOAS_setupeval.R: wrapper to load some scripts and derive some indices
* miniDOAS_maincode.R: actually performs the DOAS calculation 
* miniDOAS_cleanupalittle.R: removes some objects from the R-workspace after calculations

Calculations eat a lot of RAM, so I recommend to restrict data analysis to periods of <= 2 days on x86 (Windows) machines. x64 machines can process more days at once

***********************************************************************************************************
***********************************************************************************************************
In order to perform calculations, run "1_miniDOAS_average_refspecs.R" to create reference spectra (if not yet existing) and run "2_miniDOAS_do_data_eval_here.R" to perform concentration calculations.
You have to modify the beginning sections these scripts to provide information on miniDOAS model, path length, averaging periods, folder directories etc.
***********************************************************************************************************
***********************************************************************************************************

Main workflow of the DOAS procedure as described in Sintermann et al. (2016):

1) tentatively average calibration and reference spectra (I.N2, I.NH3, I.NO, I.SO2, I.dark.cal, Iref, I.dark.ref, I.dark.meas; first 5 spectra reflect conditions during calibration, Iref & I.dark.Iref reflect 'reference' condition, I.dark is for actual measurement; all these are time averages of raw spectra as defined by the user)

2) calculate calibration 'DOAS curves' (i.e. DOAS crosssections to be fitted to the measurements) for NH3, NO, SO2 (process same as for measurement data, see below) => 'dc.cal'

3) read measurement data, subtract dark spectra (must be representative for CCD temperature and integration time) (separate dark spectra are applied individually for calibration, Iref and measurement conditions), correct for CCD linearity (as specified by manufacturer), subtract 'straylight-offset' (monitored at the CCD's far end where no light should be received from the grating) => 'Iadj'

4) optional: calculate time averaged spectra (e.g. 30 minutes)

5) calculate Iadj/Iref

6) high-pass filter Iadj/Iref (be aware: filter parameters are 'expert-knowledge'-optimised for the used spectrometer, grating and measurement situations) => 'diffspec'

7) calculate measurement doas curve = ln(diffspec) => 'dc'

8) multiple linear fit dc.cal to dc using ARMA model with a set of different parameter combinations (p=0:2, q=0:2) (p=0 & q=0 corresponds to OLS fit), select best model-parameter set by corrected Akaike's information criterion, iterate that procedure for CCD shifts of +- 1 pixel, select best shift by maximum likelihood value

9) calculate concentrations, standard errors and partial correlation coefficients, using the fit's coefficients, covariance matrix, residuals and the measurement path length

10) plot calibration spectra and DOAS curves

11) optional: plot intermediate DOAS evaluation steps

12) check the analysis results as a time series; possibly repeat the concentration calculation by defining Iref at a suitable period (of lowest concentrations) in order to obtain best results (as the smoothed rfbaseline local regression filter performs on-sided, i.e. 'negative' concentrations due to absorption in Iref is described with increased uncertainty). Follow ideas like discussed in Sintermann et al. (2016) for the definition of a suitable Iref or cref

... some more auxiliary parameters are calculated and recorded
... all calculated concentrations (ĉ) are relative to cref (see Sintermann et al., 2016)



