Quality Control#
The functionality of the PIA Software includes a range of quality tests applied to the data sets. For the quality control the python tool SaQC (https://doi.org/10.5281/zenodo.5888547) is used. These tests are listed here:
Flags on single data points#
These tests are performed on single data points. The 0-test on instrument data is performed on the raw data with a 1-second-frequency. All other test are performed on the 3-second-averaged data.
0-test on instrument data#
The raw data from the instrument file is checked for 0-values. If all parameters (temperature, pressure, flow, dew point) have 0-values at the same time there has been a connection error. If this is the case the 0 values will be replaced with NaNs and handled as missing values for the further analysis.
Global range test on instrument data#
The instrument data is checked to lay within a valid range to enable ice formation or according to instrument specifications:
Gas and wall temperature (Ti, Tw): −60 − 0 °C
Chamber pressure (Pch): 495 − 1055 mbar
Main flow and expansion flow (Fm, Fe) during expansion and flush: 0.9 − 5.1 L/min
Fm, Fe during refill: 0 − 0.02 L/min
Dew point temperature (DP): −60 − −8 °C
Gaps in instrument data#
If there is missing data as a result from the 0-value test or for other reasons the data will be flagged with error.
Flatline test on instrument data#
This test looks for neighbouring data points with the same value. The number of points is 5 for the temperature, since the temperature is decreasing during the expansion and 10 for pressure and dew point temperature. The test is performed for:
Pch
DP
Ti during expansion
Outlier test on flow#
The flow isn’t constant over all run modes, therefore this test first filters the data for the relevant run modes (for Fe expansion, for Fm flush and expansion) and performs a Z-Score test on the flow rate. If the first or last data points of a run mode are flagged, this will be ignored because it usually takes a few seconds for the flow rate to adjust. If other outliers are detected they will be flagged with warning.
Global range test on particle concentration#
This checks if the particle concentration is in a plausible range. The range is 0 − 700′000 stdL−1. The data will be flagged with warning.
Particles during refill#
During refill the valve in front of the OPC is closed. Therefore, no particles should be detected. The first two time bins of the refill mode are ignored, as it is possible that there are still particles in the line after the valve is closed. If particles are detected in any other bins, the data will be flagged with warning. This could be a sign for a contaminated OPC.
Flags on runs#
These tests are performed once per run. If they fail the whole run will be flagged.
Test on run times#
This test checks that for each run the run times are valid. This means that the run times (start, expansion, refill, end) must be increasing. If this test fails the run will be flagged with error.
Variability of ice threshold#
For this test the mean and standard deviation of all ice thresholds in one operation are calculated. If a single ice threshold deviates more than 2 standard deviations from the mean the run will be flagged with a warning.
Large aerosol particles during flush mode#
If during flush mode more particles above the ice threshold than a certain concentration threshold are detected the run is flagged with WARNING. In this case it is assumed that there is a high ice background inside the chamber. The threshold for the INP concentration during flush mode is 50 % of the INP concentration during the expansion.
Missing OPC data#
If there is no data logged inside an opc file, it is assumed that there was a connection problem with the OPC. In this case the data will be flagged with error.
Comparison to pre-set values#
For each run the end pressure during the expansion and the mean flow during flush and expansion are defined and stored in the run file. The measured values are then compared to the pre-defined ones. The following ranges are accepted:
Pch: pre-set value ±5 %
Fm, Fe: pre-set value ±15 %
If this tests fail the run will be flagged with warning.
Temperatures during expansion#
The temperatures reached at the end of expansion should have the following order: Ti1 > Ti2 > Ti3 > Ti4 > Ti5. If this isn’t the case for the temperature sensors Ti1 - Ti4 the data will be flagged with warning. If Ti5 is higher than Ti4 it will result in an error. If this is the case it can be due to a malfunction of the temperature sensors or a wrong positioning of the sensors inside the chamber.
Global range test for duration of expansion#
The duration of the expansions has to be within the following ranges:
Ti5 below −35 °C: 9 sec - 2 min
Ti5 above −35 °C: 20 sec - 2 min
If this test fails the run will be flagged with error.
Check for supersaturation#
If supersaturated conditions are not given inside the chamber ice can not be formed. This is tested by comparing Ti5 to the dew point temperature at the beginning of the expansion. If the temperature is higher than the dew point temperature supersaturation is not given and the run will be flagged with warning.
Icing of inlet#
If the sampled air has a high humidity the inlet of the chamber could get iced. This will lead to a lower flow rate into the chamber and results in a pressure drop when the expansion starts. Therefore the end pressure is reached faster than expected. To catch this corrupted data the theoretical length of the expansion is calculated from the pressure at the beginning of the expansion \(P_0\), the pre-set values of the end pressure \(P_{end}\) and the flow during expansion \(Fe_{set}\), and the volume of the chamber \(V_{ch}\):
\Delta t_{exp} = \frac{\frac{P_0}{P_{end}} \cdot V_{ch} - V_{ch}}{Fe_{set} \cdot 60}
If the actual length of the expansion is shorter than the theoretical value, the run will be flagged with error. The test only works if a certain threshold is reached. This means that runs that are only slightly influenced by the iced inlet are not detected. To make sure that we don’t miss any biased runs all runs after three consecutive flagged runs and the five before are flagged as well. If only 5 or less unflagged runs are left in an operation the whole operation is flagged.
Gaps in OPC data#
This tests checks for time gaps during expansion in the OPC data. This is first done for the 3-sec-binned data and if one bin without OPC data is found the test will be repeated with 1 second bins. This tests is only performed if the total particle concentration of the run is higher than 180 stdL\(^{−1}\). If this test fails the run will be flagged with error.
Ti5 below 238K#
If the end temperature in the chamber is below 238K, part of the ice particles was propably created by homogeneous freezing. In this case the ice threshold finder does not function properly anymore. The run is therefore flagged with a warning.
Stripes during expansion#
Sometimes stripes or spikes of data appear in the OPC data. We don’t know what the origin of that is, but if it happens during the expansion it would lead to wrong calculations of the INP concentration. To catch these cases the INP concentration is first tested for outliers. If outliers are found, a second test is applied to these runs: If one time bin contains at least 50% of the total INP particle count, it will be flagged with an error.
Note
Please note that there have not been many cases of these spikes and the test is therefore not verified yet.
Check for liquid cloud#
If no liquid cloud is present the ice threshold finder will be placed at an arbitrary diameter and the resulting INP concentration is wrong. To check if a liquid cloud exists, it is testes if the particle number during expansion is at least 8 and if the INP concentration is less than 30% of the particle concentration. If this is not the case the run will be flagged with warning.