Further information on preprocesssing#

Block diagram of the preprocessing step#

Schema Focus on the preprocessing step (needs to be slightly modified)

Generalities#

The preprocessing step is made by a specific command line (inputs are listed in the page before) calling several scripts. One script is dedicated for the opening and reformatting of each of the three input data sources, with a xarray.Dataset as output, and after this, the three reformatted datasets are merged and the result is saved to a daily netCDF file after filling the corresponding metadata.

Outline of the preprocessing and different scripts involved#

  • The script ccres_disdrometer_processing.open_weather_netcdf is used to open and reformat input daily weather netCDF files from CloudNet, and outputs a xarray.Dataset :

    – the main weather variables are renamed, and zonal and meridional components of wind are computed :

Variable Name

Input Name

Output Name

Wind Speed

wind_speed

ws

Wind Direction

wind_direction

wd

Zonal Wind

u

Meridional Wind

v

Air Temperature

air_temperature

ta

Relative Humidity

relative_humidity

hur

Ground Pressure

air_pressure

ps

Precipitation Rate

rainfall_rate

ams_pr

Rain Accumulation

rainfall_amount

ams_cp

– The variables are resampled on a perfect time vector, at a 1 minute sampling. at time T, we keep the first value found in the original dataset within the \([T-30s, T+30s]\) interval

  • The script ccres_disdrometer_processing.open_radar_netcdf is used to open and reformat input daily DCR netCDF files from Coudnet, and inputs a xarray.Dataset :

    – Reflectivity (\(Zdcr\)) and Doppler velocity (\(DVdcr\)) data is extracted from the input file corresponding variables (Zh, v), from the ground to the maximum range specified in the configuration file (by default, from 0 to 2500m).

    – The values are resampled on a perfect time vector, at a 1 minute sampling. At a range \(r\), we have :

    \(Zdcr(T, r) = median(Zh([T-30s, T+30s], r)\)

    \(DVdcr(T, r) = mean(v([T-30s, T+30s], r)\)

  • The script ccres_disdrometer_processing.open_disdro_netcdf is used to open and reformat input daily disdrometer netCDF files from CloudNet ant to compute forward-modeled reflectivity that will be compared to radar measurement, plus additional scattering-related variables and metrics characterizing the droplet size distribution at each timestep.

    – The function read_parsivel_cloudnet_choice (changer le nom de la fonction pour quelque chose de plus clair) deals with the input data opening, renaming of disdrometer variables, resampling on a perfect time vector at 1 minute sampling, and add of useful metadata related to disdrometer data.

    The key variable is the droplet size distribution (”psd”), giving the number of particles at each timestep, for each diameter and speed classes. Its dimension is (time, size_classes, speed_classes), with :

    \[\begin{equation*} \begin{cases} size\_classes = 32, speed\_classes = 32 & \text{ for Parsivel²} \\ size\_classes = 22, speed\_classes = 20 & \text{ for Thies LPM} \end{cases} \end{equation*}\]

    The data opening slightly varies depending on the disdrometer model (Thies LPM or Parsivel²). The result is stored in a new Dataset, and a dimension called radar_frequencies is created, to be able to store the scattering-related variables at all the specified frequencies when they will be computed.

    Remark: The frequencies at which the code is asked to compute the scattering-related variables are 10, 24, 35 and 94 GHz, which are the frequencies that can be found within the CCRES DCR network. These values are specified in the configuration file for each site, but it is possible to add/change these frequencies in the configuration if the user wants the code to compute forward-modeled reflectivity at other frequencies.

    – The function reflectivity_model_multilambda_measmodV_hvfov computes the following values :

    • characterization of the droplet size distribution at time \(t\):

      • \(N_i(d, t)= \sum_{v \in speed\_classes}^{} psd(t,d,v), \forall d \in size\_classes\)

      • average measured droplet fall speed for each diameter class is computed and will be used to normalize several values later in the code : \(\forall d \in size\_classes : V_{measured}(d,t) = \frac{1}{N_i(d,t)} \sum_{v \in speed\_classes}^{} psd(t, d, v) * v\)

      • it is also possible to use modeled droplet fall speed (as a function of the diameter) for the normalisation of the values computed later in the code. We use the formula given by Gun and Kinzer (1949) : : \(V_{model}(d) = 9.40 \times \left( 1 - \exp\left( -1.57 \times 10^3 \times \left( d \times 10^{-3} \right)^{1.15} \right) \right), \forall d \in size\_classes\)

        Other formulas could be implemented in the code as alternatives but it is not done yet. A field in the configuration file is ready for the chose of the formula used for modeling droplet fall speed as a function of diameter, when the implementation will be done.

      • moments of the droplet size distribution : \(M_i(t) = \frac{1}{V(d, t)FT} \sum_{d \in size\_classes}^{} N_i(d,t) \times d^{i}\), with \(V(d)\) either computed from DSD data or modeled, \(F\) the sampling surface of the disdrometer (available in configuration file) and \(T\) the time resolution of the resampled data i.e. \(T=60s\) in our case. Moments are calculated for both modeled and measured droplet fall speed

      • \(D_m(t) = \frac{M_4(t)}{M_3(t)}\), the mean volume weighted diameter in \(m\), computed in 2 variants (modeled / measured droplet fall speed)

      • \(re(t) = 0.5 \times \frac{M_3(t)}{M_2(t)}\), the effective radius in \(m\), computed in 2 variants

      • \(LWC(t) = 10^{6} \times \frac{\pi}{6} \times M_3(t), \text{in } g . cm^{-3}\) the liquid water content, computed in 2 variants

      • \(N_0^{*}(t) = \frac{4^{4}}{\pi \times 10^{6}} \times \frac{LWC(t)}{D_m^{4}(t)}\), the total number concentration in \(cm^{-3}\), computed in 2 variants

    • scattering-related variables at time t, with the help of a T-matrix code : hypothesis of Mie backscattering and non-spherical droplets/scatterers

      • Rayleigh reflectivity : \(Z_{e_{ray}}(t) = 10 \cdot log \ (\frac{1}{V(d, t)FT} \sum_{d \in size\_classes}^{} N_i(d) \times d^{6})\), computed in two variants

      • Mie reflectivity : reflectivity is calculated with the assumption of Mie backscattering, for spherical droplets. The values are computed in 8 variants : 4 frequencies \(\times\) 2 variants of computation for droplet fall speed.

        The function compute_bscat_mie() in the script ccres_disdrometer_processing.ccres_disdrometer_processing.scattering is dedicated to computing backscattering coefficients under the assumption of Mie scattering. It takes as input the vector of disdrometer size classes, the wavelength at which to compute the values, and the complex index of refraction of water. It outputs a vector \(bscat_{mie}\) containing the backscattering coefficients for the different diameter classes. From there, we derive the equivalent reflectivity :

        \(Z_{e_{mie}}(t,f) = 10 \cdot log \ [\frac{1}{V(d, t)FT} \sum_{d \in size\_classes}^{} (N_i(d,t) \times bscat_{mie}(d, f))]\)

      • T-matrix reflectivity and attenuation : reflectivity and attenuation are calculated with the assumption of Mie backscattering, for non-spherical droplets. The droplets are modeled as ellipses with an axis ratio that depends on the diameter. For the diameter-dependant axis ratio we use a \(4^{th}\) order polynomial fit suggested by (Andsager et al. 1999), based on a formula given by (Beard, Chuang 1987). The values are computed in 8 variants : 4 frequencies \(\times\) 2 variants of computation for droplet fall speed.

        The function compute_bscat_tmatrix() in the script ccres_disdrometer_processing.ccres_disdrometer_processing.scattering is dedicated to computing backscattering and attenuation coefficients under the assumption of Mie scattering. It takes as input the vector of disdrometer size classes, a vector containing the axis ratios for the different size classes, the wavelength at which to compute the values, the complex index of refraction of water, and the geometry (horizontal or vertical pointing). It outputs two vectors : \(bscat_{tmatrix}\) contains the backscattering coefficients for the different diameter classes, and \(att_{tmatrix}\) contains the attenuation coefficients. From there, we derive the equivalent reflectivity and attenuation :

        \(Z_{e_{tm}}(t,f) = 10 \cdot log \ [\frac{1}{V(d, t)FT} \sum_{d \in size\_classes}^{} (N_i(d,t) \times bscat_{tmatrix}(d, f))] \ \text{in} \frac{mm^6}{m^3}\)

        \(att_{tm}(t,f) = 10 \cdot log \ [\frac{1}{V(d, t)FT} \sum_{d \in size\_classes}^{} (N_i(d,t) \times att_{tmatrix}(d, f))] \ \text{in} \frac{dB}{km}\)

        These values are computed in 16 variants : 4 frequencies \(\times\) 2 variants of computation for droplet fall speed \(\times\) two geometries (horizontal/vertical pointing)

        The reflectivity values that we will use for comparison with radar data are for vertical pointing and modeled droplet fall speed. The corresponding variable is named Zdlog_vfov_modv_tm in the output preprocessing netCDF file

        Remark : when we derive Ze_tm we remove the biggest size classes to avoid artefacts from the calculation (5 classes are truncated by default in the code)

        We can also derive the doppler velocity :

        \(DV_{tm}(t,f) = \frac{\sum_{d \in size\_classes}^{} (N_i(d,t) \times bscat_{tmatrix}(d, f))}{\sum_{d \in size\_classes}^{} (\frac{N_i(d,t) \times bscat_{tmatrix}(d, f)}{V(d,t)})} \)

Summary of the assumptions for the forward-modeling of reflectivity values#

  • The disdrometer sampling surface F varies from an instrument to an other and is given in configuration file. All Parsivel² instruments have the same value \(F=0.0054m²\). For Thies disdrometers, each instrument has a scale factor called “AU” that has to be applied to a standard surface \(F_0 = 0.0046m²\) to get the sampling area : \(F = AU * F_0\)}. F is used to normalize reflectivity data, and one needs to be careful to specify the good value as a scale factor can introduce an offset in log-reflectivity.

  • The formula used to model droplet fall speed is described in (Gun and Kinzer, 1949) and is of common use. Other formulas are given in the litterature (Khvorostyanov and Curry 2002, Atlas and Ulbrich 1977, …), that are not implemented in the code yet, but the code is ready to host their implementation if necessary (including a field in config file for the method to use)

  • For T-matrix computations, droplets are assumed to be non-spherical : elliptic, with a shape descriebd by the ratio between major and minor axes. This ratio is a function of droplet diameter, the relationship used in the code is a 4th order polynomial fil suggested by (Andsager et al. 1999) based on a formula from (Beard, Chuang 1987).

  • Droplets are assumed to be liquid. In the code we use the complex index of refraction for liquid water n = 2.99645 + 1.54866j Je crois que c’est valid @ f=94GHz and T = 25°C. Should we use n(T,f) or at least minima n(f), how much would it impact the computed reflectivity values ? Some ways to compute the complex index of refraction of water as a function of the frequency are given in the literature ; a way to calculate the index it is given for example in (Ray 1972) with the use of the equations described by (Cole and Cole 1941) and tables of values and fits for the relationship n(f) are also provided in more recent papers. The code is ready to host a refraction index function of the frequency : in the configuration file, one must specify the couples (frequency, index) and the code uses them to compute backscattering coefficients.

The output preprocessing file is the result obtained after the three reformatted datasets (DCR, disdrometer, weather station when available) are concatenated. The command line ccres_disdrometer_processing preprocess, enables to launch this preprocessing with Paths to CloudNet data files, name of the configuration file and save path as inputs.

Detailed content of the preprocessing file#

  • Coordinates :

Coordinate Name

Dimensions

Size

time

time

1440

size_classes

size_classes

32/22 according to disdrometer model

speed_classes

speed_classes

32/20 according to disdrometer model

radar_frequencies

radar_frequencies

4 by default

range

range

depends on DCR range resolution : values between ground and 2500m

  • Data variables :

Variable Name

Long Name

Unit

ws

Wind speed

m s-1

wd

Wind direction

degree

ta

Air temperature

Celsius

hur

Relative humidity

%

ps

Air pressure

hPa

ams_pr

Met station precipitation rate at 1m agl

mm/hr

ams_cp

Rainfall amount

mm

u

Zonal wind

m/s

v

Meridional wind

m/s

ams_longitude

Longitude of site

degree_east

ams_latitude

Latitude of site

degree_north

ams_altitude

Altitude of site

m

F

Disdrometer sampling area

m^2

disdro_pr

Disdrometer-based precipitation rate

mm/h

disdro_cp

Disdrometer-based cumulated precipitation since 00:00 UTC

mm

psd

Number of droplets per diameter and fall speed bins

#/bin/mn

particles_count

Number of particles in time interval

1

size_classes_width

Width of the diameter bins

mm

speed_classes_width

Width of the speed bins

m/s

disdro_longitude

longitude of disdrometer set-up

degree_east

disdro_latitude

latitude of disdrometer set-up

degree_north

disdro_altitude

altitude of disdrometer set-up

m

disdro_model

Disdrometer model

N/A

time_resolution

Time resolution of the preprocessed file

s

disdro_pr_from_raw

Disdrometer-based precipitation rate from raw data

mm/h

measV

measured fall speed at each timestep as a function of the diameter

m/s

modV

modeled droplet fall speed as a function of the diameter

m/s

psd_sum_n

Normalized precipitation size distribution

#/cm^3/mm

psd_sum

Normalized precipitation size distribution

#/cm^3/bin

Zdlin_hfov_measv_mie

Disdrometer Mie reflectivity in lin scale for measured fall drop velocity and horizontal field of view

mm^6.m^-3

Zdlog_hfov_measv_mie

Disdrometer Mie reflectivity in log scale for measured fall drop velocity and horizontal field of view

dBZ

Zdlin_hfov_modv_mie

Disdrometer Mie reflectivity in lin scale for modeled fall drop velocity and horizontal field of view

mm^6.m^-3

Zdlog_hfov_modv_mie

Disdrometer Mie reflectivity in log scale for modeled fall drop velocity and horizontal field of view

dBZ

Zdlin_vfov_measv_mie

Disdrometer Mie reflectivity in lin scale for measured fall drop velocity and vertical field of view

mm^6.m^-3

Zdlog_vfov_measv_mie

Disdrometer Mie reflectivity in log scale for measured fall drop velocity and vertical field of view

dBZ

Zdlin_vfov_modv_mie

Disdrometer Mie reflectivity in lin scale for modeled fall drop velocity and vertical field of view

mm^6.m^-3

Zdlog_vfov_modv_mie

Disdrometer Mie reflectivity in log scale for modeled fall drop velocity and vertical field of view

dBZ

Zdlin_hfov_measv_tm

Disdrometer geometric reflectivity in lin scale for measured fall drop velocity and horizontal field of view

mm^6.m^-3

Zdlog_hfov_measv_tm

Disdrometer geometric reflectivity in log scale for measured fall drop velocity and horizontal field of view

dBZ

Zdlin_hfov_modv_tm

Disdrometer geometric reflectivity in lin scale for modeled fall drop velocity and horizontal field of view

mm^6.m^-3

Zdlog_hfov_modv_tm

Disdrometer geometric reflectivity in log scale for modeled fall drop velocity and horizontal field of view

dBZ

Zdlin_vfov_measv_tm

Disdrometer geometric reflectivity in lin scale for measured fall drop velocity and vertical field of view

mm^6.m^-3

Zdlog_vfov_measv_tm

Disdrometer geometric reflectivity in log scale for measured fall drop velocity and vertical field of view

dBZ

Zdlin_vfov_modv_tm

Disdrometer geometric reflectivity in lin scale for modeled fall drop velocity and vertical field of view

mm^6.m^-3

Zdlog_vfov_modv_tm

Disdrometer geometric reflectivity in log scale for modeled fall drop velocity and vertical field of view

dBZ

Zdlin_measv_ray

Disdrometer Rayleigh reflectivity in lin scale for measured fall drop velocity and horizontal field of view

mm^6.m^-3

Zdlog_measv_ray

Disdrometer Rayleigh reflectivity in log scale for measured fall drop velocity and horizontal field of view

dBZ

Zdlin_modv_ray

Disdrometer Rayleigh reflectivity in lin scale for modeled fall drop velocity and vertical field of view

mm^6.m^-3

Zdlog_modv_ray

Disdrometer Rayleigh reflectivity in log scale for modeled fall drop velocity and vertical field of view

dBZ

attd_hfov_measv

Disdrometer attenuation for measured fall drop velocity and horizontal field of view

dB/km

attd_hfov_modv

Disdrometer attenuation for modeled fall drop velocity and horizontal field of view

dB/km

attd_vfov_measv

Disdrometer attenuation for measured fall drop velocity and vertical field of view

dB/km

attd_vfov_modv

Disdrometer attenuation for modeled fall drop velocity and vertical field of view

dB/km

DVd_hfov_measv_mie

Disdrometer Mie Doppler velocity for measured fall drop velocity and horizontal field of view

m.s^-1

DVd_hfov_modv_mie

Disdrometer Mie Doppler velocity for modeled fall drop velocity and horizontal field of view

m.s^-1

DVd_vfov_measv_mie

Disdrometer Mie Doppler velocity for measured fall drop velocity and vertical field of view

m.s^-1

DVd_vfov_modv_mie

Disdrometer Mie Doppler velocity for modeled fall drop velocity and vertical field of view

m.s^-1

DVd_hfov_measv_tm

Disdrometer geometric Doppler velocity for measured fall drop velocity and horizontal field of view

m.s^-1

DVd_hfov_modv_tm

Disdrometer geometric Doppler velocity for modeled fall drop velocity and horizontal field of view

m.s^-1

DVd_vfov_measv_tm

Disdrometer geometric Doppler velocity for measured fall drop velocity and vertical field of view

m.s^-1

DVd_vfov_modv_tm

Disdrometer geometric Doppler velocity for modeled fall drop velocity and vertical field of view

m.s^-1

m2_measv

Second order momentum for measured fall drop velocity

mm^2

m2_modv

Second order momentum for modeled fall drop velocity

mm^2

m3_measv

Third order momentum for measured fall drop velocity

mm^3

m3_modv

Third order momentum for modeled fall drop velocity

mm^3

m4_measv

Fourth order momentum for measured fall drop velocity

mm^4

m4_modv

Fourth order momentum for modeled fall drop velocity

mm^4

dm_measv

Mean diameter of the precipitation for measured fall drop velocity

mm

dm_modv

Mean diameter of the precipitation for modeled fall drop velocity

mm

re_measv

Effective radius of the precipitation for measured fall drop velocity

mm

re_modv

Effective radius of the precipitation for modeled fall drop velocity

mm

lwc_measv

Liquid Water Content for measured fall drop velocity

g/cm^3

lwc_modv

Liquid Water Content for modeled fall drop velocity

g/cm^3

n0_measv

Total number concentration for measured fall drop velocity

#/cm^3

n0_modv

Total number concentration for modeled fall drop velocity

#/cm^3

radar_longitude

Longitude of site

degree_east

radar_latitude

Latitude of site

degree_north

radar_altitude

Altitude of site

m

radar_model

Radar model

N/A

radar_frequency

Frequency of the DCR, in Hertz

Hz

radar_wavelength

Wavelength of the DCR, in meter

m

alt

Range from instrument

m

Zdcr

Radar reflectivity factor

dBZ

DVdcr

Doppler velocity

m s-1

weather_data_avail

Availability of weather data at the station

N/A

Global attributes in the output file :

Attribute Name

Value

year

[year]

month

[01-12]

day

[01-31]

location

[Location of the NF]

ams_source

[Weather station / None]

ams_pid

https://hdl.handle.net/[…]

disdrometer_source

[Name of disdrometer model : Thies Clima LNM / OTT HydroMet Parsivel²

disdrometer_pid

https://hdl.handle.net/[…]

radar_source

[Name of radar : MIRA-35 / RPG-FMCW-94 / BASTA]

radar_pid

https://hdl.handle.net/[…]

station_name

[Name of the station]

axis_ratioMethod

BeardChuang_PolynomialFit

fallspeedFormula

GunAndKinzer

title

CCRES pre-processing file for Doppler cloud radar stability monitoring with disdrometer at Palaiseau site

summary

Disdrometer ([disdrometer_source]) data are processed to derive the equivalent reflectivity factor at 4 frequencies (10.0, 24.0, 35.0, 94.0 GHz). Doppler cloud radar ([radar_source]) data (reflectivity and Doppler velocity) are extracted up to some hundreds of meters, and weather station data (temperature, humidity, wind and precipitation rate) are added to the dataset if provided. The resulting pre-processing netCDF file has a 1-minute sampling for all the collocated sensors.

keywords

GCMD:EARTH SCIENCE, GCMD:ATMOSPHERE, GCMD:CLOUDS, GCMD:CLOUD DROPLET DISTRIBUTION, GCMD:CLOUD RADIATIVE TRANSFER, GCMD:CLOUD REFLECTANCE, GCMD:SCATTERING, GCMD:PRECIPITATION, GCMD:ATMOSPHERIC PRECIPITATION INDICES, GCMD:DROPLET SIZE, GCMD:HYDROMETEORS, GCMD:LIQUID PRECIPITATION, GCMD:RAIN, GCMD:LIQUID WATER EQUIVALENT, GCMD:PRECIPITATION AMOUNT, GCMD:PRECIPITATION RATE, GCMD:SURFACE PRECIPITATION

keywords_vocabulary

GCMD:GCMD Keywords, CF:NetCDF COARDS Climate and Forecast Standard Names

Conventions

CF-1.8, ACDD-1.3, GEOMS

id

naming_authority

history

created on [date in ISO 8601] by ccres_disdrometer_processing, v X.X.X

source

surface observation from [radar_source] DCR, [disdrometer_source] disdrometer and AMS, processed by CloudNet

processing_level

2a

comment

acknowledgement

license

CC BY 4.0

standard_name_vocabulary

CF Standard Name Table v84

date_created

[date in ISO 8601]

creator_name

ACTRIS-CCRES

creator_email

ccres_contact@listes.ipsl.fr

creator_url

https://ccres.aeris-data.fr

creator_type

institution

creator_institution

institution

project

publisher_name

publisher_email

publisher_url

publisher_type

publisher_institution

contributor_name

contributor_role

geospatial_bounds

POLYGON ((lat_min, lon_min), (lat_max, lon_min), (lat_max, lon_max), (lat_min, lon_max))

geospatial_bounds_crs

EPSG:4326

geospatial_bounds_vertical_crs

EPSG:5829

geospatial_lat_min

[lat_min]

geospatial_lat_max

[lat_max]

geospatial_lat_units

degree_north

geospatial_lat_resolution

0.001

geospatial_lon_min

[lon_min]

geospatial_lon_max

[lon_max]

geospatial_lon_units

degree_east

geospatial_lon_resolution

0.001

geospatial_vertical_min

[alt_min]

geospatial_vertical_max

[alt_max]

geospatial_vertical_units

m

geospatial_vertical_resolution

0.1

geospatial_vertical_positive

up

time_coverage_start

[start epoch in ISO 8601]

time_coverage_end

[end epoch in ISO 8601]

time_coverage_duration

[Delta in ISO 8601]

time_coverage_resolution

[Time resolution in ISO 8601]

program

ACTRIS, CloudNet, CCRES

date_modified

[date of modification in ISO 8601]

date_issued

[same date a priori]

date_metadata_modified

product_version

[X.X.X]

platform

GCMD:In Situ Land-based Platforms, GCMD:OBSERVATORIES

platform_vocabulary

GCMD:GCMD Keywords

instrument

GCMD:Earth Remote Sensing Instruments, GCMD:Active Remote Sensing, GCMD:Profilers/Sounders, GCMD:Radar Sounders, GCMD:DOPPLER RADAR, GCMD:FMCWR, GCMD:VERTICAL POINTING RADAR, GCMD:In Situ/Laboratory Instruments, GCMD:Gauges, GCMD:RAIN GAUGES, GCMD:Recorders/Loggers, GCMD:DISDROMETERS, GCMD:Temperature/Humidity Sensors, GCMD:TEMPERATURE SENSORS, GCMD:HUMIDITY SENSORS, GCMD:Current/Wind Meters, GCMD:WIND MONITOR, GCMD:Pressure/Height Meters, GCMD:BAROMETERS

instrument_vocabulary

GCMD:GCMD Keywords

cdm_data_type

metadata_link

references