Project resources

1. Summary

A new PP APOCS is proposed with the aim of the application of alternative weather data acquired by the Personal Weather Stations (PWS) and other Opportunistic Sensors (OS) into research and operations activities at the level of national weather services (NWS).

The initial initiative in this area performed in the Priority Task EPOCS concentrated on the survey of the available PWS networks in the European area, the development and testing of the available QC software mainly for the quality check of the precipitation data, and finally the preparation and verification of the rainfall composite products.

The main scientific aim of this Project is focused on building up useful database with PWS for research purposes; application and testing of the developed RainGaugeQC and Titan-Quality Control (QC) software packages and on evaluation of the quality of different physical parameters including precipitation, temperature and humidity; and finally the application of the products developed from the PWS data to model assimilation and numerical forecast verification. The Project plan fits WG5 (currently WG:V/A) Guidelines (2021), in respect to short term (exploitation of spatial verification methods) and long term (utilization of non-conventional observations) area activities.

The PP is involving multiple COSMO Partners (IMGW-PIB, CIMA, ArpaP, CNMCA, HNMS) with a total of 5.7 FTE, plus 0.2 FTEs from external partner (Polytechnic of Turin).

2. Motivation

Alternative atmospheric data sources collected by the citizen/private weather station networks which are rapidly increasing in prevalence and becoming an emerging source of valuable weather information. The low-cost, easy-to-use, consumer-grade devices are able to collect observations in real time and store them in databases with either restricted or open/public data access. Moreover, there are independent station networks organized and maintained by the governmental agencies and private business sectors with many opportunistic (in situ and remote) sensors installed on their premises.

This dense network of observations possesses the potential to capture high-resolution meteorological information, especially in the urban area where the number of stations is highest. The spatial advantage of PWS is based on their prevalence, high coverage and massive redundancy with many independent neighboring stations being available to confirm or reject a given observation. The increasing potential of PWS is documented in several publications, as e.g. Goodchild (2007), Muller et al. (2015), Nipen et al. (2020).

PWS are owned and operated by private individuals responsible for setting up proper station location, its calibration and further maintenance. If these tasks are done improperly, without complying to standardization, the large potential of crowdsourcing thanks to widespread availability of observations may be limited in terms of data reliability. In this context, there is an important prerequisite for the assessment of data quality and removal of the portion of data which is of poor quality (Fiebrich et al. 2010).

PWS data has great potential to enhance standard weather observation networks, increase quality of weather forecasts on a local scale and give positive impact on numerous consumer applications in the area of energy, transportation, agriculture or health. The number of research projects evaluating potential of PWS data is rapidly growing in recent years, especially in the field of contemporary NWP models which increase its spatial resolution to resolve the variability of weather phenomena at the local scales.

The improvements in forecast quality demand high-quality observational data for assimilation and verification purposes, as well as in machine learning postprocessing algorithms applied to reduce model forecast biases. For example, Mass and Madaus (2014) tested impact of pressure measurements from high density of smart phones on the accuracy of frontal and convective systems forecasts; the use of PWS data in model initialization analyses was explored in Madaus et al. (2014), Gasperoni et al. (2018).

3. Requirements

The main requirement of the Project is access to the high-density crowdsourcing data and to the software identified in the process of data quality control (QC):

• Database with a network of PWS data including locally operated experimental PWS
• QC software (RainGaugeQC, TITAN)

In the new Project we incorporate PWS data into the process of preparation of key model parameters important for the nowcasting and data assimilation of NWP models, forecast verification and machine-learning postprocessing. We will collect observations from the freely available PWS databases (e.g. Netatmo), supplemented with other professional observational networks operated by independent agencies and the newly created PWS from COSMO countries PWS networks. The coverage of these datasets is sufficient for the project purpose, and the data is accessed in the near-real time either directly or by programming interface (API).

4. Proposed actions

The integration of PWS observations comes with a number of challenges such as the data accessibility and its legal limitations, methods of data quality control, data reliability for the high-resolution applications. In order to overcome such issues, the following actions are proposed within the Project timeframe.

• Task 1. Development of PWS database
• Task 2. Further testing of different QC to new databases and parameters
• Task 3. Adaptation of PWS to verification over Greece
• Task 4. Application of PWS products to model data assimilation and verification

In order to ensure PWS data credibility and sustainability (Lukyanenko et al. 2020) we will utilize software developed and tested during PT EPOCS, i.e. RainGaugeQC (developed at IMGW-PIB) and free access (TITAN package).

The deliverables of the Project are presented for each subtask

The QC methods should be applicable to other high-density networks with similar characteristics.

Task 0. Project Coordination

The PP APOCS includes several activities within several well-defined research tasks. A coordination Task is dealing with the organization of meetings between Project participants, writing of reports, and the e-mail exchange.

Task 1. Development of COSMO PWS database.

An amateur station consists of an outdoor unit and a base. The external unit sends data to the database through the data receiving server in a given format and protocol. The database communicates with the server by exchanging messages in a given format. The project assumes the creation of a server receiving data from PWS stations, and the data handlers/protocols connecting server to a database. Professional amateur (higher quality than amateur) stations allow any configuration (e.g. address of a server other than the weather service). Ordinary amateur stations send files in a different format, additionally encoded by the manufacturer. Receiving data from these stations requires a parser that decodes the messages and writes the data to a database.

The IMGW-PIB CMM project assumes the use of a noSQL database (open-source Cassandra). The first tests with employee devices and the test database were successful. The project assumes a larger number of stations in internal tests and, ultimately, the possibility of adding external participants from COSMO partners and other collaborating institutions. The final stage will be data verification and visualization in the web application. Initially through swagger documentation (prose API), and ultimately on company websites.

Subtask 1.1 Survey on available PWS from COSMO employees

This task is devoted to performing a survey and availability of different PWS stations belonging to COSMO countries amateur NWS networks and collaborating institutions who will be willing to send their own data to the new database in the real-time mode. Currently there are identified at least 40 stations of the IMGW-PIB employees, there is a prerequisite to conduct a survey within every COSMO partner and collaborating institutions on how much private stations are available in total, where are they located, which are their manufacturers, data formats and transfer protocols. Effort will be given all datasets used in the various Tasks of the project to be included in the database that will be built in the following subTasks.

Subtask 1.2 Building up parser for decoding data formats

Receiving data from the PWS stations requires a parser that decodes the messages and writes the data to a database. The following formats will be considered for the PWS database within this project:

1. JSON (JavaScript Object Notation) stands out as a prominent lightweight data interchange format. In the realm of PWS data imports, its semi-structured nature facilitates seamless representation of complex data structures. This section delves into the technical intricacies of JSON, dissecting its syntax, key-value pairs, and nested structures. We explore the efficiency of JSON in encoding meteorological parameters, its support for arrays, and the role of schema flexibility in accommodating evolving data requirements;
2. XML (eXtensible Markup Language) emerges as a robust contender in the structured data landscape. This section scrutinizes XML's verbosity and hierarchical representation, showcasing its utility in encapsulating diverse PWS datasets. Technical discussions include the exploration of XML schemas, namespaces, and the flexibility to define custom data structures. We evaluate the intricacies of parsing XML, emphasizing its role in ensuring data integrity and interoperability across disparate systems;
3. CSV (Comma-Separated Values), while JSON and XML excel in hierarchical and nested data structures, this section shines a light on the technical prowess of CSV in tabular data representation. CSV's simplicity and human-readability make it a pragmatic choice for certain PWS datasets. Technical considerations include delimiter variations, escaping mechanisms, and encoding nuances. We investigate the efficiency of CSV in representing time-series meteorological data, discussing its compatibility with database systems and parsing libraries;
4. other (f.e. txt, marginal)

The development in the project API at the data-server will have security endpoints that processes the incoming/outgoing SEND/GET requests by utilizing parser decoding software.

Subtask 1.3 Adaptation or building up transfer protocols

Files may be transferred via various protocols, which could be either modified or adopted for the purpose of COSMO PWS Database. Several solutions available in the marker will be evaluated for this purpose.

1. open source (easy to transfer to server),
2. producer-close (harder to catch and store on server, need parser)

There are a few protocols which depend on the producer or manufacturer. The biggest, developed in China-Ecowitt protocol-is used by Ecowitt group weather stations to transmit weather data to servers. This protocol allows weather stations to send real-time weather data, such as temperature, humidity, wind speed, and rainfall, to a server for monitoring, analysis, and storage.

Another option is to utilize the APRS (Automatic Packet Reporting System) an amateur, automatic system used for determining the position of moving objects, based on the amateur radio technique-Packet Radio. Packet Radio is a communication technique between stations in the form of small digital information packets modulating the radio signal (often colloquially called frames). Packet Radio is based on the AX.25 protocol, which is a modification of the X.25 network protocol known from the TCP/IP network.

The system supports serial data transmissions from various types of amateur weather stations (Davis, Ultimeter, Peet Brothers, and others). It is even possible to remotely mount an Ultimeter station using only a TNC and a radio for reporting and creating charts. APRS is also perfectly suited for weather observer initiatives. Some stations in the APRS network have additional functionality of transmitting messages received from the radio network to dedicated APRS servers (IGATE – internet gateways for APRS), which collect information about APRS traffic and make it available to users via the Internet. Messages broadcasted in the APRS network usually contain geographic location, often based on a GPS receiver connected to the station, telemetry data, weather data, and short text information.

The aim of the study will be to describe the ARPS system, test it at the available stations and examine the possibility of using weather data from APRS available in Europe.

Subtask 1.4 Development database functionalities

The database developed at IMGW-PIB will also be primarily considered for research purposes, including collection and processing of the PWS data from COSMO NHMS and other partners. In order for the database to be useful and functional, the several technical aspects need to be explored and developed within database framework during the project timeframe, including:

• efficient solutions for GUI with web frameworks (e.g. React),

• drawing charts and graphs based on the data stored in the database

• application and testing of the available QC algorithms

The typical GUI functionalities include technical details of the station, user friendly access to current and historical data, data tables, time series and GIS plotting, at the example based on ECOVITT.

A similar but more user-friendly version of the GUI will be developed for the current database within the current project task. Several IMGW-PIB packages, developed in earlier projects may be adopted for these purposes.

The QC package developed in the previous PT EPOCS project (RainGauge QC) may be also adapted to the database.

Task 2. Extended testing of QC algorithms for different precipitation data sources and other parameters (temperature, humidity)

The initial Project Task EPOCS successfully ended with the development of new functionalities of the RainGaugeQC software at IMGW-PIB and application of the Titan QC software at CIMA for the QC of the precipitation. The current Priority Project extends this work with further applications of the QC software for larger available data sets, wider range data station types and additional physical parameters including temperature and humidity.

Subtask 2.1: Adaptation of the RainGaugeQC system to CML-based precipitation data

The RainGaugeQC system is designed for quality control (QC) of telemetric rain gauge data in real-time, using radar precipitation as auxiliary information. In the frame of the work in the APOCS project, RainGaugeQC will be adapted to rainfall estimates from signal attenuation in commercial microwave links (CMLs). CML-based precipitation is characterized by a spatial distribution and error structure very different from point rain gauge or spatial radar data. The CML-derived data are distributed along links with lengths ranging from a few to several tens of km, and the uncertainty in the precipitation estimation depends on the length of the link, the signal frequency, interference from moisture in the atmosphere, among others. Many of these factors are impossible to quantify. It is therefore necessary to develop a QC methodology that takes these specificities into account.

The verification of the new version of the system will be carried out using as reference:

1. data from Hellmann's manual rain gauges (daily totals),
2. climatological multi-source RainGRS Clim estimates (daily totals),
3. real-time multi-source RainGRS estimates (30-min and, in the future, 10-min totals).

Subtask 2.2: Titan lib applied to "not conventional" raingauge datasets.

The previous experience of testing and application of the open-source software package (TITAN) for a quality control of precipitation data coming from the "official" network of stations of the Italian National Civil Department has led to an understanding and knowing the potential of the tool in term of flexibility and degrees of freedom in the choice of thresholds and/or radius of influence: these values depend on the characteristics of the network and must be found empirically.

Now, it is useful and interesting to apply the QC (Titan software) also to "non-official" free networks, for instance "Meteonetwork".

Subtask 2.3: Quality control procedure for T2m and RH2m

A first study and experimental tests using Titanlib have been applied to a temperature dataset (range check, isolation check, buddy check, Spatial Consistency Test, etc.) from the "official" network of stations of the Italian National Civil Department. Additionally, a new independent R code for SCT has been developed in order to go deeper into the analysis and to provide robust elements for comparison with Titanlib results.

The primary scope of this work is to apply the SCT code to the official temperature dataset and to study and investigate the results. Furthermore both Titalib and SCT R code can also be applied to relative humidity (RH) official dataset for the quality control.

A new quality control procedure will then be developed in Python specifically for NetAtmo data and compared to the Titanlib package.

The next step will be to implement and apply the different QC techniques (Titan lib,R code and Python script) to the "non-official" networks of "Meteonetwork" and "Netatmo". The results will be evaluated to understand the limits and opportunities of the citizen datasets and to determine if temperature and relative humidity data could be useful for scientific purposes (assimilation, urban heat island).

TASK 3. Adaptation of private/citizen datasets for verification purposes over Greece

Subtask 3.1: Titan lib applied to meteorological data in the Greek domain

Installation testing and application of the open-source software package TITAN (www.github.com/metno/titan) for a quality control of temperature and precipitation data over the Greek domain. Firstly, effort will be made to gather meteorological datasets from private and crowdsourced data, focusing mainly on areas that the coverage of the Hellenic National Meteorological Service (HNMS) network is inadequate. The terms of operational/research use of such data will be identified.

Secondly, application of the open-source software package TITAN for a quality control of ground measurements from the HNMS "official" network od stations, and "non-official" networks such as the ones from the National Observatory of Athens (meteo.gr) and the Hellenic Centre for Marine Research (www.hcmr.gr). The scope of this activity will be to set up an optimised methodology (buddy check, isolation check etc.) for production of a reliable-quality controlled-homogenised dataset of point measurements that can be used for the creation of gridded datasets (Task 3.2).

Subtask 3.2: Climatology based Interpolation methodology of precipitation data from various sources

A verification procedure must be based on equally detailed observations as high-resolution model forecasts that are also realistic in areas where HNMS ground based observations are not available. Precipitation observations from "official" and "unofficial" networks together with remote sensing data can only increase the accuracy of the "true" location of rainfall events.

An approach to create precipitation gridded datasets for areas with complex terrain as Greece is suggested in this subTask, which is based on an interpolation method that uses long climate data series to determine the geographical characteristics that this parameter is best correlated with, as well as remote sensing estimates as background information to cover the areas where observations are insufficient. Based on previous experience with other meteorological parameters, the MISH method, which is a meteorological interpolation system designed to use all meteorological, climatological and basic information of the geophysical environment of the area of interest, will be applied for the production of gridded observations.

The modelling of a long climatological time series will produce correlations (optimal parameters per month) with the topographic characteristics for a grid with a resolution of ~0.008◦. This will be followed by a process of interpolating surface observations (PWS from Task 3.1 and from the official network), using the results of the modeling and the optimal statistical parameters of the corresponding month, as well as using additional background fields from precipitation estimates from satellite (H-SAF).

Task 4. Application of PWS based gridded products

The current Task is dedicated to the application of PWS data and PWS based products as an input for the numerical weather models (e.g. nowcasting or rapid update cycle-RUC models) and for the verification of model results in the sensitive areas (e,g. TERRA-URB simulations) and for exploring advanced techniques of spatial verification of precipitation.

Subtask 4.1: Improving RainGRS+ precipitation estimates by combining different types of data in a more efficient and seamless manner.

The RainGRS+ system, developed as part of the PT EPOCS project, requires improvements related to the specifics of the additional data included in the extended "+" version of the system, in particular the inclusion of OPERA radar composite maps. These data, in addition to their lower spatial resolution (approximately 2 km), have a different quality, which depends on the institutions providing their data and the QC algorithms they use. This requires the development of a technique to combine radar data of different origins in order to achieve a seamless merging of POLRAD+ and OPERA data.

In the frame of further work, a case study of implementation of the methodology for incorporating PWS and CML-based precipitation data into RainGRS+ system will be performed. Verification of the results will be carried out using precipitation from Hellmann's manual rain gauges (daily totals) as a reference.

In addition, the RainGRS+ system will be adapted to the new domains of the NWP model versions COSMO2.8 and ICON2.5.

Subtask 4.2: Spatial verification techniques based on gridded precipitation observations enhanced with crowdsource data

Spatial verification methodologies are subsequently applied to certain convective events that accentuate the relative skill of high resolution forecasts in revealing characteristics in the precipitation patterns such as structure and intensity that can be applied with equally detailed gridded observation fields.

Subtask 4.3: Application of PWS to NWP model verification in urban areas.

Surface properties in NWP models are derived from land use classifications. Until recently, urban areas were described by one urban class. To improve the description of cities, the implementation of urban parameterization TERRA_URB in COSMO and ICON models has been made in the frame of PP-CITTA. Additionally, Local Climate Zones (LCZs) have been incorporated into the COSMO model based on ECOCLIMAP-SG land use data. On the one hand, this allows for better representation of urban heterogeneity in high-resolution NWP, but on the other hand, a denser observational network is necessary for model verification.

For this purpose, additional sources of measurement are essential in urban areas. In the frame of the subtask, COSMO and ICON-LAM simulations will be evaluated based on measurements (temperature, humidity and wind) from PWS or other professional observational networks operated by independent agencies. Additionally, simulated precipitation fields will be compared against the RainGRS+ product for severe convective events.

Subtask 4.4: Application of gridded RainGRS+ precipitation estimates for assimilation in NWP models (COSMO-RUC).

Numerical very short-range precipitation forecasts fundamentally depend on the initial conditions. Relationships among model variables in the initial conditions are important to forecasting because they determine the future development of forecasted fields, specifically the precipitation. The LHN data assimilation available in the COSMO models allows to integrate surface precipitation data from radar composites with a model over a short time interval (an assimilation window). The currently implemented Rapid Update Cycle (RUC) at IMGW-PIB is based on the COSMO model version v6.01 at 2.8km resolution with radar SRI products available from the OPERA project at the resolution of 2km, and frequency of 15 minutes. In the current Task the new dataset RainGRS (developed from the combination of the 3D radar volumes, satellite and rain gauge measurements) will be implemented. The spatial resolution of the RainGRS product is 1km and its frequency is increased to 10 or even 5 minutes.

The new version of this composite developed in PT EPOCS, denoted RainGRS+ is enhanced by inclusion of additional PWS data. The performance of the new RUC setup and the quality of the NWP forecast with the use of additional PWS data for RainGRS+ composite will be performed.

Subtask 4.5: Application of Meteonetwork PWS at CNMCA: including observations in ICON-IT DA cycle.

Inclusion of the observations from the crowdsource data network "Meteonetwork" in the operational ICON-IT analyses cycle. Production of statistics for ICON-IT performances with the new set of observations. Comparison of the obtained results with runs produced removing those same dataset from the assimilation process.

5. Risks

The proposed work is in the agreement with long term activity plans of the respective COSMO Members, and the Guidelines of the WG5 (currently WGV/A). Some tasks may be affected by the risk of a reduction of FTE resources, or delay of the research activities due to current operational needs of the COSMO Members’ met services.

Because most of PP tasks are independent from each other, the above issue of the lack of resources should not be harmful to the final development of Project outcomes. The tasks are also not dependent from other COSMO activities, including COSMO to ICON transition or delays in the development of model codes or other software.

6. Participants

CIMA: Massimo Milelli, Elena Oberto, Francesco Uboldi

CNMCA/ Italian Air Force Weather Service: Valerio Cardinali

ARPA Piemonte: Valeria Garbero

Politecnico di Torino: Tanguy Houget

HNMS: Flora Gofa, Dimitra Boucouvala

IMGW-PIB: Artur Surowiecki, Katarzyna Ośródka, Jan Szturc, Anna Jurczyk, Magdalena Pasierb, Radosław Drożdżoł, Marcin Grzelczyk, Bartłomiej Sobczyk, Adam Jaczewski, Witold Interewicz, Andrzej Wyszogrodzki

7. Tasktable & FTEs summary

Task table

Time/Tasks	2025			2026				2027
	Q2	Q3	Q4	Q1	Q2	Q3	Q4	Q1
0.0	x	x	x	x	x	x	x	x
1.1	x	x	x
1.2	x	x
1.3	x	x
1.4			x	x	x	x	x	x
2.1	x	x	x	x
2.2					x	x	x	x
2.3	x	x	x	x
3.1	x	x	x	x
3.2		x	x	x	x
4.1	x	x	x	x
4.2					x	x	x
4.3			x	x	x
4.4			x	x	x	x	x	x
4.5	x	x	x	x

FTEs summary

Task/institution	IMGW-PIB	CIMA	ARPA PIEMONTE	CNMCA	HNMS	POLITO (external)	FTE/Task
0	0.1						0.1
1.1	0.1	0.1	0.1	0.1	0.1	0.1	0.6
1.2	0.25						0.25
1.3	0.25						0.25
1.4	1.1						1.1
2.1	0.5						0.5
2.2		0.5					0.5
2.3		0.3	0.1			0.1	0.5
3.1					0.32		0.2
3.2					0.5		0.5
4.1	0.3						0.3
4.2					0.3		0.3
4.3	0.3						0.3
4.4	0.4						0.4
4.5				0.1			0.1
Total FTEs	3.3	0.9	0.2	0.2	1.1	0.2	5.9

8. Bibliography

COSMO WG5 guidelines

Dee, D. P., L. Rukhovets, R. Todling, A. M. da Silva, and J. W. Larson, 2001: An adaptive buddy check for observational quality control.
Quart. J. Roy. Meteor. Soc., 127, 2451–2471, DOI: 10.1002/qj.49712757714.

Fiebrich, C. A., C. R. Morgan, A. G. McCombs, P. K. Hall, and R. A. McPherson, 2010: Quality assurance procedures for mesoscale meteorological data.
J. Atmos. Oceanic Technol., 27, 1565–1582, DOI: 10.1175/2010 JTECHA1433.1.

Frei, C., 2014: Interpolation of temperature in a mountainous region using nonlinear profiles and non-Euclidean distances.
Int. J. Climatol., 34, 1585–1605, DOI: 10.1002/joc.3786.

Gasperoni, N. A., X. Wang, K. A. Brewster, and F. H. Carr, 2018: Assessing impacts of the high-frequency assimilation of surface observations for the forecast of convection initiation on 3 April 2014 within the Dalla–Fort Worth test bed.
Mon. Wea. Rev., 146, 3845–3872, DOI.

Giazzi M, Peressutti G, Cerri L, Fumi M, Riva IF, Chini A, Ferrari G, Cioni G, Franch G, Tartari G, Galbiati F, Condemi V, Ceppi A. Meteonetwork: An Open Crowdsourced Weather Data System.
Atmosphere. 2022; 13(6):928. DOI

Gofa, F.; Flocas, H.; Louka, P.; Samos, I. A Coherent Approach to Evaluating Precipitation Forecasts over Complex Terrain.
Atmosphere 2022, 13, 1164. https://doi.org/10.3390/atmos13081164

Goodchild, M.F. Citizens as sensors: The world of volunteered geography.
GeoJournal 2007, 69, 211–221.

Lussana, C., F. Uboldi, and M. R. Salvati, 2010: A spatial consistency test for surface observations from mesoscale meteorological networks.
Quart. J. Roy. Meteor. Soc., 136, 1075–1088, DOI.

Lukyanenko, R.; Wiggins, A.; Rosser, H.K. Citizen Science: An Information Quality Research Frontier.
Inf. Syst. Front. 2020, 22, 961–983.

Madaus, L. E., G. J. Hakim, and C. F. Mass, 2014: Utility of dense pressure observations for improving mesoscale analyses and forecasts.
Mon. Wea. Rev., 142, 2398–2413, DOI:10.1175/MWR-D-13-00269.1.

Mass, C. F., and L. E. Madaus, 2014: Surface pressure observations from smartphones: A potential revolution for high-resolution weather prediction?
Bull. Amer. Meteor. Soc., 95, 1343–1349, DOI: 10.1175/BAMS-D-13-00188.1.

Muller, C., L. Chapman, S. Johnston, C. Kidd, S. Illingworth, G. Foody, A. Overeem, and R. Leigh, 2015: Crowdsourcing for climate and atmospheric sciences: Current status and future potential.
Int. J. Climatol., 35, 3185–3203, DOI:10.1002/joc.4210.

Nipen, T.N.; Seierstad, I.A.; Lussana, C.; Kristiansen, J.; Hov, Ø. Adopting Citizen Observations in Operational Weather Prediction.
Bull. Am. Meteorol. Soc. 2020, 101, E43–E57.DOI: 10.1175/BAMS-D-18-0237.1

Zahumensky, I., 2010: World guidelines on QC procedures for data from automatic weather stations.
Guide to the Global Observing System, WMO-488, 198–207, read.