Priority Project "CARMA"
Common Area with Rfdbk/Mec Application

Last updated: 17 Mar 2021
get the pdf document

Project leader: Amalia Iriza-Burca (NMA)

0. Resources 1. Motivation and goals 2. Background information 3. Definition of Common Verification Activities and Requirements 4. Proposed Actions 5. Technical Requirements 6. Description of individual tasks 7. Recommendation - Future plans 8. Links to other projects or work packages 9. Risks 10. Participants 11. References 12. Estimated resources 13. Project extension

Project resources

Project meetings

• Minutes of the PP-CARMA project status (PL and WG5c, 1/9/2021)
• Minutes of the PP CARMA meeting, ICCARUS 2021 (16/03/2021)
• Minutes of the April 2020 web conference

1. Motivation and goals

The goal of this Priority Task/Project is to replace the existing VERSUS verification software environment with the MEC-Rfdbk software developed by DWD, as a Common Verification Software (CVS), in order to perform part of the verification activities in the consortium. The main use of the new CVS will be the production of the Common Plot (CP) verification while spatial verification should rather be performed with other available tools (VAST, etc.) in each service.

MEC-Rfdbk is also suitable for EPS verification, but this type of application will not be included in the current project as EPS verification is not part of CP activity, but can be considered in a next phase. The CVS that is based on MEC-Rfdbk is chosen with the intention of being a useful user-friendly tool for the entire COSMO community. It addresses the need to perform traditional point verification both for the surface and the upper air using conventional methods, which arise from operational and research activities.

Moreover, a centralized transfer and visualization of CP statistics on COSMO web server will facilitate the easier analysis of the outcome of this activity and is one of the main goals of this project.

Results of the project can be visualized on the dedicated web-page FDBK apps. Note: in the various dirs there, fill textbox "User logname" with "carma" to use carma-related datasets.

2. Background information

In 2006, the STC approved the development of a common, unified verification 'library' through the VERSUS project, which was followed by the PP-VERSUS2 in 2009. The objective of this strategic decision was to develop a common COSMO verification software package that would enable the production of homogeneous and comparable statistical results.

In recent years, partly due to technical limitations of VERSUS and the ceasing of further development, the WG VA members have considered the possibility of utilizing multiple verification modules that would not necessarily be linked to one software package (e.g. VAST software for spatial methods). In 2015, after careful consideration of the various aspects of the common verification issues, a special document "Recommendations: Strategy on Verification Tools was prepared by WG VA, in which possible scenarios for the future of Verification Tools were proposed, with the vision of adopting the new DWD verification tool using Feedback Files as the new COSMO-CVS for point verification (see wgVA: verification tools Strategy).

For the CP activity, it is essential to maintain a software as a common tool for their production as this will ensure the adoption of the same verification practices that will allow for the easier long-term monitoring of the derived results.

The Model Equivalent Calculator (MEC) software for the production of Feedback Files, and verification scripts based on the R package Rfdbk, are tools that were developed and are currently used operationally at DWD for the operational verification of both COSMO and ICON model chains. wgVA circulated a questionnaire in 2017 and a second phase in 2018. The replies strongly indicated the interest of most countries to invest resources in the near future to install and use MEC-Rfdbk system for COSMO activities or for operational national verification needs.

The proposed adoption of this system by COSMO countries for CP production will rely on DWD experience and support at the initial stage of installation through a training for the Project Support Team which will then be able to provide first level support to all countries participating in this project. The code that will be used has to be well documented but the adaptation to input datasets, the connection to local databases and the installation on various machines and any maintenance issue will be the responsibility of each member that is interested in using them.

3. Definition of Common Verification Activities and Requirements

According to the answers to the Questionnaire on Verification Tools that was filled by WG members (April 2015), the necessity to have Common Verification Software and Tools (CVS) for Common Verification Activities was underlined by the majority. Below are given the activities that such a system will serve.

Common Plot Seasonal Reports

Verification results of statistical indices for main weather parameters derived using the operational COSMO (or ICON-LAM) model implementations in each service. The domain (common or custom), resolution, statistical scores/methods, frequency and graphical representation are decided on an annual basis during WG meetings.

The main findings of this organized analysis, together with their long term trends are presented during the GM plenary session, providing a basis to track the performance of the COSMO model. A CVS allows for a homogeneous, standardized and objective way to apply, calculate and present the verification scores.

Science Plan Strategic Priorities

As included in the relevant chapter of the Science Plan, there are actions dependent on model development and subsequent verification needs. Some of these SP priorities, such as Investigation on statistical methods to identify the skill of convection-permitting and near convection-resolving model configurations, Probabilistic and Ensemble forecast verification or severe and high Impact weather verification are closely related to the existence of the necessary verification tools that can be common for the community.

Design of System

The most striking advantages of the new verification system that is proposed to be adopted for CP preparation activity, are the shortfall of data pre-processing (all data in one place, observation and forecast correctly assigned to each other, quality control done by data assimilation, small files), the fast and simple calculation of standard verification scores and the interactive browsability and online production of results. The main components of the code are analysed below and a schematic diagram of the system is also provided.

The MEC-Rfdbk verification system is based on the use of feedback files. Feedback files (FF) hold information on observations and their usage in the data assimilation (DA) system and are available for each observation system used in DA. They contain information regarding the observations (including meta-data) and corresponding model analysis, first-guess and past forecast (also ensemble) in the NetCDF format. Information from these files can be used for verification tasks (e.g. name, location, level, weight in DA, ensemble spread, talagrand index and so on).

Feedback files have a relatively small size (e.g. 4.8Mb SYNOP for COSMO-7km run for the NWP Test Suite domain) and are produced as one file for each valid-time (time window), model and observation system. The FF are produced by the Model Equivalent Calculator (MEC) within the data assimilation system or as stand-alone.

The Model Equivalent Calculator (MEC)

applies the observation operators from the data assimilation scheme (Nudging, 3Dvar, Ensemble Kalman Filter) to model forecasts (COSMO, ICON) and stores the results in verification files (NetCDF feedback file format). Observation operators are either those implemented in the DWD global data assimilation code (3dvar/EnVar/LETKF) or those implemented in the COSMO model, as used in the nudging and in the KENDA/LETKF data assimilation scheme. The results of MEC are stored in the NetCDF feedback file. The data assimilation code (3Dvar/EnVar/LETKF/KENDA) already writes the observational data (observed values, metadata, coordinates, etc.) and the analysis data (first guess and analysed values, quality control and data usage flags) in this format. In this case MEC takes these files as input, copies them to the output directory, replaces the file name prefix cof, mon, ekf by ver, and appends the model equivalents of forecasts with larger lead time.

Alternatively the fof-files (feedback file format as well) written by COSMO (in the nudging mode or in the first guess run for KENDA) may be taken as input for MEC. A further option is to use the original observational data (CDFIN, i.e. BUFR format converted to NetCDF). In these cases verification files will be generated separately for each observation type (TEMP, PILOT, SYNOP, etc). For this project, this second approach will be adopted by using a common set of observations (e.g. bufr data retrieved from ECMWFs mars archive), converting them in NetCDF format and using them as input in MEC.

While it is not the purpose of the CARMA PP, if the needed input is available, MEC can produce feedback-files for any model (including IFS), which can then be used as input to produce verification statistics using Rfdbk. As an example, DWD already produce IFS feedback-files (at ECMWF), using the DWD own set of observations. Alternatively, IFS feedback-files can be produced with any observation set by the interested users.

Rfdbk package

is an R-based code of useful functions that have been developed aiming to exploit the information contained in feedback files. The package allows: to load FF content (partially, parallel), to calculate basic verification scores (deterministic & EPS) and to perform some convenience functions like data adjustment, re-labeling, binning etc. Rfdbk exploits the functionality of the R data.table format and can therefore handle huge data tables efficiently with a concise syntax that allows to apply functions on sub-categories. Rfdbk itself is not a verification package but an assortment of R function. Base on Rfdbk there exists R scripts to quickly and reliably produce verification results. These scripts can be modified and adjusted according to the needs of each particular system that will be linked to.

According to the presentation that was given during GM2017 by F. Fundel (DWD), Rfdbk development is in the following stage:

• Verification is now done for a number of observation types (SYNOP,TEMP (radiosondes), SATOB (AMV), GPSRO (radio occultation), SCATT (scatter meter), AIREP (aircraft), PILOT (wind profiler)
• Methods: continuous and categorical
• EPS: ensemble and probabilistic
• Aggregation on e.g. sub-domains and height bins or levels or periods
• Verification of 'hindcast' mode is possible
• Cross model verification (e.g. COSMO vs. ICON, COSMO vs. IFS) is possible

R Shiny web server

On top, an interactive (R based) tool for the online visualization of verification results has been developed and will be also adopted in this project. This is based on the R Shiny web server that is currently setup on the COSMO web-site for the NWP Test suite needs. Through this capability it is possible to have on demand interactive plot web browser application, plot and arrange scores, summary score charts and browse data, with separate web based apps for each observation and verification type.

4. Proposed Actions

The migration to the new verification system that is proposed to be adopted for the CP preparation activity will be divided in three phases: training and implementation, testing and validation and CP production.

Because MEC-Rfdbk verification system is meant to be implemented by all the member countries in the consortium, a Project Support Team (PST), from NMA and HNMS, will ensure and support the implementation of the system for all the other partners. Additionally, a document for verification task specifications (common areas, models, parameters, statistical indices) included in CP activity will be prepared and the final product of the project will be based on the fulfilment of the preparation of the statistical information described inside it. The content of the CP activity is decided on an annual basis in the WG meeting during COSMO General Meetings.

The training and implementation phase will be done in two steps. First, a training (preferably @DWD) for both MEC and Rfdbk will be provided by DWD experts to the PST, including installation and run of a complete cycle of model evaluation for a test case over the CP area (including visualization of results with Shiny). This training will help the PST to implement MEC-Rfdbk system @NMA and @HNMS (with support from DWD if necessary), similar to the ones that will be installed in all participating countries. All necessary documentation for best installation and use practices will also be prepared.

The PST will also be responsible to prepare a complete example set of data (one season, one model). This data set will include model output and observations in the format required by the MEC-Rfdbk system, containing the parameters which are part of CP requirements. It is intended that, once the operational CP SPRT activity has adopted the migration from VERSUS to MEC-Rfdbk as verification framework, the observations necessary for this activity will be prepared on a seasonal basis by the same team in order to be used by all the COSMO countries.

Moreover, the PST will be responsible to develop and test scripts following the CP requirements defined in Task 1.1, which will later be provided to the other participants. These scripts will be used to produce the necessary statistical information according to the CP requirements.

After the MEC-Rfdbk system has been installed and tested by the PST (including provision of the necessary data set for validation), project activities will proceed with the second step of implementation, which is aimed at all countries in the consortium. In a second phase, a remote support through email and forum will be given from PST to all contact points of the participating services. Following these activities, the MEC-Rfdbk verification system will be locally implemented by all the member countries (NMS responsible person) in the consortium, with support from the PST and from DWD (the latter only when necessary, through the PST).

A web interface will be built/adapted on COSMO web server, by making use of Shiny R routines. This interface is intended to host CP activity outputs once the system becomes operational and will also be used during the next phases of the PP activities, for the visualization and evaluation of the statistical results obtained with the MEC-Rfdbk verification system. As this application is interactive, all users will be able to generate graphs that will be based on their model outputs over the common domains.

During the second phase of the project, after implementation in all the member countries, testing of the new MEC-Rfdbk verification system by each national service will be achieved through practical use (cross validation). A complete seasonal test (with all the necessary output for the CP reports) will be performed by the NMSs responsible persons. For this test, the data and scripts developed and provided by the PST in the first phase of the Project will be employed. The visualization of statistical results (R tables obtained with Rfdbk) will be achieved using the web interface on the COSMO server previously mentioned. A cross validation against VERSUS (or other verification system, depending on each institution’s strategy) will also be performed, using the same test data (period).

Finally, in order to enable the use of the MEC-Rfdbk system for CP activities (including seasonal verification), a series of procedures (scripts adaptation to local DBs) will be developed by all services. These procedures will be tested and evaluated for CP verification activities for a season. NMS will prepare the MEC output files based on their models for JJA 2018 season while PST will be responsible to run Rfdbk (on a central machine) and produce the statistical tables. The visualization of results obtained by will be possible on the COSMO web server (user and password access).The project will also provide a guidelines document on the implementation, use and application of the MEC-Rfdbk system for CP production.

DWD can provide support only for installation of MEC and Rfdbk in the "as-is-versions concerning both software and documentation, i.e. in the way it is implemented now at DWD.

Finally, while the opinion of members from the STC to include EPS verification is taken into consideration, the present proposal is focused primarily on replacing the existing VERSUS verification software environment with the MEC-Rfdbk software developed by DWD, as a CVS, in order to perform the CP activity in COSMO (deterministic models monitoring). As previously mentioned, the MEC-Rfdbk system is also suitable for EPS verification. After a successful completion of the current project proposal, EPS verification issues will be considered separately in a next phase (e.g. within the framework of a new PP/PT), after consideration and discussion between wgVA and wgEPS.

5. Technical Requirements

The restriction to the "as-is-version implies the following technical requirements for installing and running MEC and Rfdbk:

• As Compiler (Fortran, C), only GCC (GNU Compiler Collection) is supported, currently Version 4.9 is running at DWD (more recent versions should be ok, too - a workaround has been implemented in the code to make LETKF/MEC running with Version 5 at MeteoSuisse); older versions or other compilers such as the Intel compiler may (or may not) also work (but without support from DWD);
• NetCDF: Version 4.3 or higher is needed;
• CGRIBEX is required
• GRIB-API or ECCODES is required to be installed with the COSMO definition files; ECCODES (at least version 2.6) is preferred over GRIB-API (at least version 1.16, better 1.23), but the main point here (and with the Gribtables version) is that what is used for MEC must be compatible with (possibly the same as) what has been used to produce the forecast files (i.e. used for the COSMO (or ICON) model)
• MPI standard version 3 (issued Sept. 2012) is required; at DWD, both OpenMPI and MPICH implementations are tested; Without MPI the DACE/MEC code should compile but DWD has not tested for a long time (and will not support) MEC without MPI (which would require large processor memory or a very small model domain and limited amount of observations);
• Only Grib2 is supported; Grib1 may also run but is not tested at DWD (Grib1 can be converted into Grib2 with FieldExtra COSMO software);
• For BUFR to NetCDF conversion of observation files, the COSMO software is bufr2netcdf from ARPAE-SIMC (not bufrx2netcdf from DWD).
• For verification and visualization the installation of R>=3.1.1 with some packages from the CRAN archive and a free version of shiny-server is required.

If different software/library versions are used by the various services, then they hold the responsibility to adapt the installation files.

6. Description of individual tasks

Task 0. Administrative Tasks (ongoing)

Due to the distributed nature of the project participation team, administrative activities will be included in this task, in order to maintain a good collaboration and information flow between all participants (regular web conferences, workshops, etc.). A mailing list for the project will be used in order to support communication and information exchange between project participants.

Task 1. First Level Support Implementation and Training (completed)

As the system will be implemented by all the member countries in the consortium, a first level support implementation and training of the PST will be included, who will, in turn, ensure and support the implementation of the system for all the other partners.

Note:Since it was decided to duplicate the existing shiny platform also used for the NWP Test Suite, FTEs for task 1.7 were left unused. (0.025FTEs moved to task 3.6)

1.1 Documentation review (MEC-Rfdbk)

Preparation of a document for verification task specifications (common areas, models, parameters, statistical indices, representation) included in CP activity. The content of the CP activity is decided on an annual basis from WG members during General Meeting but it will be enriched with more representation possibilities following Rfdbk features (e.g. cross model, choice of station stratification, etc.).

1.2 Documentation preparation

Documentation preparation (MEC-Rfdbk) with all the necessary installation and use notes for the PST training.

1.3 Example dataset

Preparation of a complete example set of data (one season, one model) to be used during the training and testing period. Model output as a test bed, including parameters that are part of CP requirements will be prepared. Adaptation of observations in NetCDF format (with bufr2netcdf COSMO software). When CP activity through MEC-Rfdbk will be part of support activity, the seasonal observations necessary for CP activity will be prepared by the same team and will be disseminated to other COSMO countries.

1.4 Training

Training provided by DWD experts for first level support to the PST that will include A. Iriza-Burca, B. Maco and F. Gofa. During this training, a "clean installation of MEC, Rfdbk and Shiny routines for visualization of results will be performed on a test machine. A complete cycle of model evaluation for a test case (outcome of Task 1.3) over CP area will be performed (observation preparation, run of MEC for Feedback files preparation, run of Rfdbk for extraction of statistical indices for both continuous and dichotomic parameters, visualization of results with Shiny).

1.5 Implementation of MEC-Rfdbk

Implementation of the MEC-Rfdbk system @NMA and @HNMS. PST will have the Task to set-up systems similar to the ones that have to be installed from all participating countries with support from DWD experts if problems arise.

1.6 Adaptation of MEC-Rfdbk

Adaptation of scripts to produce the necessary statistical information for the production of the CP requirements, on a seasonal basis. Preparation and testing of scripts for semi-automatic use of the system.

1.7 Web interface

Setup of web interface with the use of Shiny R routines on COSMO server to host CP activity outputs once the system becomes operational.

Task 2. Second Level Implementation and support (ongoing)

Implementation of the MEC-Rfdbk system by all the member countries in the consortium, with support from PST. DWD support will be provided only when necessary and always through the PST.

2.1 Remote training

Remote training PST for users from each center. Dissemination of instructions, mailing list creation for problems solving, videoconferences, etc.

Note: Due to various issues encountered for the installation of the system (and use with the COSMO model), PST activities were entirely taken over by NMA (PL and Mr. B. Maco).

2.2 Implementation at all centres

Implementation of MEC-Rfdbk system in each participating center with support of PST. Almost all participants have finished the task, Rfdbk implementation still on-going in some centers.

Task 3. Cross-validation of implementation (ongoing)

Testing of the verification system implementation and training through practical use

3.1 Seasonal test

Performance of a complete seasonal test with all the necessary output for the CP reports. This task will include development/adaptation of automatic procedures (scripts adaptation to local DBs) for seasonal CP activities based on the MEC-Rfdbk system by all services. Scripts to run MEC-Rfdbk are provided from Task 1.6.

3.2 Visualization

Transfer to statistical output to COSMO web server and visualization of results through installation of Shiny server (capability developed in Task 1.6) for comparison purposes.

3.3 Comparison

Optional comparison of test output with VERSUS or any other 'home' verification system.

Note: Some of the participants have already finished tasks 3.1 and 3.3 (NMA, DWD, IMGW), while some work is on-going (CNMCA). First results are presented during the on-line PP CARMA parallel session and the plenary session. Given the interest of participants in migrating to ICON and the delay in these tasks, we propose to freeze them and move the remaining FTEs from tasks 3.1 and 3.3 to the new task 3.4.

3.4 Setup and testing of MEC+Rfdbk capabilities for ICON-LAM

Set-up of system and performance of a test with all the necessary output of ICON-LAM for the CP reports. This task has been added due to the requirements of the participants, in view of the new ICON implementations. Some centers are unable to verify COSMO because of MEC model data requirements and lack of storage resources (e.g. not all required model data are archived). This additional sub-task also takes into consideration the migration from COSMO to ICON.

Note: Some of the FTEs for this task are obtained by redirecting planned resources from tasks 3.1 and 3.3.

3.5 Setup of individual shiny server for visualization

At the moment, the shiny server on the COSMO web-site is used for visualization of results from Rfdbk (CP verification, NWP test suite). The set-up of individual shiny servers for each interested center would be advised, to facilitate the visualization of the verification results and avoid loading the COSMO shiny server unnecessarily.

Similar to the implementation of the MEC+Rfdbk system, the shiny server should first be set-up for the PST team (NMA, with support from HNMS), who will, in turn help the other participants set-up their own systems.

Note: This task is optional.

Task 4. Elaboration of guidelines for CARMA (MEC-Rfdbk) system use

Preparation of documentation on the use of the MEC-Rfdbk system (including CP activities specifications)

7. Recommendation - Future plans

After the successful completion of the project and based on the experience gained by the participants, wgVA will evaluate the necessity of a follow-up project to proceed with additional components of verification (e.g. EPS, spatial verification approaches) based on Rfdbk package features developed by DWD.

Furthermore, additional developments and applications of MEC-Rfdbk by the COSMO members beyond CP after the completion of PP CARMA are desirable but implementation and support, would require a re-assessment of resources, responsibilities and technical as well as organisational constraints which will be discussed in wgVA and decided by the STC.

8. Links to other projects or work packages

• Support Activities - NWP Test Suite: the MEC-Rfdbk system is currently installed and used on the ECMWF platform for the evaluation of new model versions before they are officially released.
• PP C2I Task 3 for the evaluation and comparison of COSMO and ICON-LAM forecasts
• Support Activities - Common Plot verification (SPRT)

9. Risks

1. DWD can provide support only for installation of MEC and Rfdbk in the "as-is-versions concerning both software and documentation. There is a high risk that adaptations needed at the NMHS to the specifics of their model might require set-ups different from the DWD one.
2. MEC - Depending on the exact MEC implementation and resource requirements which are to be determined during Task 1.1, delays in the installation of this software may occur.
3. COSMO Shiny server - The COSMO shiny server capability adapted for the visualization of results might only be accessible to restricted users, due to web security issues. As a counter measure, two people from the PST team (T. Andreadis from HNMS and B. Maco from NMA) will ensure the uploading of results on the web server.
4. Use of VERSUS - Task 3 involves a cross validation of the verification results against VERSUS (or another "home verification system). The issues often reported with bugs and slow verification process for the VERSUS system may cause some delays in the activities of this task. This subtask will be optional for participation.
5. Model data for MEC - Task 4 requires the use of model data generated by each center. If the necessity will arise for model (re)runs for MEC, this task can be delayed due to restricted human and computing resources available at the national centers.
6. Rfdbk package limitations - Spatial methods are not currently included in the Rfdbk package and the software does not allow for conditional verification applications if the required information about the observations is not in the FF, as it is not linked with a RDB.
7. Requirements for new verification scores - This is a problem most likely to appear after the end of the PP. It has been decided that any maintenance issue will be the responsibility of each member that is interested in using the MEC-Rfdbk system. However, the introduction of new verification scores to the Rfdbk system might be problematic if not performed unitary.
8. Updates to newer MEC and Rfdbk versions - in the eventuality that DWD will not provide the new MEC and/or Rfdbk versions as they are developed, either the system will be maintained as it is at the moment of the implementation or each member that is interested in using the MEC-Rfdbk system will be responsible for later development.

10. Participants

11. References

12. Estimated resources

13. Project extension

Deliverables of PP CARMA have been delayed due to unprecedented conditions caused by COVID-19. Work is on-going in successfully finishing the final 2 Tasks of the project. However, despite the considerable effort from the contributing scientists, communication impediments and difficulties in maintaining work flow have been unavoidable.

Apart from this unforeseeable situation, requirements from the participating scientists with regards to their verification needs have led to the necessity of shifting the main task of the project from COSMO to ICON, in view of the COSMO to ICON migration.

As a consequence, there is a need to include two additional sub-tasks in the current project plan and extend the project management and support tasks. While task 4, dedicated to preparation of guidelines for the use of the system will go on as planned (only timeline is proposed to change for this task), sub-tasks 3.1 and 3.3 should be replaced by a new sub-task 3.4 dedicated to ICON issues.

This additional sub-task is directly related to the COSMO to ICON migration and the need to test the new verification system for ICON-LAM implementations, rather than continue to invest much effort in adapting it for COSMO needs. As previously mentioned, this addition of the ICON-related sub-task also leads to an additional extension of the support task and administrative activities. It is important to emphasize that, due to requirements of the participants, some of the work in the additional task 3.4 has already been done. Also, a considerable amount of FTEs from tasks 3.1 and 3.3 is left unused at the moment and can instead be invested in efforts related to issues concerning verification of ICON (task 3.4).

Apart from this, an optional sub-task is also proposed, which was not foreseen at the beginning of the project. The current shiny server (COSMO web) is used for both CP activities and the NWP Test Suite verification platform. Set-up of individual shiny servers would be advised, in order to avoid loading the COSMO shiny server unnecessarily.

As a consequence, a one-year extension is proposed. For this purpose, an amount of unused 1.105 FTEs has been moved to the new tasks and an amount of maximum 1.03 additional FTEs is requested for the extension of the project. Besides lower efficiency due to the COVID-19, the FTEs requested stand for additional activities, including support and testing of the verification system for ICON-LAM.