Priority Project "VERSUS 2"
VERification System Unified Survey 2

Last updated: 27 July 2015. Meetings and 8-year tasklists below

Project leader: initially Angela Celozzi, later Antonio Vocino and finally Roberto Bove (USAM)

1. Introduction

The development, implementation and operational use of the VERSUS package should satisfy most of the needs for traditional verification but the ever increasing model resolution as well as the current and planned ensemble systems will ask for more suitable information from verification activities. At the same time users of model forecasts will ask for more information about the real quality of the model: this means a step towards user-specific verification information.

The extension of the VERSUS package to VERSUS 2 will cover these items and will provide the COSMO community with a "state of the art" verification tool. The "System Archtecture Design (SAD)" specification document was released in May 2009 and updated in January 2010 (pdf, 0.9Mb).

2. Motivation

The aim of the VERSUS-1 Priority Project, started at the end of 2006 and concluded in February 2009 with the official delivery of the package, was the development of a common and unified verification 'library' including a Conditional Verification (CV) tool. This library will be included in the public COSMO Software along with the new software from Priority Project VERSUS2. The package and all its main features have been planned and realised with the intention to be a useful and friendly tool for the entire COSMO community for operational as well as research activities to perform traditional and conditional verification, based on conventional methods.

As the typical approach to CV consist of a selection of one or several forecast products and one or several mask variables or conditions, which would be used to define thresholds for the product verification, the flexible way to perform such a selection is to use an 'ad hoc database', planned and designed for this purpose. In this case the masks or filters can be simple or complex SQL statements. After the selection of the desired conditions, a classical verification tool to turn out statistical indexes can be used, along with a graphics package to plot the scores.

The general VERSUS architecture is basically composed of a Report Data Base Management System (RDBMS, to store observations (OBS) and forecast (FCS) data, Station Data, Configuration data to perform verification, Verification results, Scores and plots, and User permissions; MySQL in case of the initial VERSUS version), a "daemon" process (Loader) to load data from different sources into the RDBMS (File system or another DB), a process performing verifications through specific requests (Integration into the "R" statistics package), and a web-based user interface (server-client architecture).

The VERSUS1 project was divided into 10 tasks and most of them were completed (a demo version of VERSUS has been presented during the last WG 5 meeting in Zurich in May 2008). The delivery of the complete package occurred during a Workshop/Tutorial in February 2009.

In the course of the VERSUS project development, most of the COSMO members moved towards higher resolution models as well as ensemble forecasting systems (at least in their plans). The increasing resolution has, in general, the impact that objective scores for weather parameters can be worse for high resolution models than for low resolution models. Increased resolution, in fact, generally produces better defined mesoscale structures, greater amplitude features, and larger gradients, and inevitably leads to space and time errors. This is known as the "double penalty" problem. Nevertheless such a model can provide very useful information. The need for verification techniques that allow for some tolerance to reasonably small space and time errors is then clear, retaining the main feature to be capable to test the real quality of information provided by the high-resolution model and the possible connections between forecast deficiencies and specific weaknesses in the model formulation.

At the same time, as COSMO model future plans are in the probabilistic and ensemble forecast direction, suitable verification methods have to be applied. In general there are three methods to evaluate this kind of forecasts (B. Casati et al. Met. Appl. 15, 2008):

All the above verification techniques, and others specialised for extreme events and user needs, have to be analysed and then included in the VERSUS package to have a common "state-of-the-art" tool for verification for the entire COSMO community.

3. Project tasks

In a first phase (COSMO year 2009), a "User Requirements Specification" has been compiled, which will be implemented during the following phases of the project itself (COSMO years 2010 - on).

Any single phase should be concluded in just one COSMO year and a delay in the realisation of one single phase will result in a delayed start of the next one. Every year, based on the real deliverables produced, the plan will be updated accordingly along with a quick review of the requirements of the next phase, and the new phase to be developed will be fully described.

Phase 1. COSMO year 2009

It has been completed as planned in Cosmo year 2009 , with the delivery of System Design Overview document (updated in July 2009 as requested by the STC) and System Architecture Design (approved by the STC and to be updated as soon as possible). These were the tasks of PHASE 1:

Task 0
Requirements specification for VERSUS and accepted review
Task 1
Implementation of VERSUS at every service within COSMO:
Phase 2. COSMO year 2010
Task 0
Help Desk, bugs fixing activities and release of VERSUS update
Task 1
Implementation of COSI global Index:
Task 2
Implementation of weather type dependent verification and improvement of CV:
Task 3
Use of new Observation data:
Note: Task 3 will be completed with the use of Feedback Files (TASK 7) further to the decision made during the VERSUS workshop in Langen 2010, see email from PL of 25/03/2010 on "the new structure of the project plan"

Task 4
Use of gridded observation (analysis fields):
Task 5
Identification of scores for Ensemble and probabilistic forecasts:
Task 5a: Identification and full description of probabilistic scores to be implemented with description and example plots to be implemented for these probabilistic score
Task 6
Production of common statistical scores and scores graphic:
Task 7
Preparatory work on feedback files implementation:
Task 8
Statistical features like confidence intervals and the Bootstrap method
Phase 3. COSMO year 2011

Phase 3 of the project will address the implementation of the remaining PRIORITIES A and PRIORITIES B-C in the SDO document. The tasks of Phase 3 are:

Task 0
Help Desk, bug-fixing activities and release of a VERSUS update

Task 1
Improvement of VERSUS security, "plug&play" installation and Web pages:
Task 2
Final Implementation of Feedback Files in VERSUS:
Task 3
Improvement of graphical representation of scores:
Task 4
Implementation of Probabilistic Scores:
Task 5
Fuzzy Verification Toolbox and Object-oriented Verification:
outlook until June 2012:
Phase 4. COSMO year 2012

Implementation of further functionalities as from the Reference documents and consolidation of the system

Task 0
Help Desk, bug-fixing activities and release of a VERSUS update
Task 1
Consolidation and fine tuning of the system:
This task it is connected with the stress tests and in general with TASK0 and it is aimed to improve the easiness in using the system, its functionalities and performances.
Task 2
Final Implementation of Feedback Files in VERSUS:
The use of Feedback Files (FF) is important to improve upper air verification and Conditional Verification activities where large amounts of data must be manipulated. A feasibility study on how to proceed with the implementation of FF has been created by the PL team and reviewed by DWD in July 2010 and the new Database Structure for FF has been created by PL team.
Software by DWD to upload FF data is under investigation at the moment. The Task 2 from Phase 3 was stopped because of other priorities in COSMO year 2011. It will now be completed in phase 4.
Task 3
Implementation of new BUFR format and GRIB2:
The adoption by WMO of a new template for BUFR is the main reason for this task as well as the foreseen use of GRIB2 as output of COSMO models.
Task 4
Implementation of Probabilistic Scores:
Following the Guidelines from TASK5 - Phase 2/3, Probabilistic Scores will be implemented, possibly using the “R” verification package. A capability of the software to handle ensemble forecasts and verifying them through the deterministic approach, has already been added in the last version of VERSUS.
Appendix 1
Task 1 detailed functionalities:
Functionalities listed here as from the Parallel session draft GM 2011. Note that not all the requirements are listed below, but only those will be implemented until next April 2012; the remaining ones will be implemented as soon as possible (see minutes of WG5 and VERSUS parallel session GM2011). Priority has been assigned by the PL.
  1. Computing of continuous scores for precipitation.
  2. The "Batch Execution"; extended to Yearly interval.
  3. The Batch Execution” will have the functionality to be restricted to fewer activities at a time.
  4. Configuring a newer verification filled with the configuration of an existing one
  5. Configuring cross-model graphics as monthly or seasonal and consequently their batch execution.
  6. Addition of a data permanence process, able to delete observation and forecast older than a configurable time presence. This process shall be enabled or disabled depending by user.
  7. Cross model of time series.
  8. Improvement of suspect observation menu.
Phase 5. COSMO year 2013

The goal of the year is complete the most crucial developments define by participants of the last VERSUS user meeting that took place in Rome in April 2012. The Task of the Phase 3 are:

Task 0
Help Desk, bug-fixing activities and release of a VERSUS update: SUB-TASK COSI Test: This Statistical Index needs a deeper test because results so far seem to be not completely reliable. During the GM we defined a test method to verify the correct execution of this index in VERSUS.
Task 1
Consolidation and fine tuning of the system:
This task is connected closely with TASK0 and it is aimed to improve the easiness in using the system, its functionalities and to increase its performance.
Task 2
Test Area and Test Procedure Creation:
A standard installation of VERSUS and a test data set on a public machine (internet) will be provided and will serve as the common test environment.
Task 3
Implementation of Probabilistic Scores:
During this year, the PL team will finish the calculation and visualization of the ensemble and probabilistic scores in VERSUS using as a reference the EPS documents edited by F.Gofa (Task 5.a phase 3) and A. Boundel (Pre-task phase 4 ).
Task 4
Implementation of the WMO BUFR standard for observations
Phase 6. COSMO year 2013-4

The goal of the year is to complete some of the existing requirements and ensure the stability of the system. The activities planned for this year as decided during the VUS 2013 and GM 2013 are:

Task 0
Help Desk, Refinements of VERSUS code
Task 2
Implementation of Feedback Files in VERSUS
Task 3
Introduction of additional statistical techniques in VERSUS.
Task 4
EPS Refinements.
Task 5
Extension of the ASCII import capabilities of VERSUS:
Task 5.a: Feasibility study and documentation to use the ATAB file (or XML files)
Phase 7. COSMO year 2014-5
Task 0
Help Desk, Refinements VERSUS code
Task 1
VERSUS - Refinement of functionalities
Task 2
Final Implementation of Feedback Files in VERSUS
Task 3:
Implementation of VERSUS functionality to ingest (or manage) GRIB2
Task 4
Implementation of station based forecasts verification activities
In the Task6 of Phase 6 a complete feasibility study was performed together with the setup of a standard XML file format (that can be provided by Fieldextra) for post-processed station based forecasts. Main Activities:
Task 5
Introduction of additional statistical techniques compatible with VERSUS
The advanced functionalities that will be implemented within the present task will concern the introduction of additional statistical techniques in VERSUS for high resolution forecast and observation data (neighbourhood methods). The COSMO Verification Community considers the use and adaptation of external packages or libraries for pre-processing of data for verification as one of its main priorities. For this reason the main goal of the activity is the integration or adaption of pre-existing packages (e.g. Beth Ebert Fuzzy Verification Toolbox). Moreover this work will be extended to the use of gridded precipitation data from composite sources (e.g. radar, raingauges) while a pre-processing tool capable of reading Grib files (for forecast and/or observation) and producing data in suitable format for the VERSUS system itself will also be developed.
Phase 8. COSMO year 2015-6

4. References

PP VERSUS 2008 VERSUS Reference and User Manual (delivered in 2008 and available on VERSUS and COSMO website)
PP VERSUS2 2009 SAD and SDO - 2009 (available on VERSUS webpage and on COSMO web site)
MET 2008 Model Evaluation Tools Version 1.1 User"s Guide. NCAR, July 2008
Casati, B. et al. 2008 Forecast verification: current status and future directions. Met App. 15: 3-18
Ebert, E. 2008 Fuzzy verification of high-resolution gridded forecasts: a review and proposed frameworks. Met. Appl., 15, 51-64
Jolliffe IT, Stephenson DB. 2003 Forecast Verification: A Practitioner"s Guide in Atmospheric Science. John Wiley and Sons: Chichester, UK; 240pp
Nurmi P. 2003 Recommendations on the verification of local weather forecasts. ECMWF Technical Memorandum 430: 19
Stephenson, D.B. et al. 2008 The extreme dependency score : a non-vanishing measure for forecasts of rare events. Met App. 15: 41-50
Eckert P. 2009 High Resolution Verification Priority Project Report, COSMO Report. To be published
Wernli, H., P. Paulat, M. Hagen. C. Frei 2008 SAL - a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev. 136, 4470-4487

5. Meetings