Priority Project "VERSUS 2"
VERification System Unified Survey 2
Last updated: 27 July 2015. Meetings and 8-year tasklists below
Project leader: initially Angela Celozzi, later Antonio Vocino and finally Roberto Bove (USAM)
1. Introduction
The development, implementation and operational use of the VERSUS package should satisfy most of
the needs for traditional verification but the ever increasing model resolution as well as the
current and planned ensemble systems will ask for more suitable information from verification
activities. At the same time users of model forecasts will ask for more information about the
real quality of the model: this means a step towards user-specific verification information.
The extension of the VERSUS package to VERSUS 2 will cover these items and will provide the COSMO
community with a "state of the art" verification tool. The "System Archtecture Design (SAD)"
specification document was released in May 2009 and updated in January 2010
(pdf, 0.9Mb).
2. Motivation
The aim of the VERSUS-1 Priority Project, started at the end of
2006 and concluded in February 2009 with the official delivery of the package, was the development
of a common and unified verification 'library' including a Conditional Verification (CV) tool. This
library will be included in the public COSMO Software along with the new software from Priority
Project VERSUS2. The package and all its main features have been planned and realised with the
intention to be a useful and friendly tool for the entire COSMO community for operational as well
as research activities to perform traditional and conditional verification, based on conventional
methods.
As the typical approach to CV consist of a selection of one or several forecast products and one or
several mask variables or conditions, which would be used to define thresholds for the product
verification, the flexible way to perform such a selection is to use an 'ad hoc database', planned
and designed for this purpose. In this case the masks or filters can be simple or complex SQL
statements. After the selection of the desired conditions, a classical verification tool to turn
out statistical indexes can be used, along with a graphics package to plot the scores.
The general VERSUS architecture is basically composed of a Report Data Base Management System
(RDBMS, to store observations (OBS) and forecast (FCS) data, Station Data, Configuration data to
perform verification, Verification results, Scores and plots, and User permissions; MySQL in case
of the initial VERSUS version), a "daemon" process (Loader) to load data from different sources
into the RDBMS (File system or another DB), a process performing verifications through specific
requests (Integration into the "R" statistics package), and a web-based user interface
(server-client architecture).
The VERSUS1 project was divided into 10 tasks and most of them were completed (a demo version of
VERSUS has been presented during the last WG 5 meeting in Zurich in May 2008). The delivery of the
complete package occurred during a Workshop/Tutorial in February 2009.
In the course of the VERSUS project development, most of the COSMO members moved towards higher
resolution models as well as ensemble forecasting systems (at least in their plans). The increasing
resolution has, in general, the impact that objective scores for weather parameters can be worse
for high resolution models than for low resolution models. Increased resolution, in fact, generally
produces better defined mesoscale structures, greater amplitude features, and larger gradients, and
inevitably leads to space and time errors. This is known as the "double penalty" problem.
Nevertheless such a model can provide very useful information. The need for verification techniques
that allow for some tolerance to reasonably small space and time errors is then clear, retaining
the main feature to be capable to test the real quality of information provided by the
high-resolution model and the possible connections between forecast deficiencies and specific
weaknesses in the model formulation.
At the same time, as COSMO model future plans are in the probabilistic and ensemble forecast
direction, suitable verification methods have to be applied. In general there are three methods
to evaluate this kind of forecasts (B. Casati et al. Met. Appl. 15, 2008):
- Verification of an ensemble distribution: methods used to verify ensemble distributions are
rank histograms (Talagrand histograms), Continuous rank probability score and its related
skill score (a measure for the difference between the cumulative distribution function of
the forecasts and observation), and minimum spanning tree (a sort of multidimensional rank
histogram, not so widely used);
- Verification measures for the pdf of a generic probability forecast: methods used are "linear
probability" and the "ignorance score". Both the methods are local, evaluating the pdf at the
observation value, and both rewards sharp distributions which are also accurate.
- Verification of the probability of an event: methods used are Brier score and its
decomposition, Brier skill score, reliability diagrams, ROC and the rank probability score.
It is useful to add to these methods also bootstrap confidence intervals to generalize score
estimates over different sample sizes.
All the above verification techniques, and others specialised for extreme events and user needs,
have to be analysed and then included in the VERSUS package to have a common "state-of-the-art"
tool for verification for the entire COSMO community.
3. Project tasks
In a first phase (COSMO year 2009), a "User Requirements Specification" has been compiled, which
will be implemented during the following phases of the project itself (COSMO years 2010 - on).
Any single phase should be concluded in just one COSMO year and a delay in the realisation of one
single phase will result in a delayed start of the next one. Every year, based on the real
deliverables produced, the plan will be updated accordingly along with a quick review of the
requirements of the next phase, and the new phase to be developed will be fully described.
Phase 1. COSMO year 2009
It has been completed as planned in Cosmo year 2009 , with the delivery of System Design Overview
document (updated in July 2009 as requested by the STC) and System Architecture Design (approved by
the STC and to be updated as soon as possible). These were the tasks of PHASE 1:
- Task 0
- Requirements specification for VERSUS and accepted review
- Task 1
- Implementation of VERSUS at every service within COSMO:
- Task 1a: Implementation of a multiplatform RDBMS version of VERSUS
- Task 1b: Installation of multiplatform DB version of VERSUS at every service
-
Phase 2. COSMO year 2010
- Task 0
- Help Desk, bugs fixing activities and release of VERSUS update
- Task 1
- Implementation of COSI global Index:
- Task 1a: Preparation of a technical reference document for COSI index
- Task 1b: Implementation of the index
- Task 2
- Implementation of weather type dependent verification and improvement of CV:
- Task 2a: Preparation of a Guidelines document for the development of Weather dependant
verification (WDV)
- Task 2b: Conditional verification on different space/time
- Task 2c: Development of VERSUS software module for the implementation of WDV, CV
and update of Web GUI
- Task 3
- Use of new Observation data:
Note: Task 3 will be completed with the use of Feedback Files (TASK 7) further to the
decision made during the VERSUS workshop in Langen 2010, see email from PL of 25/03/2010 on
"the new structure of the project plan"
- Task 3a: Identification and analysis of Observations to be loaded following SDO
priorities: Airep, Amdar, Windprofiler, TEMP (non standard pressure level)
- Task 3b: Implementation of Loader module (msgs coded in BUFR format) and creation of
tables in DB
- Task 3c: Integration of the new Upper air observation with the existing ones for
production of Upper air verification
- Task 3d: Implementation of verification of Wind gusts using Metar
- Task 3e: Analysis and description of Radar wind data for use in VERSUS
system(VAD, radial)
- Task 3f: Implementation of loader for radar wind data
- Task 4
- Use of gridded observation (analysis fields):
- Task 4a: Individuation of some aggregation methods for standard scores
- Task 4b: Implementation of Loader module for gridded observation (GRIB coded)
- Task 4c: Detailed documentation of Fuzzy verification toolbox for its integration
and implementation in VERSUS package
- Task 4d: Re-writing Fuzzy IDL code in a more manageable C or Fortran code (TBD)
- Task 4e: Implementation of Fuzzy toolbox and updating of WEB GUI accordingly
- Task 5
- Identification of scores for Ensemble and probabilistic forecasts:
Task 5a: Identification and full description of probabilistic scores to be implemented
with description and example plots to be implemented for these probabilistic score
- Task 6
- Production of common statistical scores and scores graphic:
- Task 6a: Preparation of data to be sent to the responsible member
- Task 6b: Production of seasonal common scores plots and preparation of
presentation for GM
- Task 7
- Preparatory work on feedback files implementation:
- Task 7a: Fully description of feedback files
- Task 7b: Implementation of loader module for feedback files in VERSUS system
(creation of tables, php modules...) to be integrated in VERSUS_FE loader
- Task 8
- Statistical features like confidence intervals and the Bootstrap method
Phase 3. COSMO year 2011
Phase 3 of the project will address the implementation of the remaining PRIORITIES A and
PRIORITIES B-C in the SDO document. The tasks of Phase 3 are:
- Task 0
- Help Desk, bug-fixing activities and release of a VERSUS update
- Task 1
- Improvement of VERSUS security, "plug&play" installation and Web pages:
- Task 1a: Collection of requirements
- Task 1b: Implementation of improved security of the system
- Task 1c: Development of a new installation and patch update procedures
- Task 1d: Development of back-up functionalities for the DB
- Task 1e: Review of Web GUI search pages for improved functionality
- Task 2
- Final Implementation of Feedback Files in VERSUS:
- Task 2a: Implementation of Loader module for FF
- Task 2b: Creation of new Web GUI for FF
- Task 2c: Conditional Verification for data from FF (Requirements by DWD)
- Task 2d: Stand-alone tool for FF creation for all partners with Documentation
- Task 2e: Dedicated test phase for FF tool implementation and FF functionalities
in VERSUS
- Task 3
- Improvement of graphical representation of scores:
- Task 3a: Survey and collection of requirements for new graphics and improvements of
existing graphics from the partners for both surface and upper air
- Task 3b: Creation of new plots
- Task 3c: Implementation in VERSUS system (interface with database)
- Task 4
- Implementation of Probabilistic Scores:
- Pre-Task 4: Delivery of the reviewed EPS
Document
- Task 4a: Comparison between the Guidelines and the "R" verification package
- Task 4b: Full description of R code and associated graphics
- Task 4c: Implementation of "R" package in VERSUS with documentation
- Task 4d: Creation of new Web pages for GUI
- Task 5
- Fuzzy Verification Toolbox and Object-oriented Verification:
- Pre-Task 5: Completion and test of standard verification with gridded analysis
- Task 5a: Test and documentation of C version of Fuzzy Toolbox
- Task 5b: Implementation in VERSUS system
- Task 5c: Feasibility study on the implementation of SAL and other methods
from MET software
- outlook until June 2012:
- Help Desk
- Consolidation of VERSUS software
- Improvements in Conditional Verification (space/time based and on model errors)
- Implementation of SAL and other object-oriented verification
- Statistical Features
- Verification of post-processing forecasts (TBD if useful for all the members)
Phase 4. COSMO year 2012
Implementation of further functionalities as from the Reference documents and consolidation
of the system
- Task 0
- Help Desk, bug-fixing activities and release of a VERSUS update
- Task 1
- Consolidation and fine tuning of the system:
This task it is connected with the stress tests and in general with TASK0 and it is aimed
to improve the easiness in using the system, its functionalities and performances.
- Task 1a: Collections of stress tests results, analysis and identification of critical
functionalities
- Task 1b: Implementation of a standard procedure for fine tuning
- Task 1c: Test of tuning efficiency
- Task 1d: Development of connection functionalities for the DB (operational, test)
- Task 1e: Review of Web GUI and internal functionalities for new facilities (see
Appendix 1 for details)
- Task 2
- Final Implementation of Feedback Files in VERSUS:
The use of Feedback Files (FF) is important to improve upper air verification and Conditional
Verification activities where large amounts of data must be manipulated. A feasibility study
on how to proceed with the implementation of FF has been created by the PL team and reviewed
by DWD in July 2010 and the new Database Structure for FF has been created by PL team.
Software by DWD to upload FF data is under investigation at the moment. The Task 2 from Phase
3 was stopped because of other priorities in COSMO year 2011. It will now be completed in
phase 4.
- Task 2a: Implementation of Loader module for FF
- Task 2b: Creation of new Web GUI for FF and improvement of graphics for upper air
- Task 2c: Conditional Verification for data from FF (Requirements by DWD)
- Task 2d: Stand-alone tool for FF creation for all partners with Documentation
- Task 3
- Implementation of new BUFR format and GRIB2:
The adoption by WMO of a new template for BUFR is the main reason for this task as well as
the foreseen use of GRIB2 as output of COSMO models.
- Task 3a: Update the system for the upload functionality of new BUFR template and merge
(in space and time) with the old format. The use of the new template will make more
flexible verification of more cumulated period for precipitation. A document describing
the activity will be delivered and approved in advance
- Task 3b: Update the system for the treatment of forecasts produce coded in GRIB2 format.
A document describing the activity will be delivered and approved in advance
- Task 3c: Implementation of features for verification of experimental versions of the
model with both GRIB1 and GRIB2
- Task 4
- Implementation of Probabilistic Scores:
Following the Guidelines from TASK5 - Phase 2/3, Probabilistic Scores will be
implemented, possibly using the “R” verification package. A capability of the
software to handle ensemble forecasts and verifying them through the deterministic approach, has already
been added in the last version of VERSUS.
- Pre-task: Completion of description of “R” package for EPS verification
- Task 4a: Implementation of “R” package in VERSUS with documentation
- Task 4b: Creation of new Web pages for GUI
- Appendix 1
- Task 1 detailed functionalities:
Functionalities listed here as from the Parallel session draft GM 2011. Note that not all the
requirements are listed below, but only those will be implemented until next April 2012; the
remaining ones will be implemented as soon as possible (see minutes of WG5 and VERSUS parallel
session GM2011). Priority has been assigned by the PL.
- Computing of continuous scores for precipitation.
- The "Batch Execution"; extended to Yearly interval.
- The Batch Execution” will have the functionality to be restricted to fewer
activities at a time.
- Configuring a newer verification filled with the configuration of an existing one
- Configuring cross-model graphics as monthly or seasonal and consequently their
batch execution.
- Addition of a data permanence process, able to delete observation and forecast older
than a configurable time presence. This process shall be enabled or disabled depending
by user.
- Cross model of time series.
- Improvement of suspect observation menu.
Phase 5. COSMO year 2013
The goal of the year is complete the most crucial developments define by participants of the last
VERSUS user meeting that took place in Rome in April 2012. The Task of the Phase 3 are:
- Task 0
- Help Desk, bug-fixing activities and release of a VERSUS update:
- Task 0.a: Help Desk activities for VERSUS users will be done through the
forum web page www.meteoam.it/forum
- Task 0.b: Documentation update. Any new development is followed by a documentation
update (User Manual and Technical Manual).
- Task 0.c: New VERSUS releases. Each new patch assumes the creation of a packet
instalment. The patches will be delivered to the users through the ftp server
(ftp.meteoam.it).
- Task 0.d: Test of each New VERSUS release is followed by the Test Procedure (defined
in the Task 2.c).
SUB-TASK COSI Test: This Statistical Index needs a deeper test because results so far seem to
be not completely reliable. During the GM we defined a test method to verify the correct
execution of this index in VERSUS.
- Task 1
- Consolidation and fine tuning of the system:
This task is connected closely with TASK0 and it is aimed to improve the easiness in using
the system, its functionalities and to increase its performance.
- Task 1.a: Software update (R, R packages, PHP, MySQL) and code update. Mainly due to EPS
verification implementation, the latest version of R package it has been installed.
- Task 1.b: Test of the task 1a.
- Task 1.c: Automation of processes/GUI adjustment. In this task some generic improvements
on GUI and VERSUS functionalities will be carried on in order to improve friendliness
and operational use of the system, among these are:
- cronjobs for running php scripts to compile verifications;
- automatic storage of figures with reasonable names for unambiguous identification,
distribute files in different folders;
- replication function of similar verification activities;
- Inclusion of Performance diagrams in VERSUS plots. An activity not connected to
the automation, but with the need to include more statistical indexes for
precipitation verification, in one graph.
- Task 1.d: Doxigen installation and code adjustment.
- Task 2
- Test Area and Test Procedure Creation:
A standard installation of VERSUS and a test data set on a public machine (internet) will be
provided and will serve as the common test environment.
- Task 2.a: VERSUS installation on a new machine at the CNMCA Centre. This system will
be available through internet (it will have a public web address).
- Task 2.b: Loading of data and preparation of the operational environment for testing
- Task 2.c: Definition and documentation of the test procedure.
- Task 3
- Implementation of Probabilistic Scores:
During this year, the PL team will finish the calculation and visualization of the ensemble
and probabilistic scores in VERSUS using as a reference the EPS documents edited by F.Gofa
(Task 5.a phase 3) and A. Boundel (Pre-task phase 4 ).
- Task 3.a: Implementation of probabilistic calculation inside VERSUS.
- Task 3.b: Creation of new Web pages for GUI.
- Task 3.c: Creation of the WEB page and code for the Cross model capability with EPS
- Task 3.d: Test phase. Reliability and consistency of the new scores and plots produced
by the system should be checked against an existing and settled EPS verification
software.
- Task 4
- Implementation of the WMO BUFR standard for observations
- Task 4.a: Updatethe system for the upload functionality of WMO-BUFR template and allow
for the support of both bufr formats.
- Task 4.b: Test phase of the Task 4.a following the Test procedure
- Task 4.c: Implementation of a module in VERSUS to separate reference version from exp.
Version.(Substituted: This is already possible in VERSUS, indirectly. Documentation on
how to upload experimental versions of COSMO models in VERSUS will be provided,
including examples of scripts).
- Task 4.d: Implementation of period 1 and 3 hours for precipitation cumulation
and wind gusts.
Phase 6. COSMO year 2013-4
The goal of the year is to complete some of the existing requirements and ensure the stability of
the system. The activities planned for this year as decided during the VUS 2013 and GM 2013 are:
- Task 0
- Help Desk, Refinements of VERSUS code
- Task 0.a: Help Desk activities for VERSUS users will be done through the forum web page
www.meteoam.it/forum.
- Task 0.b: Documentation updates User Manual and Technical Manual
- Task 0.c: New VERSUS releases.
- Task 0.d: Test of each New VERSUS release is followed by the Test Procedure which is
described thoroughly in a document that is uploaded in the users forum and checks the
system performance in all main functionalities.
- Task 0.e: Organization VERSUS Course for Developers will take place in Pratica di Mare
(May) for 2/3 days with the aim to familiarize potential developers with the software
architecture.
- Task 0.f: Code adjustment for DB Partitioning.
- SUB-TASK COSI Test: Task 1: VERSUS - Refinement of functionalities
- Task 1.a: Conditional Verification: apply the condition in both fcs and obs
data.
- Task 1.b: Definition of standard coding for precipitation parameter, TAG and SCA
collaboration and specification of WG5 requirements on the measures and
verification activities to be performed with such data.
- Task 1.c: Cross Plots for TS and DC:
- Task 2
- Implementation of Feedback Files in VERSUS
- Task 2a: Re-analysis and DB creation
- Task 2b: Implementation of Loader module for FF in VERSUS
- Task 2c: Creation of new Web GUI for FF and improvement of graphics for upper air
- Task 2d: Dedicated test phase for FF tool implementation and FF functionalities
- Task 3
- Introduction of additional statistical techniques in VERSUS.
- Task 3a: Definition of tool to be used (Field Extra, LIBSIM). Coordination with
TAG and SCA
- Task 3b: Description of spatial verification method that need to be implemented
(box method, catchment/alert area method)
- Task 3c: Feasibility study with documentation and software updates:
Analysis of Fieldextra functionalities and Analysis of LIBSIM capabilities and
software updated.
Integration with possible developments needed of the external software chosen and
link with VERSUS system.
- Task 3d: In order to integrate the external Fuzzy toolbox is necessary to perform:
code translation, documentation and description and Test of the new code
- Task 3e: In order to integrate the SAL tool is necessary to perform:
documentation and full description, Code adaptation and Test of the new code
- Task 4
- EPS Refinements.
- Task 4a: Implementation of CRPS/CRPSS and Plots of Spread/Skill relation.
- Task 4b: Pre-operational phase: every partners will have to:
- Install VERSUS last updates
- Configure the System
- Loading of EPS data (LEPS)
- Perform verifications for a period of 2-3 months
- Compare VERSUS results with current operational EPS verification
- Compile a feedback/suggestions document (coordinated possibly by WG7 Coordinator)
- Task 5
- Extension of the ASCII import capabilities of VERSUS:
Task 5.a: Feasibility study and documentation to use the ATAB file (or XML files)
Phase 7. COSMO year 2014-5
Task 0
Help Desk, Refinements VERSUS code
- Task 0.a Help Desk: This year the activity will be divided in two different support
levels: The first one being managed by Russian colleagues for the basic requested.
They will involve the PL team in case of more technical problems and bug fixing.
- Task 0.b Documentation updates: User Manual and Technical Manual.
- Task 0.c New VERSUS releases:Each new patch assumes the creation of a packet
installment. The patches will be delivered to the users through the ftp server
(ftp.meteoam.it).
- Task 0.d Organization of VERSUS User Seminar: It will take place in Turin (May/June)
for 2/3 days with the aim to introduce the new functionalities.
- Task 0.e Test for the VERSUS 3.2 extra: The package VERSUS 3.2 extra has been released
in the VERSUS2 Phase 6 but the test phase is necessary. This package allows the
partitioning of the grib table in the Data Base ( the foreign Keys (FK) have been
removed from the "grib" table of the VERSUS database, in orderto allow the table
partition).The data consistence in absence of FK is now guarantee by control code.
The test consists in analyzing the performance (execution of building activity
described in VERSUS- feedback.doc before and after installation of the package).
- Task 0.f Test for the concurrent use of VERSUS by more than one user: The system allows
to be used by more than one user simultaneously. The limitation, derived from
R software, has been solved in the phase 6. The goal of concurrent usability of the
software, based on MCH feedback, anda deeper test on this issue is needed.
- Task 0.g Performance Optimization: This task is a deep study on the tool and his
functionality on a big amount of data, atuning of the tool and clear lines definitions
to the system optimization. This study will be performed on three different areas:
- Machine better configuration;
- Optimization of loading phase speed;
- Optimization of verification performing speed.
Task 1
VERSUS - Refinement of functionalities
- Task 1.a: Confidence Interval integration in VERSUS: Update VERSUS Code, GUI Update
Plots adaptation.
- Task 1.b: EPS refinements: During the meetingbetween PL Team and WG7 held in Pratica di
Mare October 29th2014 activities to support task have been approved and
described; now
available in:
- Bug fixing: CRPSS computation, cross model, RPS and RPSS checking;
- Missing features addiction: ROC area value and Outliers, txt output of the values
of the graphs plotted in pdf format, all thresholds written into DB, provide the
possibility to verify 12h accumulated precipitation over the ranges 06-18Z and
18-06Z.
- Task 1.c: Automatic programs: Check the correct Installation; Automatic delete;
Background verifications execution; Realization of a script for stations loading from
WMO flatfile.
- Task 1.d: VERSUS Portability: feasibility study of VERSUS installation on FEDORA 20
and code adjustment.
Task 2
Final Implementation of Feedback Files in VERSUS
- Task 2.a: Development of new versus library to read the Feedback File in NETCDF;
- Task 2.b: Implementation of Loader module for FF in VERSUS;
- Task 2.c: Creation of new Web GUI for FF and improvement of graphics for upper air;
- Task 2.d: Dedicated test phase for FF tool implementation and FF functionalities
in VERSUS.
Task 3:
Implementation of VERSUS functionality to ingest (or manage) GRIB2
- Task 3.a: Analysis and creation of new DB area;
- Task 3.b: Adjustment and Upgrade of the Phoenix code;
- Task 3.c: Creation New FE in VERSUS;
- Task 3.d:Code and GUI adjustment;
- Task 3.e:Test phase following the Test Procedure document.
Task 4
Implementation of station based forecasts verification activities
In the Task6 of Phase 6 a complete feasibility study was performed together with the setup
of a standard XML file format (that can be provided by Fieldextra) for post-processed station
based forecasts. Main Activities:
- Task4.a: Implementation of Front-End and related verification activities foringestion
of XML filesof post-processed forecasts into VERSUS;
- Task 4.b: Test phase following the Test Procedure document
Task 5
Introduction of additional statistical techniques compatible with VERSUS
The advanced functionalities that will be implemented within the present task will concern
the introduction of additional statistical techniques in VERSUS for high resolution forecast
and observation data (neighbourhood methods). The COSMO Verification Community considers the
use and adaptation of external packages or libraries for pre-processing of data for
verification as one of its main priorities. For this reason the main goal of the activity
is the integration or adaption of pre-existing packages (e.g. Beth Ebert Fuzzy Verification
Toolbox). Moreover this work will be extended to the use of gridded precipitation data from
composite sources (e.g. radar, raingauges) while a pre-processing tool capable of reading
Grib files (for forecast and/or observation) and producing data in suitable format for the
VERSUS system itself will also be developed.
- The PL will coordinate in the execution of the Project VAST. The VAST project is a
project funded by COSMO; started in June 2014 and ended in May 2015 (1 FTE), more
details are described the tasks foreseen in the project (funded COSMO).
- extra1 LIBSIM SOFTWARE TEST: During the VERSUS 2 Phase 6, the extra software
has been releasedfor performing upscaling-based verification in Versus. In this phase
a test of this software is needed.
- extra2 VAST SOFTWARE TEST: VAST software Version Beta will be released in march;
users will be able to use it and check its functionality within a month in order to
allow possible little further modifications, whether required, within the end
of contract.
Phase 8. COSMO year 2015-6
4. References
PP VERSUS |
2008 |
VERSUS Reference and User Manual (delivered in 2008 and available on VERSUS and COSMO website) |
PP VERSUS2 |
2009 |
SAD and SDO - 2009 (available on VERSUS webpage and on COSMO web site) |
MET |
2008 |
Model Evaluation Tools Version 1.1 User"s Guide. NCAR, July 2008 |
Casati, B. et al. |
2008 |
Forecast verification: current status and future directions. Met App. 15: 3-18 |
Ebert, E. |
2008 |
Fuzzy verification of high-resolution gridded forecasts: a review and proposed frameworks. Met. Appl., 15, 51-64 |
Jolliffe IT, Stephenson DB. |
2003 |
Forecast Verification: A Practitioner"s Guide in Atmospheric Science. John Wiley and Sons: Chichester, UK; 240pp |
Nurmi P. |
2003 |
Recommendations on the verification of local weather forecasts. ECMWF Technical Memorandum 430: 19 |
Stephenson, D.B. et al. |
2008 |
The extreme dependency score : a non-vanishing measure for forecasts of rare events. Met App. 15: 41-50 |
Eckert P. |
2009 |
High Resolution Verification Priority Project Report, COSMO Report. To be published |
Wernli, H., P. Paulat, M. Hagen. C. Frei |
2008 |
SAL - a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev. 136, 4470-4487 |
5. Meetings
- WG5-VERSUS common meeting (minutes), 8 September
2014, Eretria, Greece
- WG5-VERSUS common meeting (minutes), 2-4 April
2012, Pratica di Mare, Italy
- VERSUS 2 meeting, 4 March
2010, Langen, Germany
- VERSUS2 meeting, 15-16 December
2009, Pratica di Mare, Italy
- VERSUS2 meeting, 12 March
2009, Langen, Germany
- VERSUS tutorial, 16-18 February 2009, Pratica di Mare, Italy
- CVS Meeting, 20 November
2006, Pratica di Mare, Italy