Last updated: 9 Aug 2018
Project duration: April 2015 – December 2017
FTEs (plan/used) 1.02/0.89 in COSMO year 2014-2015
2.46/2.18 in COSMO year 2015-2016
2.27/1.72 in COSMO year 2016-2017,
0.5 of unused FTEs were moved to the project extension in COSMO year 2017-2018:
0.5/0.48 in COSMO year 2017-2018
Total FTEs used: 5.27.
As numerical weather prediction models began to increase considerably in resolution, it became clear that traditional grid-point-by-grid-point verification methods did not provide material information about forecast performance. Numerous methods have been proposed in order to assess the value of very-high-resolution forecasts, including spatial verification methods. Furthermore, the plethora of spatial verification methods has led to the need to analyze how these methods relate to one another, how each method works, what information could be gleaned from each method, and whether any given method actually conveys any useful information or not. The ICP international project and its second phase MesoVICT (Mesoscale Verification Inter-Comparison over Complex Terrain) were initiated to study how these methods provide feedback about the forecast skill through well-structured experiments.
The main objectives of MesoVICT international project can be summarized as follows:
· To investigate the ability of existing and newly developed methods to verify fields other than deterministic precipitation forecasts
· To demonstrate the capability of spatial verification methods over complex terrain and gain an understanding of the issues that arise in such cases
· To encourage community participation in the improvement of spatial methods
· To provide the community with a testbed with common datasets but also to provide assistance in developing and testing these methods
The INSPECT project will run in parallel to MesoVICT and will try to summarize the experience of applying spatial verification methods to COSMO forecast systems of very-high-resolution (1-3 km) compared to high-resolution models, providing criteria for deciding which methods are best suited to particular applications. In addition to targeting the goals of the MesoVICT project, INSPECT is expected to provide COSMO users more choice when choosing verification domains and reference data, encouraging the participation of the COSMO community in the development and improvement of spatial verification methods.
Rapidly increasing computing power has enabled higher and higher NWP resolution. Most recent COSMO Consortium projects concern high-resolution modeling: high-resolution data assimilation (KENDA PP), ensembles at km-scale (COTEKINO), and consolidation of operation and research results for the Sochi Olympic Games (CORSO). Thus, high-resolution verification, which is also being addressed in the framework of VERSUS2, is indispensable. The VAST (Verification Additional Statistical Techniques) development has also been initiated (2014-2015) with the aim to provide the COSMO Consortium with the software for widespread spatial (neighborhood) verification methods and it is included in VERSUS2 project as a Task.
The new spatial verification methods can primarily be classified into two overall categories: filtering methods and displacement methods. The filtering methods can be further divided into neighborhood and scale separation, and the displacement methods can be subdivided into features-based and field deformation (Verifying Forecasts Spatially, 2010; Intercomparison of Spatial Forecast Verification Methods, 2009):
· Features-based Approaches
-Contiguous Rain Area (CRA) (Ebert and McBride, 2000)
-Method for Object-based Diagnostic Evaluation (MODE) tool (Davis et al., 2006).
-SAL technique (Wernli et al., 2008)
· Field Deformation Approaches
· Neighborhood-Based Approaches (summarized in Ebert, 2008)
· Scale Decomposition Approaches
The VAST software includes the use of Neighborhood methods as well as the interface for data preprocessing needed for such methods. Sharing experience with free packages for spatial methods (in particular, R SpatialVx program) will be also beneficial for COSMO members (including data preprocessing, formats, scripts, etc.). SpatialVx is a free R package (http://www.ral.ucar.edu/projects/ icp/SpatialVx/); its source code is developed and maintained by Eric Gilleland (NCAR). Most of the recently proposed methods are or will be included into SpatialVx. There is no need in developing new COSMO verification tools within this PP, but considerable efforts will be made to create basic verification tools required for realization of verification tasks of the project (development of local tools, input/output formats, scripts to run different methods, adjustment of existing packages and so on).
The ICP and MesoVICT projects have already provided for the setup of experiments and a set of test cases including high-resolution observations. The experiments are based on “real” cases, trying to demonstrate the capability of spatial verification methods over complex terrain and to gain an understanding of the issues that are connected from this more challenging situation. There is a set of six cases that cover a wide range of meteorological phenomena in and around the Alps for the period between June to November 2007 (Mesoscale Alpine Programme MAP D-PHASE). A unified data set of surface observations over Central Europe will be utilized (JDC), that is processed through the Vienna Enhanced resolution analysis (VERA) scheme that is focused on the interpolation of sparsely and irregularly distributed observations to a regular grid in mountainous terrain by utilizing the fingerprint technique (Steinacker et al., 2006). Several INSPECT tasks involve reruns of COSMO very-high-resolution models for MesoVICT test cases with a focus on the core experiment and case 1. Additional periods/models will be utilized for the completion of INSPECT’s main Tasks, e.g., the dataset of FROST-2014 project (Forecast and research in the Olympic Sochi testbed). The advantages of using this dataset are as follows: FROST models provide longer timeseries compared to MesoVICT test cases. The MesoVICT cases are older (2007 year mainly) and are mainly summer cases, while the Sochi data focuses on winter season, which is very important for the mountainous regions. It will be useful to carry out comparison for two complex terrains (the Alps and the Caucasus with their peculiar features). The comparison of COSMO versions with other models will be highly beneficial for COSMO, while it would not require many additional resources as the testing frame and formats will be unified within this task.
Until now, COSMO studies on spatial verification methods have been concentrated mainly on the deterministic precipitation field representation and the useful scales of high- or very-high-resolution models (INTERP project). One of the main aims of the INSPECT project will be to investigate the additional information gained by the application of such methods to other fields such as wind speed. Furthermore, any development of verification capabilities within COSMO must take into account the evolution of probabilistic and ensemble forecasting into space and time scales that would benefit from spatial methods applications that could designate the relative gain to coarser ones. MesoVICT experiments are structured in tiers representing a progression in terms of forecast type, parameter and choice of truth (observations), exploring a range of challenges, even though the mandatory participation to the core experiment is ensuring the successful intercomparison of the main results.
The scope of INSPECT is to summarize the experience of project participants and to consequently propose guidelines for application of new spatial methods. Some of the most widespread measures will be used by several participants, thus enabling intercomparison of results, but an attempt will also be made to use different types of spatial methods in order to avoid duplication of efforts.
Special attention will also be given to the verification strategy for analyzing extreme weather events, utilizing the intense precipitation cases that are included in MesoVICT experiments. Binary categorical scores (e.g. ETS) often suggest that precipitation forecasts are less skillful at higher thresholds (at the tail end of the distribution). On the other hand, while some categorical scores converge to zero as thresholds increase, implying poorer performance for more intense events, other scores exhibit the opposite behavior in certain cases. The aim, therefore, is to analyze whether this behavior is due to the properties of the scores and their sensitivity to the base rate, how precipitation features/pattern influence the performance of some scores, and finally if spatial verification techniques can be successfully applied is such cases.
The INSPECT PP is fully in line with the goals of the COSMO 2015-2020 Science Plan (in particular, the Task “Statistical methods to identify the skill of convection permitting and near convection-resolving model configurations”) in that the focus is to create a strategy for the selection of metrics and their the appropriate application to have the best possible “fit” between the observations and very-high-resolution forecasts. In the framework of INSPECT, it is planned to advance the application of radar and radar-gauge precipitation analysis that is also in agreement with the COSMO Science Plan (Task “Exploitation of available observational dataset for operational and scientific purposes”). INSPECT will also contribute to progress in the field of Application oriented verification information, especially in what concerns modelers and meteorologically educated users. Some work inside INSPECT will focus on Severe and High Impact Weather and tools for probabilistic and ensemble forecast verification, which are among the COSMO priorities.
Finally, summarizing the benefits of INSPECT, it will be that the wide range of spatial verification methods available will become commonly used within the COSMO community and the COSMO Guidelines will be proposed to ensure the correct interpretation of results of these methods.
Task 0: Administration of the project
Deliverables:
Literature reviews, meetings between contributing scientists, preparation of plans/reports, communication with MesoVICT project core team: participation in the meetings, e-mail exchange)
Task 1: Data and Models Setup
Goal: Preparation of a testing platform for spatial methods applications based on COSMO models and gridded observations.
Deliverables:
Infrastructure of experiments: datasets and access to them, unified formats
Task 2: Adaptation of statistical software and techniques
Goal: Development or adaptation of a set of software tools required for the verification tasks of the project.
Deliverables:
Installation packages for R software related to spatial methods, scripts for data preprocessing and for application of the methods. The deliverables will be available to task participants through a common repository.
Task 3: Application of methods on deterministic models
Goal: Calculation and representation of a series of scores from various spatial methods to be analyzed in task 5 for their relative value. While the main focus is given to precipitation, efforts will be given on application to continuous parameters such as wind speed.
Deliverables:
A range of reports on various spatial verification method applications as input for Task 5 preparation of guidelines.
Task 4: Overview of spatial methods on EPS systems
Goal: A basic analysis of spatial methods and their applicability on convection permitting ensemble systems. Being a strategic priority of COSMO the development of very high resolution ensemble systems, this Task aims to transfer the expertise from the international community (MesoVICT) on spatial method applications on ensemble systems and provide some first applications of different methods.
Deliverables:
Spatial verification methodology applications on EPS systems as input for Task 5d preparation of guidelines.
Task 5: Guidelines for relative usefulness of various spatial methods in decision-making
Goal: Using the knowledge gained from INTERP, ICP and MesoVICT projects and with the additional experience of the long timeseries from spatial verification scores accumulated at DWD, it will be attempted to identify the relative usefulness of each spatial method for precipitation and other weather parameters. Proposition of Guidelines for using spatial methods to the COSMO community, as well as suggestion of software tools for the application of these methods in COSMO.
General guidelines for using the innovative spatial methods, distinction of the usability of each method depending on the user needs (feedback from the various spatial methods application: How does each method inform about forecast performance overall? Does the method inform about location errors? If so, how? Which methods yield identical information to each other? Which methods provide complementary information?).
An updated strategy for decision-making. COSMO guidelines for spatial verification application on high-resolution models.
The main risk consists in delays due to large amount of data to process and analyze and the complexity in the application of some methods. It is not yet clear what methods would prove most informative. So this project contains vast scientific research with inherent high degree of unpredictability.
In addition, model reruns can be delayed due to restricted human and computing resources at national centers.
Ben Bouallègue Zied and Susanne E. Theis, Spatial techniques applied to precipitation ensemble forecasts: from verification results to probabilistic products. Meteorological Applications, Volume 21, Issue 4, pages 922–929, October 2014.
Davis, C., B. Brown, and R. Bullock, 2006: Object-based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev., 134, 1772–1784.
Dorninger, M., M.P. Mittermaier, E. Gilleland (more) , 2013: MesoVICT: Mesoscale Verification Inter-Comparison over Complex Terrain. NCAR Technical Note NCAR/TN-505+STR, 23 pp, DOI:10.5065/D6416V21. http://nldr.library.ucar.edu/repository/collections/TECH-NOTE-000-000-000-874
Ebert, E. and J. McBride, 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239,179–202.
Ebert, E., 2008: Fuzzy verification of high resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 51–64, doi:10.1002/met.25.
Gilleland E., David A. Ahijevych, Barbara G. Brown, and Elizabeth E. Ebert, 2010: Verifying Forecasts Spatially. Bull. Amer. Meteor. Soc., 91, 1365–1373. doi: http://dx.doi.org/10.1175/2010BAMS2819.1
Gilleland, E., David Ahijevych, Barbara G. Brown, Barbara Casati, and Elizabeth E. Ebert, 2009: Intercomparison of Spatial Forecast Verification Methods. Weather and Forecasting, Vol. 24, Iss. 5, pp. 1416–1430.
Le Duc, Kazuo Saito, And Hiromu Seko, Spatial-temporal fractions verification for high-resolution ensemble forecasts, Tellus A, 2013, 65, 18171, http://dx.doi.org/10.3402/tellusa.v65i0.18171
Marsigli, C., Montani, A. and Paccangnella, T. (2008), A spatial verification method applied to the evaluation of high-resolution ensemble forecasts. Met. Apps, 15: 125–143. doi: 10.1002/met.65
Steinacker, R., M. Ratheiser, B. Bica, B. Chimani, M. Dorninger, W. Gepp, C. Lotteraner, S. Schneider, S. Tschannett, 2006: A mesoscale data analysis and downscaling method over complex terrain. Monthly Weather Review 134, 2758 - 2771.
Technical Report No. 16 COSMO Priority Project "INTERP":Final Report by Pierre Eckert, December 2009
Wernli , H., M. Paulat, M. Hagen and C. Frei, 2008: SAL - a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev., 136, 4470-4487
Weusthoff, T., F. Ament, M. Arpagaus, M.W. Rotach 2010: Assessing the benefits of convective-permitting models by neighborhood verification: examples form MAP D-PHASE. Mon. Wea. Rev. 138, 3418-3433.