Agenda and Summary for WG3 and WG5 workshops

Wednesday 9 March 2005, Langen

Contents

• Part 1: Agenda
• Part 2: Summary
• Part 3: Wishes and decisions
• Part 4: Notes on "conditional" verification

1. Agenda

2. Summary

The workshop discussion was mainly based on the definition of verification procedures for the needs of WG3 members (generally called "Scientific verification") in order to localize specific problems of the operational model or to verify new LM version. The discussion starts within the new WP 5.6.2.

In relation to this argument, during the meeting, specific topics were discussed: in particular has been clarify that the better verification techniques to use for the needs of the WG3 are the so called "Conditional verifications". This kind of verifications allows a simplification of the relations between different meteorological fields and a reduction of the problem to the analysis of some selected parameters; in this way it's more "easy" to find and solve problems or to optimize the variables parameterization.

At the present time are not available, at different Centres, conditional verification tools able to satisfy the WG3 request. So, it will be necessary to develop this kind of instruments and the best way to do it will be probably with the implementation of the Common Verification Suite.

This work is behind the initial tasks of the CVS, so to do it is necessary the Steering Committee clearance and to find new resources (as discussed during the workshop in the afternoon WG5 session); after that will be created the new Work Package "implementation of the Common Verification Suite for conditional verification".

It's important to remember that the realization of the conditional verification package needs specific request, since the general criteria of the verification and the parameters that must be analyzed should be still defined (standard parameters, special parameters..). The list of criteria for conditional verification will be defined by WG3.

The heart of the conditional verifications probably will be to provide a tool ("data finding tool") that allows to find days / grid-points which satisfy specific criteria, in fact localized the couple "fcs, obs", which satisfy the requested criteria, it will be quite easy to apply the already available verification routine.

The development of the "data finding tool" routine will allow also to extend and/or adapt the already available list of test cases.

At the moment WG5 can produce only partial conditional verification, as showed in the joint workshop with the preliminary results of Schubiger et al.; for WG3 is not enough but we're working to do more and more.

There's a last point to clarify concerning the run of the new LM versions on selected WG3 test cases; this argument will be discussed at the next WG Coord. Meeting.

During WG5 workshop session, held in the afternoon, other two topics was discussed:

• the activity of Work Package 5.1.7 and 5.2.6. Nevertheless these two packages has an high priority, till know no results were obtained. The solution suggested during the workshop is to have one person responsible to make a summary of LM problems once collected the verification results produced at the different Centres. The person proposed for this year is Patrizio Emiliani. Each Centre must produce seasonal LM verification against surface and upper air parameters (as scheduled with WPs 5.1.1-2-3-4-5 and WPs 5.2.1-2-3-4-5), with a brief highlights summary, and send it to the COSMO web and also to Patrizio Emiliani, whom will match the different results and then will produce a paper of LM's common problems;
• have homogeneous verification among different Centres (same time periods, same scores, same parameters ...) in order to make right matches; the solution suggested is to produce a document to be publish on the COSMO web including general guidelines for "standard verification". The Coordinator will send to WG5 people a draft of this document, before summer 2005; this version will be open to remarks and suggestions from all WG5 people before the publication.

06/04/2005, Patrizio Emiliani

3. Wishes and decisions

The main discussion at the joint COSMO WG3-WG5 workshop was devoted to the way forward concerning conditional verification of test cases, test periods, and operational model suites (for the full agenda of the meeting, check the COSMO web-site).

The following decisions have been taken (point 1) and wishes have been formulated (points 2 to 5):

1. WG3 provides a list of criteria to use for conditional verification

   Deadline for delivery of list: end of April 2005.

2. WG5 is invited to provide a tool ("data finding tool") that allows to easily and efficiently find days/grid-points which satisfy specific criteria (e.g., the ones specified in the list mentioned under 1; since most current verification packages use in-situ observations, only grid-points for which an observation exists need to be considered at this stage). The selected days/grid-points can then be used for conditional verification and/or for the selection of test cases.

To save manpower (development of the tool!), there should be one universal data finding tool for all the COSMO members, rather than different tools at every service or even different tools connected to each verification package.

Data finding tool should be delivered by: asap, priority 1.

3. WG5 is kindly asked to supply a conditional verification of the operational model suites on a regular basis (e.g., every season, as is already done for "normal" verification of the operational model suites).

   Whether every verification should be extended to allow for conditional verifications or whether one should try to implement theses features into the common verification package is left to WG5 (or the COSMO Steering Committee) to decide. However, considering the necessary manpower to extend all the verification packages, an extended common verification package is certainly preferable.

   Start conditional verification of operational model suites: asap, priority 1.

4. Testing of the reference version should also include conditional verifications. Therefore, the common verification package should be extended accordingly, and WG5 is kindly requested to perform conditional verifications of the reference version once the process of verifying the reference version is established.

   Concerning the selection of test cases and/or test periods for testing the reference version, the data finding tool mentioned under point 2 can be used to extend and/or adapt the already available list of test cases (maintained by Chiara Marsigli)

   Start conditional verification of reference version: asap, priority 2.

5. Finally, WG3 would very much appreciate if a common and easy-to-use verification tool that allows non-WG5 members to do conditional verifications themselves could be provided, maintained (and used) by WG5. The obvious candidate for this tool is the common verification package.

   Common and easy-to-use conditional verification package should be delivered: after completion of points 2 and 3, priority 3.

In view of the points above, the common verification package gets more and more important and enough manpower should be devoted to its development to ensure it will be able to live up to expectations.

For the minutes: Marco Arpagaus, 24.3.2005.

4. Conditional verification of NWP forecast products

1. General remarks

The classical verification of forecast products is generally based on the evaluation of single elements (e.g. T_2m, RR) over specific domains in space and time. The resulting numbers, tables and plots present measures of the overall performance of the model with regard to the product considered. Potential interdependencies between various products are a priori ignored (e.g. cloud cover & near surface temperature). The interpretation of classical verification results with regard to systematic deficiencies in the model simulation of specific processes is also far from trivial. Even though obvious failures of the model may be established, finding the reasons behind these failures is hampered by the integral properties of the classical approach, i.e. the averaging over spatial domains (and time) without consideration of specific details of the atmospheric and/or surface situation. However, if tools existed, which would allow a process or situation dependent verification, e.g. the evaluation of the model forecast quality with regard to the diurnal cycle of near surface temperatures in the absence of clouds, the task of finding the cause of model shortcomings might be made much easier and model improvement would be facilitated. Formulating the verification of forecast products in conjunction with the existence of additional criteria, which are to be met, will be considered as, "conditional verification" (CV).

2. A possible approach to conditional verification of forecast products

The prime purpose of CV is the systematic evaluation of model performance in order to reveal typical shortcomings and the reasons behind them. Applied in a routine mode, it should also have the potential to provide information to the forecasters with regard to the situation and product dependent model reliability (e.g. typical clear sky forecast errors for T_2M in contrast to cloudy sky condition errors). The typical approach to CV could consist of the selection of one or several forecast products and one or several mask variables, which would be used to define thresholds for the product, verification (e.g. verification of T_2M for grid points with zero cloud cover in model and observations only). However, the masking requirements may occasionally be rather complex as in cases, where the forecast product depends strongly on the, history of the evolution rather than the current state of a certain variable. For this reason, one could define at least four classes of, masks:

1. time-independent masks
2. time dependent masks concurrent to the product
3. time dependent masks with possible time-lag to the verified product
4. masks depending on an atmospheric, "situation classification"

The first class is certainly the most simple one to handle, but carries nevertheless a certain potential with regard to the purpose of the exercise. Apart from obvious masks like the application of a land-sea filter or the stratification of verification results with regard to orographic height, one could envisage masks like the model soil type, plant cover, roughness length etc. The application of such masks might well reveal that certain deficiencies are more pronounced within certain ranges of a particular masks than outside of that range and thereby lead to the detection of a more specific model deficiency.

The second class concerns more the direct interaction between various model products. Typical examples would be the verification of T_2M in the presence or absence of snow, but also the verification of products like surface radiative fluxes stratified by cloud cover.

The third class allows in addition to take into account the "history" of the model (or observation) evolution. E.g. the near surface temperature as an instantaneous model product depends among others on the cloud cover which was present in the period before the verification time. In order to obtain a meaningful CV for such a product/mask combination, one would evaluate the verified variable with respect to the previous evolution and the current state of the mask variable. With this type of masks one might also try to exploit the potential of mask variables like the time-integrated or time-averaged Bowen ratio or other indicators of process relevant model quantities. For the example mentioned, the stratification of results might provide indications with regard to unrealistic soil moisture conditions or shortcomings in the soil and surface scheme.

Finally, the fourth class may need a sophisticated algorithm or even manual intervention in the stratification of all cases with regard to the mask criteria. E.g. if we are interested in the performance of the model in "stable" wintertime conditions, it may not be sufficient to check for a temperature inversion near the earth’s surface or relatively high pressure over the domain considered..

Masking conditions may be formulated in model or observation space or both, depending on the application. E.g. in order to obtain a detailed insight in the models ability to simulate the diurnal cycle of near surface temperatures in cloud-free conditions, neither simulated nor observed cloud cover must exceed the specified threshold value.

Generally several masks may be applied at once, e.g. simultaneous application of cloud cover, soil moisture and plant cover thresholds. Obviously the size of the ensemble will depend on the number of criteria applied. Even though this could be detrimental for the evaluation of a few case studies, in an operational environment this problem would be alleviated by the large number of forecasts generated.

One drawback of CV may be the multitude of verification products arising, if various mask criteria applied to a set of forecasts. In order to avoid excessive demands with regard to the evaluation and interpretation of the results, methods should be developed which enable the automatic distinction between "insignificant" verification results and those which, if closely inspected, bear the potential for the further insight into the model behaviour. E.g. the correlation between a verification measure (e.g. mean error) and corresponding threshold values for the mask criteria could serve for an automatic detection.

3. Scope of the CV-system for future enhancements/improvements

One should attempt to develop tools for CV in a rather generic fashion, so that modifications or extensions with regard to the criteria applied can be handled within an existing framework rather than requiring extensive updates or redesign and -development of the software. As a simple example, one could imagine that a tool which allows the CV of T_2M with an upper threshold in soil moisture, should require only a modification of a configuration file to be suitable for CV of the near surface wind within a given range of the surface roughness length.

4. Proposal for a first set of conditional verification set up

A generalised CV tool, which can handle all the potential applications mentioned above requires development resources which exceed those available for the time being. Consequently, in a first step one should try to create a tool which is able to handle at least the more basic applications mentioned above. However, this should be done in a flexible, if possible user configurable way and allow future extensions towards a fully flexible tool that may be applied both to operational and experimental forecast products. Depending on the situation restrictions in data archives will have to be taken into account, when certain CV products are considered for realisation. In the following a first attempt is made to define a small set of products to be verified in conjunction with corresponding criteria. This set should only be considered as a starting point, as it is to be expected that, additional and/or modified configurations will be considered in the future. No attempt is made to order the following table with regard to the expected benefit.