Skamarock, W.C., 2004.
Evaluating Mesoscale NWP Models Using Kinetic Energy Spectra. Mon. Wea. Rev., 132, 3019-3032.

Stephens, Graeme L., Si-Chee Tsay, Paul W. Stackhouse, Piotr J. Flatau, 1990.
The Relevance of the Microphysical and Radiative Properties of Cirrus Clouds to Climate and Climatic Feedback. J. Atmos. Sci., 47, 1742–1754.

Voudouri A., P. Khain, I. Carmona, E. Avgoustoglou, J.M. Bettems, F. Grazzini, O. Bellprat, P. Kaufmann and E. Bucchignani, 2017:
Calibration of COSMO Model, Priority Project CALMO, Final report. COSMO Technical Report, 32.
techReport32.

Voudouri A., E. Avgoustoglou and P. Kaufmann, 2017:
Impacts of Observational Data Assimilation on Operational Forecasts. Perspectives on Atmospheric Sciences, Vol.1, Springer, pp 143-150

Voudouri A., Khain P., Carmona I., Bellprat O., Grazzini F., Avgoustoglou E., Bettems J.M and Kaufmann P., 2017b:
Objective calibration of numerical weather prediction models. Atm. Res. Vol. 190, 128-140
https://doi.org/10.1016/j.atmosres.2017.02.007

Voudouri A., Khain P., Carmona I., Avgoustoglou E., Bettems J.M., Kaufmann P and Grazzini F.:
Optimization of high resolution COSMO model performance over Switzerland and Northern Italy. Atm. Res. Vol. 213, 70-85
https://doi.org/10.1016/j.atmosres.2018.05.026

Webb M., Hugo Lambert F. and Gregory J., 2013.
Origins of differences in climate sensitivity, forcing and feedback in climate models. Climate Dynamics, 40, 677

Project Plan

Introduction

The proposed project is a follow-up to the CALMO priority project; it aims at consolidating and extending the findings of this previous project, and at providing a permanent COSMO framework for objective model calibration.

The main benefits for the COSMO community of a successful CALMO-MAX project will be: (1) each COSMO member can define an optimal calibration of its own production system, including a focus on extreme events; (2) a re-calibration of the production system after a major model change is feasible; (3) an optimal perturbation of the model parameters for an EPS system is provided.

Side benefits include a better understanding of the role of the unconfined model parameters on the quality of the model, and, maybe, the introduction of a season dependency of the model parameters.

The main risk of this project is that the method remains computationally too expensive for regular usage.

Motivation

Model parameter uncertainty is a major source of errors in regional climate and NWP model simulations (Stephens et al., 1990; Knutti et al., 2002; Webb et al., 2013). State-of-the-art NWP models are commonly tuned using expert knowledge without following a well-defined strategy (Duan et al., 2006; Skamarock, 2004; Bayler et al., 2000). This is also the case for the COSMO model where ‘expert tuning’ is typically made once during the development of the model, for a certain target area, and for a certain model configuration, and is difficult if not impossible to replicate. It is questionable whether such a calibration is still optimal for different target regions (e.g. with a different climate) or for other model configurations (e.g. with an increased grid resolution). Furthermore, the lack of an objective process to re-calibrate the model is often a major roadblock for the implementation of new model features.

A practicable objective multi-variate calibration method has been proposed by Neelin et al. (2010) and applied to the COSMO model for regional climate simulations (RCM) by Bellprat et al. (2012a and 2012b). The objective method has shown to be at least as good as an expert tuning. Based on these results, a COSMO priority project (CALMO) has been proposed and accepted, at the COSMO GM 2012, with the aim to investigate how to transfer this method to NWP applications.

The CALMO project officially finished in December 2016[1], and the final report will be available in spring 2017. Some results have already been described in the COSMO Technical Report 25 and a second Technical Report will soon be published. A paper by Voudouri et al. (2017) has been published at Atmospheric Research and a second one is under preparation.

The calibration method used by the CALMO project optimizes a global model performance score[2] by adjusting the values of a set of unconfined model parameters[3]. A central element of the calibration process is the so called meta-model, which represents with a simple mathematical function the dependency of some representative model fields on the selected model parameters. The mathematical function at the core of the meta-model is calibrated by a set of full model simulations over a time period long enough to represent the variability of the atmospheric conditions. Once fully specified, the meta-model supports a fast sampling of the parameter space to find an optimal combination of the model parameters.

In a first phase of the CALMO project the method has been tested for COSMO-7 for three parameters over two 20 days’ periods; in a second phase, COSMO-2 and six parameters have been calibrated over an entire year, and in a final phase COSMO-1 and five parameters have been calibrated over a one month period. The soil history was only considered for the COSMO-1 configuration.

The CALMO project has shown that the method used by Bellprat for a RCM can be adapted for NWP applications. After a proper re-design, the meta-model is indeed able to reasonably reproduce COSMO model simulations, for all cases considered (Khain et al., 2015, 2017). Furthermore, the optimum set of model parameters improves a COSI-type score[4], by more than 10% in the case of COSMO-1, and the results of an independent verification seems to indicate that the operational verification scores are also improved (work in progress).

However, the spin-up time to acquire the knowledge to run the required multi-years COSMO simulations on the Piz Daint machine at CSCS, in GPU mode, and the time spent to solve multiple technical problems, have been grossly under-estimated in the original planning of the CALMO project. As a consequence, some of the initial goals of the CALMO project have not been reached:

• The COSMO-1 calibration is limited to the month of January 2013, instead of a full year as originally planned. This is a major limitation, reducing the robustness of our current analysis, because the history of the soil, which may substantially impact the effect of the calibration, was only switched on for the COSMO-1 configuration.
• No time remained to tackle the third important objective of the CALMO project, namely to optimize the calibration procedure with respect to the required amount of computing resources. The computational cost of the method remains prohibitive for a regular usage of the calibration procedure, and severely reduces its usefulness.

Instead of asking to further extend the CALMO project, the project team and the working group coordinator decided to propose a new project. On one side, what has been achieved with CALMO is rich enough to close a chapter, on the other side the remaining tasks and some new questions raised during the project form a logically consistent package.

The new follow-up project is called CALMO-MAX and its main goals are:

• Optimize the method to find a compromise between the quality of the calibration and the computer costs. Use a one year COSMO-1 calibration, with the soil memory switched on, as test bed for this purpose.
• Establish CALMO as a permanent optimization tool within COSMO. Create a demonstration framework at ECMWF. Create and maintain a database of unconfined model parameters. This goal may be linked with WG3a activities, to consolidate the tuning of the model during the main development phase at DWD.
• Apply the method to different climatology (e.g. Mediterranean). Besides the strong interest of some COSMO partners for a targeted model optimization, this will be used to demonstrate the applicability of the permanent framework.
• Investigate and implement specific calibrations, like a calibration for extreme events or a season dependent calibration. Further consolidate and extend the meta-model and the global model score for this purpose. This goal may be linked with WG5 activities.

The new project is build-up on the knowledge now available at HNMS and at IMS. In the long term, some permanent support for CALMO based calibration for the COSMO community could be envisaged. On the opposite, if CALMO-MAX is not accepted, there is a considerable risk that the knowledge accumulated during the CALMO project will be lost.

It should be noted that, this objective calibration methodology has the potential to bring a transformative change to atmospheric model development. More specifically, once computational cost is reduced, the developed methodology could be used by each COSMO member to define an optimal calibration over the target area of interest, for re-calibration after major model changes (e.g. higher horizontal and / or vertical resolution), for an unbiased assessment of different modules (e.g. parameterization schemes), as well as for optimal perturbation of parameters when run in ensemble mode. Furthermore, a better understanding of the sensitivity of the model quality associated with a specific parameter value, as provided by the meta-model, could benefit the quantification of the flow dependent model forecast and clarify the impact of a specific parameter on the overall model performance.

Actions proposed

• Consolidation of CALMO outcome (Task 1)
• Optimization of the CALMO methodology (Task 2)
• Establishment of a permanent CALMO platform (Task 3)
• Application of the methodology, for extremes events and for multiple climatology (Task 4)
• Documentation and dissemination of work (Task 5)

Tasks, deliverables and participants

• Provide a computationally reasonably cheap objective calibration methodology that can complement expert tuning for the calibrating of the COSMO model.
• Provide a demonstration framework at ECMWF to apply this methodology.
• Provide the associated technical and scientific documentation.

Risks

The main risk of the project is that the CALMO methodology remains prohibitively expensive in terms of required computing capacity. This would make a frequent usage of the method impossible and strongly decrease the practical interest of the method.

Another risk, which is more an inherent characteristic of any calibration methodology, is that calibrating the model for extreme events may degrade the mean operational scores.

Description of individual tasks

Tasks status sumarized in cursive red.
 

Task 0: Administration and support

This administrative task is significant due to the distributed nature of the project team. The necessary effort to keep a good information flow between all participants (e.g. by organizing regular phone or web conferences and workshops) is included in this task. The existing mailing list of the CALMO project (see http://mail.cosmo-model.org/mailman/listinfo/calmo) will be used in order to support communication and information exchange within project participants.

Task 1: Consolidation of CALMO outcome

The goal of this task is to establish the framework for the tasks 2 to 4.

1.1: Review of CALMO methodology

This sub-task aims at consolidating the knowledge gained through the application of CALMO. Review of the methodology will be performed, and urgent adjustments will be implemented, before starting the task 2. An exhaustive list of scientific questions and issues raised by the CALMO project will be preparer for later consideration in task 4.

Action points

* [HNMS] Additional sensitivity studies for uc1, radfac, kexpdec

Scientific questions

* Evaluate possibility to consider a geographical dependency of calibrated parameters in relation with soil or surface properties
(could e.g. be used for a new parameterization of the vegetation canopy which introduces a land use dependent tuning parameter).
* Clear dependency of parameters optimum on weater type (e.g. see monthly dependency in CALMO phase 2, in particular 02.2013);
can we take this into account in the optimization process?
(this question is related to the hyper-parameterization approach considered in WG3a by Matthias Raschendorfer)
 

1.2: Preparation of the technical infrastructure

The most important issue is the acquisition of the necessary computing resources throughout the project, in particular for the tasks 3 and 4. The adaptation of the tools for the target computing platform is also included in this task. Note that a lot of experience has already been gained during the CALMO project, and no major problems are foreseen.

The suggested computing platform is the ECMWF HPC system. More specifically available billing units from HNMS will be used and if needed a request for a special project will be submitted by HNMS to ECMWF. In addition a proposal for a new allocation period on Piz Daint / CSCS will also be prepared for the computing resources required by task 2 (with the benefit of having all infrastructures for calibrating COSMO-1 already in place).

Ressources at CSCS

Proposal for resources on Piz Daint / CSCS has been accepted, with excellent scientific review, but with reduced allocation time budget and disk space.

Task 2: Optimization of the CALMO methodology

The goal here is to find a compromise between the forecast quality improvement brought by the calibration method, and the computational cost of the method. 

2.1: Calibration of COSMO-1 for a full year

The aim of this subtask is to complete the COSMO-1 calibration, using a full year of statistics, with the history of the soil, as originally planned for the CALMO project. The results of sub-task 1.1 will be considered.

This calibration will be used as test bed to evaluate different options to reduce the cost of the method. The number of calibrated parameters will depend on the HPC platform available for this task, but at least 3 parameters will be calibrated.

Configuration

* Same model code and configuration as current COSMO-1 (too many risks associated with Linda developments,
both because of the update of TSA, and of the testing status of Linda developments in NWP mode),
but incorporate Linda bug fix in TERRA code
* Use single precision binary (both at CSCS on GPU and at ECMWF on CPU, about 30% less expensive)
* Use latest release of LM package (has been prepared by André)
* Calibrated parameters are those defined in the following table.

Meta-model adaptations
 * Control spatial 'spotiness' and timing of precipitation fields: (1) add neighbourhood method in MM (FSS score),
(2) use finer temporal resolution (6h instead of 24h accum)
 * Introduce near surface humidity (to avoid T2m over tuning, INCA based gridded product available over Switzerland)
* Introduce sunshine duration (gridded products available over Switzerland)
* Compute a perturbed initial condition run to estimate the internal variability of the model,
do not consider calibration points when the meta-model sensitivity is smaller than the internal variability
(noise mitigation measure)

Calibrated parameters

* Minimal diffusion coefficient for heat tkhmin Prio 1 [0. 1 ,0.4, 1] 
* Factor for laminar resistance for heat rlam_heat Prio 1 [0.1,1,2]
* Parameter controlling the vertical variation
of critical relative humidity for sub-grid
cloud formation  uc1 Prio 1 [0,0.3,1.0]   
* Factor for vertical velocity of snow v0snow Prio 1 [10,20,30]
* Fraction of cloud water and ice
considered by the radiation scheme radfac Prio 1 [0.3,0.6,0.9]
* Factor for hydraulic conductivity  kexpdec Prio1 [0,2,2]
* Uniform factor for root depth field fac_rootdp2 Prio 2 [0.5,1,1.5]

Note: The first 5 parameters are also used for the calibration experiments performed by ETHZ group.
Note: Only the six 'Prio 1' parameters will be calibrated.
Note: An iterative calibration is performed, selecting the interation terms to consider with a first simplified calibration, using min/max of each parameter only.

Verification

* Independent year (2014) will be choosen to document the impact of the calibrated parameters

2.2: Find a way to optimize the computational cost of the method

This sub-task aims at collecting ideas, and at evaluating different options to reduce the computational cost of the method, without significantly degrading the quality of the calibration. The COSMO-1 calibration performed in sub-task 2.1 will be used as test bed.

In particular, the question of the minimal number of simulations to fit the meta-model, and how this affects the accuracy of the meta-model will be considered. The best strategy to fit the meta-model will be reviewed, using in particular the ideas developed by E. Avgoustoglou during the CALMO project. The minimal geographical domain for the calibration will also be considered.

This is the action of the project whose result is the most uncertain, in the sense that it is not guaranteed that a computationally cheap enough approach will be found.

Current suggestions

* Compute hindcasts instead of assimilation cycle and regular forecast (already done)
* Use a limited domain (e.g. half of COSMO-1 in both directions) or / and
use a coarser resolution (e.g. 2.2 instead of 1.1 km)
>>> check impact on model quality over target area (CH) for these 2 different configurations;
>>> in case impact is small, the corresponding configuration can be used for the calibration
* Calibrate using a limited time period (e.g. 30 days representative for all weather patters),
but only if the soil memory is of no concern for the targeted configuration (otherwise a full year should still be considered) 
* Use only one interaction term to fit the meta-model (the one closest to the first guess obtained by min/max fitting) or / and
consider an iterative method to fit the MM (in line which Euripides’proposal)
>>> ideally up to a specified MM accuracy, but MM accuracy estimation is expensive...
>>> this reduces the required number of simulations to 2*N + N*(N-1) / 2 + 1
* Partition set of unconfined model parameters in different subsets (if weak dependency between the subsets can be assumed),
calibrate first subset, then next subset

Task 3: Establishment of a permanent CALMO platform

One important objective of this project is to provide a permanent infrastructure supporting the application of the calibration method, accessible to all COSMO members. Besides being used to run the calibration, this infrastructure could also serve as template for replication of the methodology on the user home HPC platform.

3.1: HPC framework

It is the aim of this sub-task to prepare a demonstrative technical framework. The HPC platform which is the most widely accessible for the COSMO community is the HPC at ECMWF (already used by COSMO for COSMO-LEPS and for the NWP test suite). Thus the installation of the demonstration framework on the ECMWF HPC platform to run the COSMO model, including Terra standalone and the required pre- and post-processing operations (fieldextra) in order to apply the CALMO methodology is included in this task. This platform should be opened to any registered user. Possibly many elements are already in place (e.g. to run the production suite at HNMS, or to run the COSMO-LEPS system) and could be re-used. A full installation on some HPC could help propagate the use of this method. In case of an ECWMF non-member state the software and the documentation will be available through the COSMO web site and support on applying the methodology will be provided.

3.2: Data thinning policy and application

The amount of raw data produced by the calibration method is potentially huge and an efficient data thinning policy is required to make the method applicable. The policy developed during the CALMO project, implemented with fieldextra, will be refined.

3.3: Meta-model

The guidelines for the installation of the meta-model as part demonstration framework, as well as all appropriate modifications needed to make the meta-model user friendly are included in this sub-task. A copy of the updated version of meta-model will be uploaded to COSMO web page and be available for all COSMO members.

* Code available on GitHub and, as self-contained package, on the software page of the COSMO web.1
* An Octave version is being prepared, and should be available end of October 2018 (IMS).

3.4: Database of unconfined model parameters

An exhaustive list of unconfined model parameters and their associated characteristics (default values, unconfined range, model sensitivity) has been prepared during CALMO. This should be updated and maintained, including the definition of the steps to be taken to make this action permanent.

* Very important task for the SMC
* Table on COSMO web, ideally filled up during development phase
> Including also as much as possible parameters implemented as hard coded value
(Matthias R. notes that all parameters should be available through namelist, but most would be put in a namelist targeted at experts only).
> With short description, default, minimum and maximum values
> Including information about model sensitivity (summer/winter, different target areas)
* Preliminary steps done in CALMO / CALMO-MAX
* Work also required by PP APSU, coordination workshop planned
* Should become a permanent task of COSMO
Workshop on this topic beginning of January 2019 at HNMS (Chiara M, Uli B, C-MAX team... Aim at defining and coordinating the work)

3.5: Access to observations

Full set of observations for Switzerland and Northern Italy, for year 2013, has been collected during the CALMO project. This task aims at defining a way to facilitate the access to the observations required for the calibration process (possible options run from a simple documentation, to a complete database keeping the observation local at ECMWF, in a suitable format).

Task 4: Adaptation of the methodology on Extremes

This task aims at applying the optimized calibration strategy developed in task 2 to tackle different open questions, using the platform prepared in task 3. In this process, different improvements of the meta-model will also be considered.

A set of goals based on specific questions will be prepared in task 1.1 and the cost for a single calibration is associated with the results of task 2. It should be noted that calibration cost is not a priori known while human resources for the whole project are limited. Thus it is not possible to guarantee that all questions of interest will be tackled in the frame of this project before having the results of tasks 1.1 and 2. The exact definition of the associated sub-tasks will be made at that time point, once the set of questions is finalized.

A non-exhaustive list of open issues to consider, which will be refined in sub-task 1.1, sorted by decreasing priority, is:

• Assess the forecast quality gain obtained when calibrating the model for a completely different climatology, using the Mediterranean climatology as test bed.
• Evaluate the possibility to adapt the meta-model and the global model performance score to define a calibration privileging extreme events.
• Evaluate the consequences of season or weather type dependent calibration.
• Evaluate the possibility to introduce a geographical dependency for soil and surface related unconfined parameters.
• Evaluate the usefulness of the proposed methodology for the model tuning performed during the model development at DWD.

4.1: Support for extreme events

This sub-task will be focused on preparing the necessary elements required for a calibration privileging extreme events, namely: (1) determining an appropriate set of unconfined model parameters, (2) selecting the model fields and the global model performance score, (3) collecting the associated observations, (4) selecting a set of extreme events.

4.2: Experiments using the meta-model (MM)

This sub-task deals with several open issues related to the MM, such as the use of CAPE, the definition of new regions etc. In addition, different global model performance scores will be evaluated, and the reliability of the meta-model will be evaluated. Further adjustments of the meta-model will be performed as necessary to consider extreme events.

4.3: Experimental set-up.

The optimized method developed in task 2 will be used. The computation framework developed under task 3 will be used. The latest version of the official COSMO code will be used.

4.4: Compute experiments and analyse results

Apply the calibration methodology to a Mediterranean domain; evaluate the gain in forecast quality using the operational verification.

Task 5: Documentation
The need to make public the work performed within COSMO PPs, not only to the COSMO member but also to the wider scientific community, is nowadays well supported within the Consortium. Thus, this task aims at the preparation and the submission of manuscripts in peer reviewed scientific journals as well as contribution in conference proceedings.

An updated ‘cookbook’ to facilitate the usage of this method by other COSMO members, based on the previous ‘cookbook’ provided at the end of the CALMO project, will be prepared.