Priority Project "CALMO"
Calibration of COSMO Model

Last updated: 6 September 2017

The CALMO priority project is associated with the Working Group 3b.

The project leader was A. Voudouri from HNMS. The original project plan is available here.

The overarching goal of the project was to develop a method supporting an objective calibration of the unconfined parameters of a NWP model, based on a previous work by O.Bellprat (2012).

Project resources

Team

A. Voudouri, E. Avgoustoglou (HNMS), P. Khain, I. Carmona (IMS),
F. Grazini (ARPA-E), O. Bellprat (ETHZ), JM. Bettems (MCH)

Project duration

January 2013-December 2016

FTEs (plan/used)

Total FTEs

6.48/6.32

Summary

Numerical weather prediction (NWP) and climate models use parameterization schemes for physical processes which often include free or poorly confined parameters. Model developers normally calibrate the values of these parameters subjectively to improve the agreement of forecasts with available observations, a procedure referred as expert tuning. A practicable objective multi-variate calibration method that has been developed by Bellprat et al. (2012) and implemented for a regional climate model; this method has been shown to be at least as good as expert tuning.

To evaluate the possibility to adapt the calibration method for NWP applications, a COSMO priority project has been defined and conducted (CALMO). After a preliminary study on a coarse resolution grid, calibrating only three parameters, a more realistic configuration could be introduced thanks to the Piz Daint production project "Objective calibration of weather prediction models". The model resolution was increased from 7km to 2.2km, in addition to the enlargement of the domain size of the simulations, and Northern Italy was also included in the verification area (the same tools are used independently from the verification region, but different sets of observations have to be collected). Besides the daily accumulated precipitation and the daily 2m temperature extremum, the sounding profiles were added to the set of constraining observations. The simulation period was significantly increased from 40 days of 2008 to the entire year of 2013 (a one year simulation period consistently incorporates the weather development on a seasonal basis and should be considered a strong asset toward the operational value of this work). The number of calibrated parameters was increased from 3 to 6. Significant work was also invested in the quadratic meta-model: adding the support of atmospheric profiles, introducing an option not to average temperature extremum over regions, defining a new set of regions, adjusting the RMSE-type performance score and introducing a new COSI performance score (Damrath 2009), introducing a new method for logarithmic transformation of selected parameters, and developing an iterative method to obtain the optimal parameters via convergence in a n-dimensional parameter space of exceptional cardinality.

Following these adaptations, the calibration was performed and the optimal parameters combination was obtained. Using the COSI performance score to quantify the quality of the simulation, which is a combination of root mean square score for continuous fields and equitable thread score for precipitation, a performance gain of 2-4 % is observed. An independent verification of the optimal configuration shows a small reduction of the 2m temperature and precipitation biases, but also a small increase of the 2m dew point bias. This small impact is expected, given that the chosen model configuration is very similar to the model configuration used by the COSMO core development team, which has undergone exceptional expert tuning over a period of almost two decades; arguably this small impact confirms the validity of the calibration method. However, the main learning from the CALMO project is that the meta-model is able to reasonably reproduce the dependency of the model on the unconfined parameters. This is illustrated in Figure 1, where the meta-model prediction is compared with the full model prediction for the minimum daily temperature (left panel) and the daily accumulated precipitation (right panel) at different locations. To produce these correlations plots, an arbitrary combination of model parameters, not used for building the meta-model, was used.

Figure 1

Tmin meta-model prediction for a specific parameter combination, vs COSMO simulation results during the year 2013. X axis presents the simulated Tmin minus the reference simulation. Y axis presents the Meta-Model Tmin minus the reference simulation.Each point is for a different grid point and for a different day.

Same as left panel, for daily accumulated precipitation. Each point is for a different region and for a different day.

Thanks to these developments, the calibration methodology can now be readily applied to a NWP system and the reliability of the calibration results can be trusted. However, a full assessment of the impact of the soil memory is not available; this is an important issue, because it is expected that the impact of a new set of model parameters can be substantially stronger through the accumulation of heat and humidity in the soil over the full simulation period, as indeed observed in a preliminary experiment with a 1.1 km configuration of the COSMO model (a performance gain measured by the COSI score exceeding 10% has been observed following a one month calibration). Furthermore, in order for this method to be used by the COSMO community, it is essential to reduce the computing cost of the calibration; for that purpose, having a full calibration without any compromises is a pre-requisite.

Considering that (1) the multivariate calibration method has a huge potential for multiple applications in the NWP community, and (2) the project has shown that a quadratic meta-model is able to reproduce the dependency of the model quality on unconfined model parameters, a follow-up project for the period 06.2017 - 09.2019, CALMO-MAX, was defined and accepted by the StC.

The proposed objective calibration methodology has the potential to bring a transformative change to atmospheric model development and significantly reduce model development cycle times. The calibration method will be a very useful tool to improve the quality of the multiple configurations of the atmospheric models running in Europe and beyond. More specifically, the developed methodology could be used by each COSMO member to define an optimal calibration over the target area of interest, for re-calibration after major model changes (e.g. higher horizontal and / or vertical resolution), as well as for an unbiased assessment of different modules (e.g. parameterization schemes), and for optimal perturbation of parameters when run in ensemble mode. Last but not least, the implementation of the methodology for a specific parameter can clarify the impact of the specific parameter on the overall model performance. Once the meta-model has been fitted to the full COSMO NWP model both the effect of the parameter setting and parameter space used (i.e., the maximal range of optimal values) can be determined without the use of the full NWP model.

Documents and deliverables

The final report is available here;a summary talk is available here.

A detailed description of different phases of the project is available in the COSMO Technical Reports No 25 and No 31.

The code of the meta-model is available here, including the associated documentation.

A peer reviewed paper has been published in Atm. Res. vol. 190; a second peer reviewed paper is in the review process (see references below).

The original project plan is available here, the first extension (9.2015) is here, the second extension (9.2016) is here, the final extension (12.2016) is here.

Also available are the proposal for the Piz Daint allocation at CSCS, and the corresponding final report.

References

Avgoustoglou E., A. Voudouri, P. Khain, F. Grazzini and J.M. Bettems, 2017:
Design and Evaluation of Sensitivity Tests of COSMO model over the Mediterranean area. Perspectives on Atmospheric Sciences, Vol.1, Springer, pp 49-55

Bellprat, O., S. Kotlarski, D. Lüthi, and C. Schär. 2012a.
Objective calibration of regional climate models. Journal of Geophysical Research, 117, D23115.

Bellprat, O., S. Kotlarski, D. Lüthi, and C. Schär. 2012b.
Exploring perturbed physics ensembles in a regional climate model. Journal of Climate, 25, 4582-4599.

Bellprat, O., R. Elia, A. Frigon, S. Kotlarski, R. Laprise, D. Lüthi, and C. Schär. 2016.
Objective Calibration of Regional Climate Models: Application over Europe and North America. Journal of Climate.
https://doi.org/10.1175/JCLI-D-15-0302.1

Damrath, U., 2009.
Long-term trends of the quality of COSMO-EU forecasts. Presentation at the 11th COSMO General Meeting, Offenbach.
GM2009:wg5-versus2.

Duan, Q. et al. 2017.
Automatic Model Calibration: a New Way to Improve Numerical Weather Forecasting. BAMS, May 2017.
https://doi.org/10.1175/BAMS-D-15-00104.1

Khain P., I. Carmona, A. Voudouri, E. Avgoustoglou, J.-M. Bettems, F. Grazzini, 2015:
The Proof of the Parameters Calibration Method: CALMO Progress Report. COSMO Technical Report, 25.
techReport25.

Khain P., I. Carmona, A. Voudouri, E. Avgoustoglou, J.-M. Bettems, F. Grazzini, P. Kaufman, 2017:
CALMO Progress Report. COSMO Technical Report 31.
techReport31.

Lüthi, D. et al. 2017.
Report on the objective calibration of COSMO 5.0 within the COPAT project. COSMO-CLM community internal report.

Neelin, J. D., A. Bracco, H. Luo, J. C. McWilliams, and J. E. Meyerson. 2010.
Considerations for parameter optimization and sensitivity in climate models. Proc. of the National Academy of Sciences of the United States of America, 107, 21349-21354.

Voudouri A., P. Khain, I. Carmona, E. Avgoustoglou, J.M. Bettems, F. Grazzini, O. Bellprat, P. Kaufmann and E. Bucchignani, 2017:
Calibration of COSMO Model, Priority Project CALMO, Final report. COSMO Technical Report, 32.
techReport32.

Voudouri A., E. Avgoustoglou and P. Kaufmann, 2017:
Impacts of Observational Data Assimilation on Operational Forecasts. Perspectives on Atmospheric Sciences, Vol.1, Springer, pp 143-150

Voudouri A., Khain P., Carmona I., Bellprat O., Grazzini F., Avgoustoglou E., Bettems J.M and Kaufmann P., 2017b:
Objective calibration of numerical weather prediction models. Atm. Res. Vol. 190, 128-140
https://doi.org/10.1016/j.atmosres.2017.02.007

Voudouri A., Khain P., Carmona I., Avgoustoglou E., Bettems J.M., Kaufmann P and Grazzini F.:
Optimization of high resolution COSMO model performance over Switzerland and Northern Italy. Atm. Res. Vol. 213, 70-85
https://doi.org/10.1016/j.atmosres.2018.05.026