Priority Project "C2I"
Transition of COSMO to ICON-LAM

Last updated: 26 Aug 2021
get the pdf document

Project leader: Daniel Rieger (DWD)

0. Resources 1. Motivation and goals 2. Background 3. Related Projects 4. Participants 5. Actions proposed 6. Description of individual tasks 7. Risks 8. References 9. Estimated resources

0. Project resources

1. Motivation and goals

The COSMO science plan calls for a harmonization of development of the COSMO and ICON (ICOsahedral Nonhydrostatic) models in the time horizon of 2020. At DWD, further development of the COSMO model will be reduced in 2018 to the level needed for operational production. Development and testing of new features will be restricted mainly to ICON. It is therefore necessary for the COSMO consortium to start the migration to ICON-LAM (ICON - Limited Area Model) as the future operational model.

In first tests ICON-LAM outperforms the COSMO-model in terms of the quality of the results as well as in terms of computational efficiency.

This project represents a major step in the implementation of the COSMO strategy regarding the evaluation of ICON-LAM and the transition from COSMO to ICON.

The overall goal of the proposed priority project is to ensure a smooth transition from the COSMO model to ICON-LAM within a four-year period. At the end of the PP C2I, each participating institution is free to choose when ICON-LAM replaces the COSMO model in their operational forecasting system.

2. Background

In 2013, the COSMO partners have assessed the situation of the consortium and of the COSMO model based on trends and the evolution known at that time. They have defined the strategy Future of the COSMO Modelling System and the Consortium for Small Scale Modelling (COSMO) endorsed by all COSMO directors, which foresees a convergence of COSMO with ICON. Besides a contribution of COSMO to the development of ICON with the responsibility for the ICON-LAM version for NWP application, it foresees an assessment of ability of ICON-LAM to replace the COSMO model and the transition from COSMO to ICON for the Consortium partners.

In January 2015, ICON [7] replaced GME for global numerical weather prediction (NWP) at DWD. The effective grid spacing (defined as the square root of the average cell area) of the operational global ICON forecast is currently 13 km. After a parallel phase (started in July 2015), COSMO-EU was replaced in December 2016 by an ICON nested domain over Europe with 6.5 km grid spacing for operational forecasts. ICON operated in a limited-area mode will later on also take over the regional high-resolution forecasts currently performed with COSMO-DE (COSMO-D2 from May 2018 on). The current plan at DWD estimates a parallel phase of COSMO-D2 and ICON-D2 starting in the third quarter of 2019 and the final transition to ICON-D2 in 2020. ICON-D2 is an ICON-LAM setup at approximately 2 km grid spacing with a domain corresponding to COSMO-D2.

Following the COSMO science plan, the PP CDIC (Priority Project Comparison of the Dynamical cores of ICON and COSMO) has performed comparisons of the dynamical cores of the COSMO and ICON models from September 2015 until August 2018, showing similar or improved scores for ICON. According to the project plan, PP CDIC is a preliminary study before a larger priority project that performs comparisons of the complete two models (i.e. including physical parameterizations and data assimilation).

This PP C2I (Transition of COSMO to ICON-LAM) is the follow-up project that was already announced in the PP CDIC project plan. PP C2I aims to explore the functionality of the limited-area setup of ICON, to identify and remove obstacles for a transition from the COSMO model to ICON-LAM and to compare the two models using typical current NWP setups used by the consortium members including data assimilation. At the end of the project, each COSMO member is expected to be able to perform at least a quasi-operational daily ICON-LAM forecast. Besides these aims, a major benefit of this PP is the jointly gathered experience with ICON-LAM across the whole Consortium.

Ensemble configurations are parts of the operational chains at DWD, MCH, IMGW, CNMCA and Arpae. Hence, a further integral component for a transition to ICON-LAM is the generation of ensembles using suitable perturbation methods. Exploring and testing ICON-LAM ensemble configurations will be covered within the new priority project APSU. A strong collaboration will be estasblished between the priority projects C2I and APSU.

The following list provides an overview of several important details of ICON (no claim to be complete). The list also highlights differences to the COSMO model.

• ICON includes a hybrid parallelization leading to a better computational performance on today's CPU architectures. Hybrid parallelization means that the advantages of a coarse-grained MPI parallelization are combined with the advantages of a fine-grained OpenMP parallelization.
• A so far missing feature to be added to ICON is the availability of a dedicated GPU version. This is currently tackled for the climate application by the project ENIAC (ENabling the ICON model on heterogeneous ArChitecture) involving MeteoSwiss, DWD, the Max Planck Institute for Meteorology in Hamburg, the ETH in Zurich, and the Swiss Center for Climate Systems Modelling C2SM. This effort will be complemented by a COSMO PP tackling the weather application to be approved in September 2018.
• The ICON dynamical core offers better conservation properties than the dynamical core of the COSMO model. These are exact local mass conservation and mass-consistent tracer transport
• ICON provides a two-way nesting capability. This means that the grid spacing can be locally refined within one simulation. The boundary data are then provided from the original coarser grid. Using the optional two-way interaction, information from the finer resolved grid is also transferred to the original coarse grid. Technically, the nesting capability is nearly identical to the limited-area mode of ICON. The only difference is the boundary data supply. These two capabilities can even be combined, i.e., ICON-LAM can contain higher resolution nests.
• ICON-LAM can use ICON and IFS (ECMWF Integrated Forecast System) analyses and forecasts as lateral boundary conditions.
• In contrast to COSMO, ICON-LAM does not require the data for the full domain for boundary data supply. Hence, the pre-processing tool iconremap generates a strip grid along the boundary and provides data only for this strip. This results in a vast decrease of the amount of data to be read.
• ICON output formats are GRIB2 and NetCDF.
• Besides the deterministic 13 km forecast, the current NWP system at DWD includes a 40 member global ICON ensemble at 40 km with an ICON-EU nest at 20 km. Using the information from this ensemble system, the data assimilation for the global ICON forecast is performed with an EnVar technique. This technique combines a LETKF and a 3DVar system with a (ad hoc) weighting of 70% and 30% for the B-Matrix.
• For data assimilation, the COSMO model can use either nudging or LETKF. For ICON-LAM, no nudging scheme is available. However, LETKF or EnVar will provide initial states for ICON-LAM. For more details, see task7.
• Preliminary studies comparing ICON-LAM simulations with COSMO-model simulations show improvements with respect to verification as well as computational efficiency.
• Currently, two documentations of the ICON model are updated frequently. These are the ICON Training Course document and the Namelist Documentation.
• The ICON model is being used and tested regularly at the HPC (high performance computing) systems of DWD, ECMWF, DKRZ, and KIT.

From a technical point of view, the ICON-LAM model code (without data assimilation) is ready to be used for numerical weather prediction. At this stage, COSMO Consortium has the chance to step in and take over the coordination for further development and the maintainance of ICON-LAM and parameterizations of subgrid-scale physical processes on a convection-permitting scale. This future responsibility has already been pointed out in the COSMO science plan. By taking over this coordinating role, COSMO Consortium can prepare to take over the responsibility for ICON-LAM for NWP applications. Both, the ICON developers and COSMO benefit from this cooperation. As described above, ICON-LAM offers a great deal of advanced technical and scientific features compared to the COSMO model. On the other hand, the ICON development can benefit from COSMO. Besides the scientific knowledge and long-term experience in limited-area modeling within COSMO, the infrastructure including the organizational structure, support, website and documentation offers a beneficial framework for all future ICON users.

It is suggested that the PP C2I is implemented under the roof of WG6. Nevertheless, this project is a cross-cutting activity between several WGs with strong connections to nearly every WG.

3. Related Projects

The priority project C2I covers a wide range of important subtopics within COSMO. This is reflected by a close collaboration with different working groups (especially WG4, WG5 and WG6). Furthermore, a close collaboration with other projects is already being established or has to be established within the timeframe of the PP. The following collaborations have to be pointed out:

• Within PP C2I only a deterministic forecasting system will be established. Ensemble generation and ensemble-based forecasting systems with ICON-LAM, including a comparison of ICON-LAM and COSMO performance in the ensemble mode, are parts of the new PP APSU. This collaboration is already being coordinated.
• As pointed out in task5, a common verification software based on feedback files would be desirable. A prospective PT or PP within the framework of WG5 will benefit from PP C2I and vice versa.
• The development of a dedicated ICON-LAM GPU version is beyond the scope of PP C2I. Hence, a close collaboration with the project ENIAC as well as with the COSMO PP foreseen on this matter will be established.

4. Participants

As the transition to ICON-LAM is a cross-cutting activity across the whole Consortium, each COSMO member state as well as the CLM and ART communities should participate in this priority project. The major COSMO partners ARPAE, ARPA Piemonte and CIRA take part in this PP. In addition, INMET Brazil participates as well as one of the major licensees of COSMO that has already declared interest in a transition to ICON-LAM.

5. Actions proposed

The overarching aim of PP C2I is that each participating institution has a working deterministic ICON-LAM forecasting system at the end of the project. For some of the participants, the final forecasting system includes a data assimilation procedure as well. The most natural way to achieve this goal is to perform one step after another. Hence, the PP C2I can be subdivided into three phases.

Phase 1 - Preparation & Installation

Phase 1 starts with the ICON-LAM training course in April 2018 and lasts until December 2018. In this phase, the participating institutions install ICON on their HPC systems and gain experience in running the model. At the end of Phase 1, setups that are similar to the institutions' current deterministic COSMO-model setups are defined and tested.

Phase 2 - Basic Forecasting System

In Phase 2 starting from January 2019, the participating institutions perform at least one daily deterministic ICON-LAM forecast without data assimilation. For this, the setups from Phase 1 are used.

Considering limited capacities of the HPC systems at some institutions, only one additional deterministic forecast per day is foreseen within this project. Although desirable, this does not have to be in real time.

In this phase, the verification can be adapted to ICON-LAM products. Preparatory work is performed towards a survey for the feedback from the forecasting departments. Furthermore, due to the similarity with the COSMO setups, the computational efficiency of ICON and COSMO can be compared as well. With regard to Phase 3, preparatory work is initiated towards a data assimilation system.

Phase 3 - Deterministic Forecasting System Including Data Assimilation

Starting in July 2020, data assimilation cycles are added in Phase 3 to the ICON-LAM forecasting systems. In this last phase, the most important goals are the verification of the ICON-LAM results, the feedback from the forecasting departments, and the tuning of the setups. With the end of the project in March 2022, each participating institution has a working deterministic ICON-LAM forecasting system. As stated previously, ICON-LAM ensemble prediction systems are covered within the PP APSU where a strong collaboration with the PP C2I will be established.

CLM Community

For the CLM Community, the PP C2I can also be subdivided into three phases with approximately the same timeline. In Phase 1, the model code (ICON with NWP physics packages) is installed on some of the machines which the community members typically use to produce climate simulations. The necessary adaptions to be able to run the model for a longer time period with ERA-Interim boundary conditions and NetCDF I/O are performed. In Phase 2, the infrastructure, which is necessary to conduct climate simulations, is developed and the model components important on longer timescales are adjusted if necessary. Furthermore, sensitivity tests are performed to find a suitable model setup for climate simulations. A reference simulation over a period of 20 or 30 years with ECMWF reanalysis boundary conditions is conducted in Phase 3. This simulation is the basis for the evaluation of the model performance and the comparison to CCLM.

ICON-ART

At KIT, the development of COSMO-ART regarding gasphase and aerosol processes has come to an end. Current activities concentrate on applications of the model system addressing scientific questions by academia and operational activities like pollen forecast (MeteoSwiss) and air quality forecast (Roshydromet). KIT will make sure that COSMO-ART is constantly updated to the newest version of the COSMO model.

In 2011, KIT and DWD started the development of ICON-ART with the aim to provide improved (or at least the same) functionality as COSMO-ART. This work is reaching a mature state.

KIT participates since the beginning in the planning and conduction of the ICON training courses at DWD to educate users of ICON-ART and inform about the current status. Being part of the ICON development team, KIT participates in and, since recently, even organizes ICON Developers Meetings. Multiple scientific applications of ICON-ART are already published [1, 2, 3, 4, 5, 6]. ICON-ART runs pre-operationally at DWD for daily mineral dust forecasts on the global scale including a nest with higher spatial resolution centered around Germany.

Within the framework of PP C2I, a version of ICON-ART will be built up that allows its application in the limited-area mode. The main activity here is the provision of boundary and initial conditions for gaseous compounds and aerosol particles. Furthermore, KIT will support the transition from COSMO-ART to ICON-ART with lectures and exercises during the ICON Training Courses and by supporting future operational as well as acadamic ICON-ART applications.

6. Description of individual tasks

task 1: ICON Training 2018

Phase 1 of the PP C2I is kicked off at the ICON Training 2018, which is held from 16 April to 19 April 2018. The hands-on exercises encompass the model installation, the grid generation process, the generation of input data, an ICON-LAM model setup, and post-processing tasks.

Each participating institution should send at least one scientist to the ICON Training 2018. This scientist will also be the local contact person for technical questions during the PP C2I, especially with respect to the installation of ICON and the pre- and post-processing software dwd-icontools.

task 2: Installation of ICON

After participating in the ICON Training 2018, the next steps are the installation of the model ICON-LAM and the pre- and post-processing software dwd-icontools. In order to faciliate common developments in the future, we recommend the installation of the official COSMO post-processing software fieldextra, which is already compatible with ICON. The installation is performed at the HPC cluster of each participating institution. In contrast to the COSMO-model, ICON uses the more recent FORTRAN2003 standard and a hybrid OpenMP/MPI parallelization. This in turn demands more recent compiler and library versions. Hence, a support framework for ICON-LAM has to be established. This is further described in task 8.

Deliverables:
ICON-LAM executable on the HPC of each participating institution.

task 3: Definition of the forecast setup

The setup for ICON-LAM has to be chosen in a way that it matches the highest-resolution deterministic COSMO-model forecast setup of the participant as closely as possible. This has two major advantages. Firstly, no additional data has to be transferred as the data from the global ICON or IFS forecast used for the COSMO model can be used for the ICON-LAM forecast as well. Secondly, using a similar setup for the COSMO model and ICON-LAM simplifies the comparison of the two models' forecasts. In summary, it is important to use a domain with a roughly equal extent and grid spacing as in the operational COSMO-model forecast of the participating institution. In case the COSMO-model setup changes during the course of the PP C2I, a corresponding adaption of the ICON-LAM setup is recommended.

For comparison reasons, it is important for the CLM community to use similar COSMO-model and ICON-LAM settings as well.

The basic forecasting system proposed for Phase 2 of the project starts from an analysis of global ICON, ICON-EU or IFS (depending on the COSMO-model setup of the participating institution) and retrieves its boundary data from the corresponding forecast. Forecasts are performed once per day with a leadtime of at least 27 h. The soil is initialized with the soil from the previous day's forecast (free running soil, i.e., no daily reinitialization). For Phase 3, the initial state by one or the other data assimilation procedure.

Using this setup, a hindcast simulation for a severe weather event (e.g., heavy rain) is performed at each participating institution. The outcome of this event can then be analyzed further. This leads to first experiences with the model output as well as the model performance.

The model setup for the benchmarking simulation of the CLM-Community will be determined in Phase 2. This simulation will be a hindcast with ECMWF reanalysis boundary conditions for a period of 20 or 30 years (e.g. 1971-2000 or 1981-2000).

Deliverables:
Each participating institution has a running basic deterministic forecasting system by the end of Phase 1, i.e. at the end of 2018. Additionally, a report (preferably a COSMO newsletter article) on the outcome of the severe weather event case study will be prepared.

task 4: Computational aspects

As stated in the previous sections, the setup (re.\ domain size and grid spacing)  of the ICON-LAM forecasts is similar to the setup of the operational COSMO-model forecasts. Using the same number of processors, this allows for a simple comparison of the wall clock time of the two models. For this comparison, technical features of the individual models that decrease the computational costs in forecasts should be used when possible.

This simple wall clock time comparison covers the computational aspects between the two models only on a very basic level. A more rigorous comparison is beyond the scope of this already work-intensive priority project. However, with the sum of tests at the different institutions, some first conclusions can be drawn.

Nevertheless, the individual institutions are encouraged to perform more detailed tests with ICON-LAM. Especially strong scaling, weak scaling, memory consumption and peak performance ratio have the potential to provide a valuable insight into the performance of ICON-LAM compared to the COSMO model. If a dedicated GPU version is available within the scope of PP C2I, a summary on CPU/GPU computational aspects is highly desirable.

Deliverables:
A report (preferably a COSMO newsletter article) about the outcome of the tests.

task 5: Verification

With the daily deterministic ICON-LAM forecasts starting in Phase 2 of PP C2I (January 2019) one of the major tasks will be to validate the results. ICON offers the possibility to provide output on a regular longitude-latitude grid. Hence, the post-processing and validation tools used by the individual participating institutions for the COSMO-model (e.g. VERSUS) can be relatively easily adapted to work with ICON results. The usage of the same validation software for the COSMO model and ICON-LAM eases the intercomparison of the models' performance as well. It has to be pointed out that ICON does not offer the possibility of GRIB1 output. The available output formats are NetCDF and GRIB2. However, a conversion from GRIB2 to GRIB1 is supported by the official COSMO post-processing software fieldextra. The overarching goal of this task is the identification of systematic biases and to gain an objective comparison between the COSMO model and ICON-LAM. Strictly speaking, the latter requires a similar data assimilation procedure to be used by both models.

WG5 will provide verification guidelines that can be followed by the partners. For example, in the hindcast mode analysis of an extreme event the validation of precipitation could provide much more information in terms of performance ability if it is based on spatial methods (both neighborhood and feature-based) rather than on traditional point verification. However, for a longer term validation of performance (seasonal), simple scores based on point verification can be more appropriate and easy to follow.

In a long-term perspective, it is desirable to have a unified verication system for station-based verification within COSMO. This should be achieved with the Rfdbk software developed at DWD. The use of Rfdbk requires the creation of model feedback files at each participating institution. This includes providing the observations in a specific format and installing the model equivalent calculator (MEC). This will be supported by a currently planned PT/PP on Rfdbk (to be approved in September 2018). It should be easier to set up and use this software once a data assimilation has been established. Rfdbk is about to be tested for the verification of the COSMO test suite at ECMWF. The decision whether it might also be beneficial to use an Rfdbk-MEC based verification system at each national center can be made after reviewing the outcome. However, building up the capacities for a unified verification software within COSMO requires major resources from the COSMO members. Therefore, it is left to a PP or PT within the framework of WG5 where a strong cross-link to PP C2I could be established.

The CLM community follows a similar approach. As ICON data can be provided on a regular longitude-latitude grid, the existing validation software packages (e.g., ETOOL) can be used to compare the results to measurements. In addition, a standardization of the model output should take place before any further post-processing. Therefore, the existing CMOR (Climate Model Output Rewriter) tools have to be adjusted for ICON-LAM output and included in the runtime environment.

Deliverables:
Verification reports based on individual reports from each participating institution at the end of Phase 3.

task 6: Forecasters' feedback

The main goal of the task Forecasters' feedback is the evaluation of ICON-LAM forecasts by forecasting departments and the translation of this feedback to the COSMO-ICON community. To this end, the following steps should be made:

• an assessment of the added value of ICON-LAM compared to the COSMO-model,
• identification of decreased model skills for certain regions or certain weather regimes,
• an assessment of the added value of ICON-LAM compared to the COSMO-model in case of severe weather situations, and
• additional requirements of ICON-LAM users with respect to data format and output meteorological variables should be formulated.

The above goal is achieved by providing forecasting departments and, if available, other COSMO and ICON-LAM users with regular surveys. These surveys are collected and processed by a WG4 representative from each of the participating institutions at the end of Phase 3. Using the outcome of these surveys, an assessment will be created that includes the subjective evaluation by forecasters and the objective verifications obtained by task 5. A comparison of the subjective and objective verification results is then made. It is reasonable to provide the forecasting departments with the ICON-LAM forecasts and the survey only after a data assimilation procedure is established. Retrieving forecasters' feedback before establishing a data assimilation procedure is voluntary and should only be performed in case internal verification showed an added value compared to COSMO-model forecasts.

Deliverables:
A report containing the outcome of the surveys as well as a comparison with the objective results from task 5 at the end of phase 3.

task 7: Data assimilation

Clearly, forecasting needs a proper analysis of the initial state for each single forecast. Here, data assimilation is one of the two pilars of numerical weather prediction, namely, analysis and forecasting. If the analysis for a global system is available, interpolation is an easy way to generate some initial states for regional forecasting. Then, the high-resolution forecast still generates added value in comparison to global models (e.g., by explicitely treating deep convection). However, for high-quality forecasting - in particular of precipitation and high-impact weather - the use of a state-of-the-art data assimilation system with both classical as well as remote sensing observations, such as RADAR reflectivity or RADAR radial winds, is an important ingredient of a state-of-the-art regional NWP system.

In the years 2018 and 2019, DWD plans to adapt the data assimilation system available for the COSMO model to ICON-LAM, i.e., to start with assimilation tests based on the local ensemble transform Kalman filter (LETKF) on which KENDA is based, in combination with latent heat nudging (LHN) for the radar data applied to all members of the ensemble. The further development includes several layers and steps.

1. The model equivalent calculator MEC is tested in the first half of 2018. It is a basic ingredient for verification. MEC can also be used to run a MEC based version of the KENDA ensemble data assimilation system for ICON-LAM. It can be compared to the MEC-based version of the COSMO-KENDA system and to the full COSMO-KENDA system.
2. The observation operators are adapted to ICON-LAM. Tests with the complete system are carried out in 2018, leading to a consolidated version of the ICON-LAM KENDA ensemble data assimilation system in the first half of 2019. The ICON-LAM-KENDA system corresponds to the COSMO-KENDA system operational at DWD.
3. DWD plans to develop a three-dimensional variational assimilation system (3dVAR) and an ensemble variational data assimilation system EnVAR for ICON-LAM in 2018 and 2019. This provides the opportunity for COSMO member states and partners to run.

• KENDA (LETKF) based ensemble data assimilation analysis and forecast cycle with 40 or more members,
• 3dVar based deterministic analysis and forecast cycle for ICON-LAM,
• deterministic variational EnVAR analysis and forecast cycle, based on the boundary conditions and covariances of the global ICON-EPS run operationally at DWD, and
• the full LETKF+EnVAR ensemble variational data assimilation analysis and forecast cycle.

All COSMO Partners should gain experience with the feedback file (fdbk file) generation in an early stage, using the model equivalent calculator MEC to generate feedback files for verification, visualization and diagnostics, which provides access to a large suite of diagnostic tools available within the data assimilation coding environment (DACE) at DWD.

It needs to be stressed that the development of data assimilation procedures is not part of this priority project. The task Data Assimilation aims rather at adapting established systems for ICON-LAM.

Deliverables:
A report which describes and evaluates the experiences of COSMO partners with the different data assimilation systems which are being developed for the ICON-LAM analysis cycles.

task 8: Technical framework

The installation and running ICON will require a certain amount of training and support. As a prerequisite for the installation phase, a document will be prepared based on a survey that is currently conducted. In this survey, ICON licensees are asked about their experiences with installing ICON and about compiler and library version they use. The document will contain information on the following aspects:

• lists of versions of different compilers and MPI libraries that are know to work or known to fail for the installation of ICON,
• a list of additional libraries that are necessary to install ICON including recommended versions, and
• a list of recommended pre- and post-processing software.

In case of severe problems with the installation, a fallback solution is to provide support for the installation of the complete software stack starting with the installation of gcc (Gnu Compiler Collection).

The ICON-LAM 1st-level-support can be handled by the same people (from NMA Romania) that are responsible for the COSMO 1st-level-support. For this task, DWD offers an extra training for them. The ICON developers take care of the 3rd-level-support, i.e., by providing critical bug fixes. Adding new features to the ICON code remains the responsibility of a person who wants the feature to be implemented. This support framework is targeted at CPU architectures. As soon as a dedicated GPU version is available, support for ICON-LAM on GPU architectures has to be established within a corresponding project.

As ICON-LAM will receive updates in a comparetively high frequency  from this project as well as from the ICON developers, the source code management is an important issue. The question is how to make ICON-LAM and updates to the code available to the COSMO Consortium and how to establish a development workflow. Out of convenience, the workflow should be adapted to the distributed Git repository system that is used by the ICON developers. There are several options how to achieve this goal. These have to be explored in the scope of this priority project. One seminal option that will be explored is to create a Git reprository for the COSMO community. The ICON code in this repository can then be sychronized with the ICON repository hosted in Hamburg.

The COSMO test suite is a well-established part of the COSMO-model development workflow. Building up a similar framework for ICON-LAM is highly desirable. Hence, an ICON-LAM testing environment should be established in Phase 1 of this priority project. The goal of this testing environment is to ensure that ICON-LAM is always functional at the ECMWF HPC.

Deliverables:
A document containing information on the installation of ICON as listed above. This document will be available shortly after the ICON Training course 2018. A Git repository to exchange the ICON code by the end of Phase 2 as well as a testing environment by the end of Phase 1.

task 9: ICON-ART

At the current stage, it is not possible to read boundary conditions for aerosol particles and gaseous compounds. As a first step, this funcionality has to be added to pre-processing tools and to the input modules of ICON-ART. This work is estimated to be finished within 2018.

Besides the academic usage of ICON-ART, the ART developers offer support for operational applications within the priority project C2I. This includes the transition of currently performed COSMO-ART forecasts to ICON-ART in limited-area mode as well as support for establishing new operational applications of ICON-ART. Currently, there are three operational applications of ART.

1. The pollen forecasting system could be migrated from COSMO-ART to ICON-ART. At MeteoSwiss, a pollen forecast is performed routinely with COSMO-ART. A pollen parameterization is already implemented into ICON-ART.
2. At Roshydromet, air quality forecasts are performed routinely with COSMO-ART. Currently, the formation of secondary aerosol, which is a necessary component for air quality forecasts, is under development and testing in ICON-ART at KIT. A version that could be used for air quality forecasts should be available within 2018. Therefore, a transition of the Roshydromet air quality forecast from COSMO-ART to ICON-ART can possibly be undertaken within the framework of PP C2I.
3. Within the framework of the PerduS project, preoperational global mineral dust forecasts are performed at DWD since October 2017. This data could be used as boundary condition for highly resolved mineral dust forecasts or as a replacement for the climatological values of mineral dust aerosol optical depth. The latter is currently done within the PP T2(RC)2 for COSMO-ART. This work can also be performed for the limited-area version of ICON-ART.

These operational applications of COSMO-ART und ICON-ART give an examplary overview on possible future ICON-ART applications. The ART developers will support efforts for transitions from COSMO-ART to ICON-ART and will also support new operational applications.

7. Risks

With the creation of daily deterministic ICON-LAM forecasts at each participating institution, the overall goal of this PP in the given time frame is rather ambitious. For example, technical issues due to the high requirements of ICON regarding the up-to-date software stack at the instutions' HPC can cause a delay. This problem is tackled by providing support for the installation and a fallback solution that installs the complete software stack. However, even for this setup not all technical issues during the installation phase are foreseeable.

As stated before, a dedicated GPU version of ICON does not exist yet. A currently ongoing three-year PASC (Platform for Advanced Scientific Computing) project called ENIAC is working on a GPU version of ICON with a focus on the climate physics package. As some COSMO members are using GPU architectures for operational numerical weather prediction, the creation of a GPU version has to be tackled by the COSMO consortium to efficiently use ICON-LAM (e.g., within the framework of a PP). A close collaboration with the ENIAC team should be established. There are many unpredictable issues related to the creation of a GPU-capable version of ICON-LAM. For example, OpenACC was used so far to port the ICON code to GPUs. Cray, however, decided not to support OpenACC in favor of OpenMP in the future. GCC is working on the OpenACC support but this development has not reached a mature state yet. This currently leaves the PGI compiler as the only option. One way to circumvent such problems is to use a source-to-source translator like CLAW to be flexible in terms of OpenMP, OpenACC and other paradigms. The creation of a GPU-capable version is beyond the scope of this priority project.

A possible solution could be to use ECMWF HPC within the framework of this project if a GPU version is not available and one partner using GPU has not enough CPU capacity. The institutions using GPU architectures for NWP are able to already gather experience with ICON-LAM until a GPU version is ready. This is especially the case for post-processing and verification of ICON-LAM results.

As the development of ICON is at a high rate, the ICON code should be updated frequently (half-yearly). Although tested prior to release at DWD, these updates might cause technical problems. However, as the previously used version should be working, there is always a fallback solution.

Depending on the chosen data assimilation procedures, data transfer rates might turn out to be a bottleneck for data assimilation methods using information from the global ICON-EPS.

8. References

9. Estimated resources