0. Project resources

1. Summary

The aim of the project is to further improve and extend the KENDA data assimilation system and use of observations in view of better convective-scale deterministic and ensemble forecasts with ICON-LAM particularly for weather-related quantities. On the one hand, this is addressed by extending the use of observations, with a focus on those related to cloud and precipitation, but also those in the planetary boundary layer and near the surface.

On the other hand, the current LETKF scheme will be complemented by a variational component for an ensemble-variational analysis (EnVar) and, at least experimentally, by a particle filter (PF) in order to address certain limitations of the LETKF. Furthermore, tools will be provided to perform a variational data assimilation without the need to run a (convective-scale) ensemble in the assimilation cycle. This offers the possibility to carry out data assimilation for ICON-LAM at similar low cost as it was possible with the observation nudging scheme for the COSMO model.

2. Background and motivation

In the precedent projects KENDA (Km-scale ENsemble-based Data Assimilation) and KENDA-O (KENDA for high-resolution Observations), a 4-dimensional Local Ensemble Transform Kalman Filter (4D-LETKF) scheme following Hunt et al. (2007) has been developed for the COSMO model and ported to the ICON-LAM model. Latent heat nudging (LHN, Stephan et al., 2008) of radar precipitation has been integrated into the KENDA-LETKF analysis cycle.

The main purpose of this data assimilation (DA) system called KENDA (Schraff et al., 2016) is to provide the initial conditions both for deterministic and ensemble forecasting on the convective scale (i.e. with 1-3 km model mesh size). Foci of interest include improved forecasts of high-impact weather, e.g. in convective situations or for fog and low stratus conditions, and especially in the very short time range. It is running operationally now at several COSMO member states (Germany, Switzerland, Italy), providing the initial conditions for operational forecasts.

Compared to the previous observation nudging scheme, the introduction of KENDA has resulted overall in large improvements in the EPS forecasts and more moderate but still significant improvements in the deterministic forecasts, particularly of convective precipitation in summer. This benefit of KENDA prevails throughout except for two aspects:a) low stratus, which has been degraded in some cases, notably over the Swiss Plateau, and b) 2-m humidity which has been found to be slightly degraded (in MeteoSwiss verifications) since 2-m humidity observations have been assimilated only in the nudging scheme up to now. Apart from the 2-m humidity data, these overall very positive results were achieved using a similar set of observations.

The ability of the KENDA-LETKF to derive and use flow-dependent background error covariances for the computation of the analysis increments certainly plays a major role for this. Compared to the nudging scheme with its limitations to assimilate indirect observations, the LETKF also offers much better prospects to use additional data. The greatest success so far has been the recent operational introduction of radar radial velocity and reflectivity volume data in the LETKF, with clear benefits found in the main tests (at DWD and at ARPAE (Gastaldo et al., 2018)). With this, the COSMO consortium is the first one (in Europe and at national level worldwide to our knowledge) to assimilate operationally 3-D radar reflectivity directly without any retrieval approach.

Besides this great potential of the LETKF, there are also certain limitations and issues (some of which may contribute to the shortcomings with low stratus), e.g.:

• In the LETKF, the available number of degrees freedom to fit the observations within the localisation scale does not exceed the number of ensemble members. This rank deficiency problem poses a limitation for the use of high-resolution data. Enhancing the effective degrees of freedom in the analysis by reducing the localisation scale may lead to imbalances and incomplete consideration of real background error covariances. The variational approach and inclusion of climatological covariances may mitigate this problem.
• The LETKF deploys localisation in observation space by increasing the errors for observations further away from a given analysis grid point and discarding the data beyond a certain distance limit. This poses a problem for the treatment of non-local observation such as satellite radiances and GNSS (GPS) total delay data. It also does not allow for localisation between the analysed variables, i.e. the local cross-covariances given by the ensemble cannot be reduced in the analysis equation. (Extensions of the algorithm like multi-step approaches or (univariate) additive background covariance inflation may mitigate the latter aspect).
• Systematic model errors leading to model bias are difficult to account for in data assimilation, and current operational schemes do not do this (in the troposphere, except that bias corrections can be applied to observations in order to unbias the latter with respect to the model). This problem is prominent for ensemble Kalman filters (EnKF) which in this case often underestimate the background errors as they derive the covariances from the equally (or similarly) biased ensemble members.

• The LETKF makes the Gaussian assumption. The probability density distributions of the first guess or the observation errors, however, are often non-Gaussian and multi-modal in the convective scale and in particular for weather related variables such as cloud and precipitation. Furthermore, even though the LETKF requires only the application of the full non-linear observation operator ¹, it takes implicitly a linear assumption. In cases of non-linearity and non-Gaussianity, the LETKF analysis is therefore not optimal.

  ¹  The terms ‚observation operator‘ and ‚forward operator‘ are used synonymously for the computation of a simulated observation (i.e. model equivalent related to an observation) from the model state.

Other data assimilation methods offer advantages for some of these issues but also have their own disadvantages compared to the LETKF:

The particle filter (PF) addresses particularly non-Gaussianity and nonlinearity. At DWD, two PF variants, a Localized Adaptive Particle Filter (LAPF, see Potthast et al., 2019) and a Localized Markov Chain Particle Filter (LMCPF), have been developed. They compute an ensemble transform matrix locally as in the LETKF, but replace the LETKF update in ensemble space by a PF update and resampling in ensemble space. In this way, they overcome the problems of filter collapse (i.e. having far too little spread) and divergence (from the true state) that are typical for traditional PF applied to high-dimensional problems. Potthast et al. (2019) were the first to run a PF stably over a long period for a global NWP system in a quasi-operational setting and obtained forecast quality comparable to that from the LETKF. The LMCPF has also been found to work similarly for the COSMO model on the convective scale.

Compared to global DA, non-linearity and non-Gaussianity tend to play a larger role for convective-scale DA, both with respect to the involved (convective) model processes with their short time scales and with respect to the higher priority to assimilate data related to cloud and precipitation with highly nonlinear observation operators. Therefore, it appears particularly promising to continue the research on PF in the context of ICON-LAM. This will also attain a lot of attention and visibility in the scientific community.

In variational data assimilation (Var), certain aspects of non-linearity and non-Gaussianity can also be better handled than in the LETKF. The cost function may include nonlinear terms, and variational quality control (VarQC) helps to deal with observation departure statistics with fat tails by weighting down outliers. The iterative approach can improve the analysis in the presence of a non-quadratic shape of the cost function; on the other hand, the tangent linear and adjoint are required for the observation operators and in 4DVar for the NWP model. Furthermore, Var does not suffer from the rank deficiency problem and may potentially allow for using data at higher density (depending on the background error covariances applied), and it avoids the localisation in observation space.

At DWD, an EnVar scheme following ideas from Buehner et al. (2005) has been developed and is running operationally for global DA/NWP. Technically, it can be seen as a natural evolution of the 3DVar scheme combining the climatological background error covariances (B-matrix) of 3DVar (currently with a weight of 30 %) with the B-matrix derived from the ensemble perturbations of the associated LETKF (with a weight of 70 % as obtained by tuning). In this way, this hybrid EnVar (‘hybrid’ commonly denotes this type of blending of B-matrices) combines the main advantages of the two methods, Var and LETKF:It avoids or mitigates the above-mentioned shortcomings of the LETKF and at the same time benefits from the flow-dependent ensemble B-matrix. As a result, the forecasts started from EnVar analyses have been found to be significantly superior to those from the former 3DVar and to those from the analysis mean of the LETKF (which however is run at three times lower resolution). A four-dimensional version of EnVar known as 4D-EnVar is under development at DWD, details will be discussed below..

Note that EnVar provides only a single analysis for a deterministic forecast (unless a whole ensemble of EnVar’s is run) and still relies on the LETKF for the ensemble initial conditions. However, it is possible to apply ‘re-centering’ of the analysis ensemble around the EnVar deterministic analysis (by shifting all ensemble members in the model space in the same way such that the mean of the re-centered analysis ensemble coincides with the deterministic analysis). Recent tests for the global ICON system have shown large improvements of the EPS forecasts, when re-centering was applied to the LETKF ensemble during the data assimilation cycle and to the final analysis ensemble. Small improvements are obtained even in the deterministic analysis and forecast which obviously benefit from the improved B-matrix derived from the re-centered ensemble.

These results and considerations strongly promote the development and test of the EnVar for the convective scale (ICON-LAM) even though EnVar produces only a deterministic analysis and the focus of the COSMO consortium is on convective-scale EPS which would still rely on the (possibly re-centered) LETKF. For DWD, using EnVar operationally for ICON-LAM would also further harmonize the global and regional DA systems and increase synergies in research, development, and maintenance. It is worth mentioning that this development moves in parallel to the major trend of methods used at other major centres or European consortia for convective-scale DA:The UK Met Office deploy 1-hourly 4DVar (currently with climatological B-matrix, but working towards a hybrid B-matrix, see Milan et al., 2020). MeteoFrance, HIRLAM (HARMONIE-AROME), and JMA all use 3DVar but are working towards 4DVar and testing hybrid EnVar and 4D-EnVar (Gustafsson et al., 2018). NOAA-NCEP plan to introduce hybrid EnVar (HRRRDAS) in late autumn 2020.

Despite DWD’s encouraging experience with EnVar and recentering of the ensemble, it has yet to be seen, however, whether EnVar will really be able to improve upon the current KENDA-LETKF, for several reasons:

• At the convective scale, balances are much more flow-dependent and unknown than at larger scale. Therefore, adding a climatological part to the ensemble B-matrix in a hybrid EnVar will likely have less benefit than at global scale, if at all.

• The standard EnVar (currently used for the global system at DWD) is 3-dimensional. The time dimension for the computation of the first guess departures and for the ensemble B-matrix in the current 4D-LETKF of KENDA would be lost. Past tests with KENDA have shown that 3D-LETKF does not perform as well as 4D-LETKF even though the difference was smaller than expected.

  It is worth mentioning however that a first version of a FGAT-EnVar (FGAT:First Guess at Appropriate Time) has already been developed at DWD. This computes the model equivalents to the observations by including temporal interpolation of the fields from several time slots of the first guess run to the observation times. Furthermore, a 4D-EnVar is under development (focusing first on the global system) which will account additionally for the time dimension in the ensemble B-matrix. Note that unlike classical 4DVar, 4D-EnVar does not require the tangent linear and adjoint of the forecast model (or an approximation of it).

• A part of the performance gains from re-centering the ensemble in the global ICON system can be explained by the three times higher (horizontal) resolution of the deterministic analyses and forecasts compared to the ensemble. Current plans for convective-scale configurations however do not envisage different resolutions for the deterministic and ensemble runs. Due to the small scales, strong non-linearities and non-Gaussianities involved in situations with (explicitly simulated) deep convection, it is not clear whether re-centering of the ensemble will work beneficially at all at the convective scale (in convective situations). If not, the potential for improvements from EnVar would be limited to the deterministic forecasts (except for secondary effects e.g. from VarQC).

• Using the framework of LETKF has allowed the COSMO consortium to be the first (to our knowledge) to introduce the direct assimilation of 3D radar reflectivity operationally (albeit still in combination with latent heat nudging (LHN)). The other consortia and major centers, all deploying some variants of variational DA (Var) as primary algorithm, chose to resort to other methods for their operational applications up to now.. Meteo-France, HIRLAM, and JMA assimilate columns of relative humidity pseudo-observations derived from reflectivity (Wattrelot et al., 2014). NOAA-NCEP deploy a particular latent heating method for their convective-scale HRRR model (Benjamin et al., 2016). The Met Office applies LHN for their UKV model even though the development of radar reflectivity assimilation within 4DVar is at an advanced stage and set to replace LHN (Milan et al., 2020). Issues to assimilate reflectivity in Var directly include the choice/extension of the control vector, explicit linearization of strongly nonlinear processes and relationships, and formulation of background error covariances related to moisture and hydrometeors. COSMO will also face these issues when tempting to introduce EnVar and will need to come up with a practical solution to assimilate these data.

  This kind of issue is prominent for radar reflectivity, however it may also be present for other types of data, and quite certainly for some observation operators developed and tested in experiments with the LETKF. This includes the assimilation of nowcast objects and the way the assimilation of cloud top height (based on SEVIRI and radiosonde data) was implemented experimentally in a former KENDA task (Schomburg et al., 2015). Since Var requires the tangent linear and the adjoint of the observation operators, it can make their formulation and implementation much more intricate and restricted than LETKF or PF.

  As a result, the benefit from EnVar can be severely hampered if these types of data, often related to cloud and precipitation, are to be assimilated.

Yet, there is another important reason for the COSMO consortium to develop and provide Var. For the COSMO model, observation nudging has allowed institutes from member states or licensees with quite limited computational resources to carry out data assimilation as the nudging scheme is cheap and does not require to run a (convective-scale) ensemble in the data assimilation cycle in contrast to LETKF. As observation nudging is not available for ICON, a cheap surrogate approach without the need to run a convective-scale ensemble is needed, and this can be provided by Var.

• One solution is classical 3DVar, using a climatological B-matrix. For convective-scale NWP, this does not appear ideal because the real background errors are very flow-dependent at that scale, notably for high-impact weather. Furthermore, the climatology needs to be computed for each new model configuration in order to derive a suitable, adjusted B-matrix unless one wants to rely on the statistics of another, possibly similar configuration.

• Another solution is to derive an ensemble B-matrix from the ensemble members of a global ensemble (data assimilation) system. NOAA-NCEP have deployed this approach for their convective-scale system HRRR (Gustafsson et al., 2018) until June 2020 and also for their regional system RAP since it has resulted in immediate significant improvements over classical 3DVar (Wu et al., 2017). Even though convective motions are missing, the method is still able to capture the larger-scale flow dependency of the background errors. To avoid misunderstandings, we will tag this approach CEnVar (Coarse-Ensemble Var). This avoids the need to run an ensemble (data assimilation) at convective scale, however the prerequisite is to have enough bandwidth to transfer the ensemble fields in time from the center (e.g. DWD for ICON) that runs the global system and provides these fields.

  Naturally, there is also the possibility then to combine this ensemble information and a climatological B-matrix for a hybrid CEnVar.

Besides developing these algorithmic aspects, it is also very important to further extend the set of assimilated observation types for operational purposes. By and large, this will continue the research and development carried out in the previous KENDA-O project. Besides Mode-S aircraft data, radar radial velocity and reflectivity (for earlier studies, see Bick et al., 2016; Gastaldo et al., 2018) could be introduced operationally and successfully into KENDA within the framework of that project. It is now important to carry on maintaining, refining, improving, and (geographically) extending the use of these data and possibly adjusting it to significant changes in the model (e.g. 2-moment microphysics) and in the assimilation scheme as mentioned above. For some other observation types such as GNSS (Global Navigation Satellite System) zenith and slant total delay (ZTD and STD) and 2-m temperature and humidity from Synop stations (used to modify the atmospheric state), promising results have already been obtained, and it appears to require only few development steps and more testing to allow for operational use. There are also observation types such as all-sky satellite radiances where work is ongoing and more work is needed despite the progress achieved.

Even though any type of observation can be beneficial and important to be used, there are two classes of data for which further research, development, and extended use are particularly relevant for convective-scale analysis and forecasting of weather, in particular locally driven (or influenced) high-impact weather such as deep convection or low stratus and fog. The first class is high-resolution data related to humidity, cloud, and precipitation. Besides radar reflectivity (precipitation) and ZTD/STD (humidity) as mentioned above, a focus is on the all-sky VIS and IR satellite radiances (here from Meteosat) in view of assimilation of information on cloudiness (for an overview on all-sky satellite DA, see Geer et al., 2018; Kurzrock et al., 2018). For VIS radiances, a novel efficient forward operator (MFASIS) has been built, and promising results have already been obtained in a comprehensive case study for COSMO-KENDA by Scheck et al. (2020) and even for longer test periods with ICON-LAM at DWD. Related to the assimilation of IR water vapour (WV) radiances, Hutt et al. (2020) assimilated clear-sky WV radiances in COSMO-KENDA, while for cloudy (all-sky) IR WV radiances, Harnisch et al. (2016) introduced a model for observation errors, and Otkin and Potthast (2019) introduced a conditional nonlinear bias correction. Even though these are important steps towards the assimilation all-sky IR radiances in the operational setting of KENDA this topic will require more efforts.

The second class of observations that appear particularly relevant are observations describing the planetary boundary layer (PBL), down to the surface level. These include ground-based remote sensing devices such as microwave radiometers (MWR:for low-resolution temperature and humidity profiles), wind lidars, Raman lidars (temperature and humidity profiles), DIAL lidars (humidity) but also direct measurements for drones and towers. Some of these devices are still rare and experimental, while others are increasingly available. Leuenberger et al. (2020) have shown that high-frequent profile observations from Raman lidars and drones can improve analyses of the pre-convective boundary layer and the stable boundary layer favourable for fog and forecasts for cloudiness and precipitation up to +9h into forecast time. There are national projects dealing with several of these data types which need to be coordinated and supported as part of the further KENDA development. For the above-mentioned screen-level observations (2-m temperature, 2-m humidity), a novel approach for station-dependent bias correction has been devised which can also be deployed for observation error specification. It has great promise to be able to deal with networks with very variable quality. Therefore, it offers also a potential to obtain a positive impact from data from other observing networks than operated by national weather services, e.g. roadside sensors. These methods may also be further refined with AI and machine learning algorithms.

Another future data source considered highly important due to its vast amount and high spatial and temporal resolution is the MTG IRS (Meteosat Third Generation Infrared Sounder) which is scheduled to be operationally available in 2024. Preparations for the use of these data should possibly start soon.

Finally, some analysis tools related to the lower boundary conditions need to be consolidated and refined, including the soil moisture analysis using soil moisture data. An important step will be their port to resp. re-write in the framework of the DACE code which will make their maintenance and further development (including upgrades to more advanced analysis methods) much more efficient. This refers to the sea surface temperature (SST) analysis, the snow analysis, the soil moisture analysis and the related 2-m temperature analysis. Depending on the available resources at the later stages of the project, more efforts should go into a coupled data assimilation of the soil (moisture and possibly temperature) and atmosphere.

3. Actions proposed

1. Algorithmic developments:Maintain and further refine the KENDA-LETKF; develop EnVar for operational purposes for ICON-LAM and compare with LETKF; further develop, refine, and test 4D-EnVar and PF; provide CEnVar and the tools for 3DVar as a means to carry out data assimilation for ICON-LAM without the need to run an ensemble (data assimilation) at the convective scale.
2. Extend the use of observations in the KENDA system, in particular those related to the PBL, clouds and precipitation. Also prepare the use of MTG IRS data.
3. Consolidate/redevelop analyses related to the surface:SST analysis, soil moisture analysis. The coordination related to and further development of the snow analysis will take place in a dedicated priority project, outside KENDAscope.

Even though some of the development work (outside DWD) in actions 2) and 3) may be carried out still with the COSMO model in the first years of the project, KENDAscope will be clearly focussing on the further development of KENDA for ICON-LAM.

4. Risks

KENDAscope does not consist of one main single task or aim which can be fulfilled or not. Therefore, the risk of an overall failure is low. Instead, there is a variety of individual tasks, each one dealing with its own scientific issues (many of which are mentioned above or in the respective tasks) and risks. For instance, there is a general risk that the gain in forecast quality from the use of each of the observation types addressed in KENDAscope judged on a larger scale may be difficult to prove and does not meet the expectations. As mentioned above there is e.g. also a risk that EnVar does not perform better than LETKF, in particular if observations related to cloud and precipitation, such as radar reflectivity or VIS radiances, are used. It is noted that even though operational applicability of the related development is the aim (deliverable) in many tasks, this is not always the case. For some of the topics, the work will very likely continue after KENDAscope in order to achieve further improvements.

The human resources appear to be secured in the earlier phases of the project for almost all actions (tasks). However, this is not always the case for the later stages of the 5-year period. Even though the past experience with the former KENDA-related 5-year projects makes us confident to be able to (re-)allocate the envisaged or required resources, this may not always be the case, particularly in case of severe financial cut-downs e.g. as a consequence of the pandemic. For tasks relying on national projects and hence on non-permanent positions, there is always a risk that the scientists leave the project early and cannot be replaced (adequately). Furthermore, unexpected urgent matters with highest priority (e.g. maternity leaves, operational duties, etc.) may detract planned resources and lead to delays (Task 1 may be exposed most to this risk). From today’s point of view, the largest risks for failing to make available the required FTE’s that are not yet allocated appear to be for adapting the direct assimilation of radar reflectivity to EnVar, for the particle filter, and for the soil moisture analysis using satellite soil moisture data.

5. Links with other projects or activities

COSMO WG7/PP PROPHECY:

• On the one hand, there is a direct dependence of WG7 aspects on KENDA:Most convective-scale EPS in the COSMO consortium rely on KENDA-LETKF initial conditions. Analysis ensemble perturbations may be adjusted in the final analysis step in order to provide spread and structures more optimal for (a realistic growth rate during) the EPS short-range forecast (see Task 2 in PP PROPHECY).

• On the other hand, KENDAscope may benefit from WG7 developments:stochastic perturbation as well as parameter perturbation approaches are further developed and evaluated in WG7/PP PROPHECY which may be useful potentially also during the KENDA assimilation cycle. Stochastic perturbations may utilize the stochastic pattern generator (SPG) developed in the former KENDA-O project.

  Note that the work on further refinement of the SPG and its port to ICON-LAM has been moved from the KENDA-related project in WG1 to the EPS-related project PROPHECY in WG7. This is because the SPG has been used so far only in the forecasting component in WG7, and there is no firm plan yet to apply it in the KENDA assimilation cycle.

COSMO WG3a:

Apart from the general importance to reduce model biases for the data assimilation, there are some strong direct links:

• Biases in first guess departure (observation minus first guess) statistics for observations with complex observational forward operators can be due both to errors in the NWP model and in the forward operator. Collaboration of WG1/KENDAscope with WG3a can be very important to help reducing these biases, potentially both in the model and also in forward operators addressed in KENDAscope (e.g. VIS radiances, radar reflectivity, …).
• Model biases can interact in various ways with the assimilation of (biased or unbiased) observations, lead e.g. to spin-up effects, and hamper the benefit from DA. Expertise from WG3a is needed to help understanding such effects.

COSMO WG3b:

A separate priority project on snow analysis will be set up soon by WG3b together with WG1, see further comments in KENDAscope Task 3.

COSMO WG5:

COSMO WG5:The observation-based verification against single (non-gridded) observations mostly relies on the MEC (Model Equivalent Calculator) which has been developed and is maintained in KENDA priority projects (under the umbrella of the task on the further development of the LETKF). Requests from WG5 e.g. to monitor additional observation types will make it necessary to implement additional code types and observation operators in the KENDA code.

SINFONY (Seamless Integrated ForecastiNg sYstem):

This is a large project at DWD for the development of a coupled probabilistic system of precipitation nowcasting and very short-range NWP. Some FTE’s in KENDAscope are funded in the context SINFONY; topics include more realistic model physics (e.g. 2-moment microphysics), a stronger focus on the first few hours of the forecasts and therefore on use of high-resolution data, assimilation of (nowcast) objects, etc.

ICamCloudOps:

Joint project of DWD and the private company Reuniwatt; this includes 2 positions (Q2/2020-Q2/2024) at DWD on intelligent observation operators for cloud information from conventional and infrared cameras, using artificial intelligence/machine learning (link to KENDAscope Task 2.3).

SPP PROM (Polarimetric Radar Observations meet atmospheric Modelling) programme funded by DFG:

This includes a 3-year project on the development of forward operators for polarimetric parameters and inclusion in EMVORADO (which is used in the KENDA system for the assimilation of radar reflectivity and radial velocity) (link to KENDAscope Task 2.1).

RealPEP (Near Realtime Quantitative Precipitation Estimation and Prediction) ‘Research Group’ funded by DFG:

This includes 3-year sub-projects on the assimilation of radar polarimetric information and observation-based nowcasted fields (link (mainly) to KENDAscope Task 2.1).

Hans-Ertel Centre (HErZ)/LMU University of Munich:

• Further refinement, optimization, and use of observation operator for VIS satellite radiances (link to KENDAscope Task 2.3)
• Studies on assimilation of radar data, mainly in an idealised setup (link to KENDAscope Task 2.1)
• Research on algorithmic aspects, e.g. sampling error correction, covariance inflation, localisation, non-linear effects, parameter estimation, etc. (link mainly to KENDAscope Task 1.1)

IAFE GPU (4-year project at DWD starting Q1/2021):

The main aim is to explore and understand which parts of the DACE data assimilation code are feasible to be ported to GPU and how. This is in order to prepare the operational usability of DACE on new computer architectures and therefore very important for the future applicability of the whole KENDA system and developments. However, it is not the task of this project to perform the whole porting. For this, more resources will likely be required, and if other COSMO members will plan to contribute to this, a dedicated PP outside KENDAscope may possibly be set up.

Other

A proposal for a 4-year project starting in 2021 with realistic chances to be funded has been submitted by DWD on the exploration of artificial intelligence and neural networks for the estimation and specification of observation errors.

6. Description of individual tasks

Task 1. Algorithmic developments

Core developments related to the atmospheric data assimilation scheme, using the standard conventional or the reference (operational) set of observations. Besides the main analysis algorithm itself, these developments also relate to all kinds of components necessary for operational data assimilation, including quality control, bias correction, specification of observation errors, etc. (unless they are developed for a particular observing system).

Task 1.1: Refinements/extensions of reference KENDA

Refinement and further development of the reference KENDA system wherever it is found to be required and appropriate depending on results. The reference KENDA is currently based on LETKF, but could include EnVar or PF in the later stages of the project.

General issues continue to be the ensemble size, analysis update frequency, covariance inflation, localisation, specification of observation errors, etc.

Candidates for likely implementation and assessment are:

• parameterized nonlinear online bias correction (BC) and obs error variance specification, non-diagonal observation error covariance
• improved additive covariance inflation
• further development of diagnostic tools (e.g. FSO:Forecast Sensitivity of Observations) and their inclusion in the experimental suites

Some further ideas include:

• perturbed parameters
• multi-step analysis with variable localisation for multi-scale analysis and/or with simple localization in the space of the control variables (the motivation for this might be reduced if a climatological B-matrix (where humidity errors are currently uncorrelated to temperature/wind errors) can be included in EnVar);
• sampling error correction by Necker et al. (2020) (an appealing idea in context of EnSRT filter but not yet clear whether/how to implement in LETKF); model space localisation in EnKF (see Lei et al., 2018)
• approaches designed to reduce position or phase errors, e.g. using pseudo observations derived from warped fields that account for position errors, or use of ensemble members with lagged valid time
• Kalman smoother in order to use observations valid after analysis time
• blending techniques to combine information from the larger scales of the nesting model

The task will likely also include the implementation and assessment of machine learning techniques for different parts of the processing (potentially e.g. for quality control, observation error specification, bias correction, observation operator components).

(Work plan at CNMCA for 2021:improve use of observations in high-resolution KENDA analysis, looking at statistics of observations and analysis increments; test use of radar volumes and GPS ZTD/STD.)

Task 1.2: Variational DA (EnVar)

Development of variational data assimilation (Var) for the convective scale, namely:

• EnVar, with KENDA-LETKF providing the ensemble B-matrix (background error covariances); hybrid 3D-EnVar includes (a fraction of) a climatological B-matrix;
• CEnVar (Coarse-Ensemble Var), with ensemble B-matrix derived by interpolation of external, coarse-scale ensemble perturbations (typically from a global ensemble DA system), thus avoiding the need to run a convective-scale ensemble; hybrid 3D-CEnVar includes (a fraction of) a climatological B-matrix;
• 3DVar with climatological B-matrix only;
• 4D-EnVar; this will be developed for global DA first and, depending on the experience gained, then possibly adapted and applied to the regional (convective-scale) DA (KENDA).

Note that the variational scheme provides only a single analysis for a deterministic forecast (unless a whole ensemble of variational analyses is run). However, it is possible to apply ‘centering’ of the analysis ensemble around the variational deterministic analysis (this replaces the analysis ensemble mean by the deterministic analysis). Previous tests with this kind of centering for the global system where the deterministic analysis has a higher resolution than the ensemble have been promising. The benefit has yet to be shown for the convective scale and with deterministic and ensemble runs at the same resolution.

Major work packages for 3DVar/3D-EnVar include:

• code rewrite of observation operators for conventional obs (replacing all COSMO modules and COSMO data structures in DACE)
• TL (tangent linear) + Adjoint of obs operators; inclusion of EMVORADO in DACE
• regional climatological B-matrix

 

Task 1.3: Particle filter (PF)

To address non-Gaussianity, research is continued on the refinement and optimization of the PF to complement or replace the LETKF core for the computation of the transform matrix in the ensemble filter in DACE. This work will not only be done with ICON-LAM but to a rather large extent with toy models and with the global ICON model.

Task 2. Observations (from surface to clouds)

Implementation and optimization of the use of observation types other than the original set of conventional observations. Of special interest is high-resolution information in the planetary boundary layer (PBL, e.g. on low-level convergence), but also on cloud and precipitation. In general, the deliverables are forecast improvements as a result of the use of these observations.

In the advent of corresponding projects, additional sub-tasks on further observation types may be added, e.g. on AMSU (ATMS, IASI,…) radiances from polar orbiting satellites (interest of ARPAE after 2022).

Task 2.1: Radar data

Refinement of the use of these data. Some (potential) issues, mostly related to radar reflectivity Z, are:

• modelling of observation errors (by taking into account the first guess departure), potentially bias correction
• covariance inflation method(s) (e.g. by introducing artificial Z to RH correlations in the LETKF step)
• tuning of LHN and assimilation of reflectivity for 2-moment microphysics
• adaptation to VAR (incl. development of TL + adjoint observation operators; implementing and testing of extended control vector, etc.)
• extension of observation coverage (more countries)
• ARPAE:operationalization of reflectivity (2020) and of radial winds (2021); further topics will be discussed later on, probably on a yearly basis, and may be related to satellite data

Task 2.2: Ground-based GNSS data (STD + ZTD:slant + zenith total delay)

Operationalisation and refinement of the use of these data. (Potential) issues are:

• (improved) station and elevation dependent online bias correction, and specification of observation errors
• adaptation to VAR (requires tangent linear and adjoint observation operators)
• adaptation to changes in the observation input (e.g. multi-GNSS real-time data, depending on availability) incl. testing and maintenance
• improved code efficiency (for NEC; 2-D instead of 3-D ray tracer by omitting perpendicular horizontal direction) and code clean-up

Task 2.3: SEVIRI + FCI (MTG IMAGER) all-sky (cloudy) IR + VIS radiances

Direct use of all-sky (cloudy) SEVIRI mainly in view of assimilating cloud information (horizontal distribution, cloud top height). Scheduled to start operations in autumn 2023, the first MTG imager FCI will thereon complement the MSG SEVIRI data.

Data from 3 types of channels are available (and at least the first 2 will be addressed):

1. Reflectance from visible channels (VIS), providing information during daytime mainly on cloud cover, in particular of low clouds not seen well by other channels; this sub-task will build on previous work done at DWD and HErZ at LMU Munich (e.g. Scheck et al., 2020).
2. Brightness temperature from infrared (IR) water vapour (WV) channels, providing information related to water vapour and/or cloud cover and cloud top height of high and mid-level clouds all day; this sub-task will extend the experimental use of clear-sky WV data developed in KENDA-O (Hutt et al., 2020) to an all-sky approach.
3. Brightness temperature from infrared (IR) window channels, providing information related to cloud cover and cloud top height of all cloud types all day.

Particular issues:bias correction, observation error specification, non-Gaussian distribution of first guess departures, non-linearity, localisation (for non-local observations), thinning, tailored covariance inflation, etc.; further potential approaches may include variable transformation (non-Gaussianity), smoothing and warping (non-linearity, displacement errors), etc; an important aspect is also the combination of the different channels.

A further issue is the adaptation to VAR, and this includes the need of the TL and adjoint of the forward operator. (Note that in VAR, the forward operator has to be formulated such that the model equivalent to any observation is computed solely from those prognostic model fields (called control vector) which are updated in the analysis. If the forward operator requires additional prognostic or diagnostic fields as input then these fields have to be treated as fixed values or as sink variables. Sink variables may vary during the variational procedure, but the information on the modified values of the sink variables is not conveyed to the analysis state.) For the satellite radiances, the forward operator consists of two parts:Firstly, the radiative transfer (RT) model (MFASIS/RTTOV) for which the TL and adjoint have already been developed. Secondly, the computation of the input fields of the RT model (e.g. cloud fraction) from the prognostic model variables. This computation should reflect the diagnostics e.g. of cloud cover as described in the NWP model and in particular its radiation parameterization as well as possible. In collaboration with WG3, this will be done by using the formulations of these parameterizations (or approximations thereof), or alternatively by using neural networks. Another pivotal question for the adaptation to VAR is whether to include hydrometeor mixing ratios in the control vector (for the minimization) or not.

Task 2.4: MTG IRS (Meteosat Third Generation Infrared Sounder)

Even though the ultimate goal might be the use of all-sky data, the main benefit from the use of these multi-spectral radiances on top of the cloud information from the imagers (Task 2.3) is expected from profile information on temperature, humidity, and stability. Therefore, the focus will lie on the use of clear-sky radiances. Later, this can be extended to radiances sensitive to regions above cloud or affected (slightly) by clouds, always with the main aim to derive information on temperature and humidity.

The IRS instrument on board of MTG-S1 is currently scheduled to start operations no sooner than autumn 2024 (with first test data likely available half a year before that). In order to prepare the use of these data, test data for IRS can be used for development:

• simulated data based on IFS profile/surface data + RTTOV, to prepare for formats, data coverage, to adapt RTTOV interface;
• data from IASI which is the most similar hyperspectral sounder;
• data from SEVIRI for studying certain aspects such as assimilation of cloudy WV channels, or use of data with very high spatio-temporal resolution

Research topics/issues include:

• emissivity/skin temperature retrieval and background errors (based on work already done for global DA)
• slanted radiative transfer (also for global DA)
• cloud detection over land (testing/tuning e.g. with IASI data); all-sky data assimilation
• bias correction (online; coupling to global data assimilation?)
• field over view size in first guess equivalent (supermodding)
• LETKF aspects:data density/thinning/superobbing (use SEVIRI input), localisation, …

Task 2.5: Screen-level observations

Use of local observations from surface stations. Primarily, this means use of 2-m temperature, 2-m humidity, and 10-m horizontal wind observations from Synop stations, which are considered potentially beneficial for describing the (pre-) convective environment (low-level convergence) and very low stratus.

In addition, this task may also include observations from other types of stations, e.g. roadside sensors (with temperature/humidity/wind at e.g. 2, 4, 5, 7, 10 m above the ground).

Particular issues: Station-dependent conditional nonlinear online bias correction (see Otkin and Potthast, 2019), specification of observation error and quality control, vertical localisation, observation operators for observations on levels other than 2 m resp. 10 m above the ground; TL + adjoint of observation operators for VAR.

Task 2.6: Boundary-layer profiling observations

Development of assimilation of a variety of data types which typically deliver profile information (i.e. data at several vertical levels) mainly in the planetary boundary layer. These include:

1. ground-based microwave radiometer (MWR) radiances, providing information on (a.o.) temperature and humidity with low vertical resolution; the focus is on the direct assimilation of the radiances rather than retrievals (temperature, humidity, IWV, liquid water).

   Deliverable: Implementation and assessment of assimilation of MWR direct radiances in the LETKF.

2. Doppler wind lidar (wind profiles)

   Deliverable: operational applicability.

3. Raman lidar (temperature + humidity profile)

   Deliverable: Development and test of observation operator (using mixing ratio as humidity variable), assimilation experiments

4. drones

   Deliverable: Assessing impact of assimilation of drone observations.

5. towers

   Deliverable: Technical implementation, monitoring, assimilation tests depending on monitoring results

Other observation types may be added when available, e.g. DIAL lidars (humidity profiles and potentially 3-D scans; some FTE e.g. for observation monitoring could be taken from Jasmin Vural (DFG/DWD) for this).

Task 3. Soil/surface

The main topics here are soil moisture analysis (SMA) or initialization (Tasks 3.1, 3.2), sea surface temperature (SST) analysis (including analysis of sea ice) (Task 3.3), and snow analysis (namely analysis of snow depth, cover and/or snow water equivalent).

However, the snow analysis is not part of this project. Instead, it is considered more suitable to be addressed in a Priority Project on its own because it is more closely related to (other tasks in) WG3b. Having a separate PP for this topic will facilitate the management of the work and communication between the staff involved.

Further sub-tasks may be added during the course of the PP if resources can be allocated for them. A candidate might be a soil temperature analysis, possibly including the use of satellite-derived land surface temperature data.

Task 3.1: Soil moisture analysis using satellite soil moisture data

Use of satellite-derived soil moisture products, generated by HSAF from MetOp scatterometer (ASCAT) data and possibly from L-band microwave imagers (SMOS, SMAP), in the LETKF. Assimilating such data with the same scheme (e.g. LETKF) as for the atmospheric variables enables the system for coupled DA:atmospheric observations (e.g. of 2-m temperature) may be used to influence also the soil, while the soil moisture data may influence the atmospheric analysis as well. Technically, one or several additional analysis levels for the soil have to be added to the control vector of the LETKF.

Issues: situation-dependent vertical localization between soil and atmosphere depending on realism of ensemble background error correlations across surface, observation error specification, bias correction, etc.

The work plan for 2021 consists of:

• technical work: merge implementation into official DACE code, generation of feedback files in MEC
• perform real-time tests (comparison to operational "no-SMA" run)
• investigate further improvements (new soil moisture observations or other-yet to be decided; note that previous tests showed no or only marginal positive impact so that further improvements are likely needed in order to render a beneficial impact)

Task 3.2: Soil moisture analysis/initialization

Currently, a stand-alone variational ‘soil moisture analysis’ (SMA) scheme using daytime 2-m temperature observations (interpolated to the model grid) as input is applied to the global ICON / ICON-EU nest. The soil moisture of the convective-scale configuration at DWD, ICON-D2, is then nudged towards the (interpolated) ICON-EU soil moisture, which has the same soil model (TERRA).

In this task, it is planned to re-write the SMA in the DACE framework. There are several possibilities to derive the sensitivity of daytime 2-m temperature to soil moisture (in the morning):a) using the ensemble covariances (LETKF approach), b) by means of additional forecasts with perturbed initial soil moisture (approach formerly adopted for COSMO-EU), or c) by parameterization (based on latent heat fluxes, approach used for the global ICON/ICON-EU). It has yet to be decided which approach(es) should be implemented (and which one first).

The primary goal of the task is to implement a first approach for testing and to compare it to the current nudging towards the interpolated ICON-EU soil moisture. Note that this implies the need to re-write also the 2-m temperature gridded analysis based on Synop observations in the DACE framework; a variational approach is envisaged.

Later on, the task might be extended in one or several ways:further refinement of the implemented scheme, implementation of another approach, use of further observations (e.g. satellite data related to soil moisture), etc.

Note that as long as only atmospheric observations are used in this scheme, one may prefer to speak of a soil moisture initialization rather than analysis.

Task 3.3: Sea surface temperature (SST) analysis

Currently, a stand-alone SST analysis is run for COSMO-D2/ICON-D2. In-situ data are used in a simple Cressman-type successive correction method to correct the background field which is based on the Ostia analysis from the Met Office.

In this task, it is planned to completely re-write the SST analysis in the DACE framework and replace the old scheme by a 2-d variational scheme. (This implies to implement a 2-dimensional version of the (currently 3-dimensional) B-matrix.)

7. References

8. Appendix: Task table

Planned resources in the COSMO-year 2021:

Resources in 2021 meet the requirements.

Task	Contributing scientist(s)	FTE total	FTE 2021	Start	Deliverables	Date of delivery	Preceding tasks
1.1	Reich, Stephan, Potthast, Anlauf, Schraff (DWD) Leuenberger (MCH) Torrisi, Marcucci, Cardinali (CNMCA)	7.5	1.45 1.1 0.3	09/20	Refinements/extensions of reference KENDA	08/25	-
1.2	Bauernschubert, Holborn, Burba, Schraff, Potthast (DWD)	7	1.8	09/20	Variational DA (EnVar):(C)EnVar 3DVar 4D-EnVar	06/21 12/21 08/25	-
1.3	Schenk (DWD)	4	0.5	09/20	Particle Filter	08/25	-
2.1	Khosravian, Stephan, Blahak (DWD) Poli, Gastaldo (ARPAE)	7.2	0.7 1.4	09/20	Radar reflectivity Z + radial velocity Vr Vr at ARPAE	08/25 12/21	-
2.2	Bender (DWD)	1	0.2	11/20	GNSS ZTD + STD	12/22 c	-
2.3	Bach, Schomburg, Deppisch (DWD)	1.5	1.3	09/20	SEVIRI/FCI cloudy radiances VIS IR WV	12/21 c 12/22 c	-
2.4	N.N. (DWD/IAFE)	4	0.5	01/21	MTG IRS	08/25	-
2.5	Sgoff (DWD)	1.5	0.3	09/20	Screen-level obs RH2m uv10m, T2m, etc	03/21 c 08/25	-
2.6	Leuenberger, Merker, Monhart, N.N1. N.N2 (MCH) Vural, Schraff (DWD)	6.5	2.2 0.85	09/20	PBL profiling obs MWR wind lidar Raman lidar	06/23 09/22 09/22	-
3.1	Cardinali (CNMCA)	1	0.1	09/20	Soil moisture analysis using satellite soil moisture data real-time tests	08/25 08/21	-
3.2	Geppert, Lange (DWD)	0.6	0	01/22	Soil moisture analysis/initialization	12/22	-
3.3	Geppert (DWD)	0.2	0.2	01/21	SST analysis	12/21	-