Priority Project "AWARE": Appraisal of "Challenging WeAther" FoREcasts Endorsed by HIWeather

Last updated: 6 Jul 2022
get the pdf document

Project leader: Flora Gofa (WG5) and Anastasia Bundel (WG4)

1. Introduction 2. Motivation and goals 3. Status and available expertise within COSMO 4. Links to other projects 5. Description of individual tasks 6. Risks 7. Project Participants 8. References 9. FTEs

Project resources

Project News

• PP-AWARE has been endorsed by WMO HiWeather project.
• During the IVMW 2020, PP-AWARE was presented at the joint HIW-User Value Chain Session that was hosted by WMO. 
• In Section 5 (Description of individual tasks) reports are added for any concluded work (reports on Tasks 1.1, 2.1, 2.2, 2.3, 3.1, 3,2, 3.4, 3.5, 3.6, 4.1, 4.2, 4.3 are now available).
• During ICCARUS 2021, a session on PP-AWARE took place.

1. Introduction

The importance of accurate forecasting of challenging weather occurrences is obvious. With the term challenging weather (CW) or high impact weather (HIW), we consider the events the local society is not routinely accustomed to experiencing. Such events could be extreme in amplitude (intense winds, or heavy convective precipitation), rare (lie in a tail of climatological distribution for a particular location) or high impact by being prolonged ‘regimes’ (droughts, heat-waves or cold-spells), while others even if not very rare can be considered challenging if society is particularly vulnerable to them (e.g. impact of fog on transportation). In theory, a weather event could be high-impact when it is inherently less predictable and society does not have sufficient forewarning to take mitigating action.

Despite significant progress in short-range forecasting, HIW continue to cause most part of damage to the economy and society, up to losses of human lives. WMO initiated a project dedicated to HIW research (WMO HIW implementation plan). The overall objective of the High Impact Weather (HIWeather) project is to: “Promote cooperative international research to achieve a dramatic increase in resilience to high impact weather, worldwide, through improving forecasts for timescales of minutes to two weeks and enhancing their communication and utility in social, economic and environmental applications”. One component of the project is the verification and the research is focused on approaches with relevance to hazard predictions. PP AWARE is intended to follow closely the advances of the WMO HIW project. In COSMO consortium, there have been several studies partially related to CW aspects. However, up to now there have been no project explicitly focusing on evaluation and development of HIW forecasts.

Forecast methods and verification are important aspects of any CW consideration. While traditional verification methods have limited usefulness in this context, many of the newer diagnostic approaches may provide useful information to aid understanding of errors in model processes for such weather regimes. Verification of multi-scale prediction of CW has much in common with routine verification performed at most national meteorological centers. It mainly concerns surface variables such as precipitation, wind, temperature, etc., using both pointwise and spatial approaches to meet the needs of a variety of users. On the other hand, CW phenomena as fog or lightning are usually not directly forecasted by NWP models, and thus appropriate post-processing methods are applied for their prediction.

Key forecast quality and verification aspects to consider in this project include, therefore:

• How well high-impact weather is represented in the observations, including biases and random errors, and their sensitivity to observation density.
• How well high-impact weather is represented in models, including systematic and stochastic errors, and their sensitivity to model resolution.
• How well high-impact weather is represented in postprocessing.

2. Motivation and goals

The increased demand to provide accurate forecasts of extreme weather leads to the question to how objectively to evaluate forecasts of extreme weather. The main weather phenomena of interest are: thunderstorms (heavy precipitation, lightning), severe wind (and wind gusts), min-max temperature (persistence), visibility (fog), and other extreme phenomena like tornadoes, dust devils, clear-air turbulence (CAT), etc.

As the resolution of state-of-the-art NWP models is growing, there is more detailed and precise information on the variables necessary for calculating, for example, the electrical properties of the atmosphere as dependent on temperature, humidity, wind, ice, water content of particles, etc. The physical and microphysical processes leading to local CW of convective nature are better reproduced in modern NWP models. This should improve the direct forecasting of CW of convective nature, such as thunderstorms, hail, squalls, and showers. However, the models cannot satisfy as yet all the needs for CW predictions. Thus, different postprocessing methods are required. It is important to compare the forecasts based on direct model output (DMO) and postprocessing, where it is feasible Standard verification approaches for medium range NWP have limited usefulness for very high resolution (mesoscale and convective scale) forecasts. Several new verification methods have been proposed for evaluating the spatial structures simulated by high resolution models and this remains an active area of research. While most of these spatial methods measure forecast quality, some of them (e.g., variograms) address the realism of the forecast, which may be of particular interest to modelers. Spatial verification approaches are now starting to be applied to high resolution ensemble forecasts, but much remains to be done to understand what can be learned from these approaches, both in terms of quantifying ensemble performance, and in calibrating and postprocessing ensembles to improve forecast quality and utility. The utility of spatial verification for evaluating hazard impact forecasts (e.g., flood inundation, fire spread, blizzard extent and intensity, pollution cloud) needs to be explored, especially since graphical advice and warnings are becoming more common

The goal of the proposed PP is to try out a number of forecast methods and evaluation approaches that are linked to high impact weather (not necessarily considered extreme to all users) and to provide COSMO Community with an overview and recommendations as to how CW/HIW situations should be handled. Based on the experience of the COSMO countries and the study of the state-of-the-art methodologies in the world, a series of observed and predicted CW/HIW cases will be investigated and evaluated, and a number of most successful and promising CW/HIW methods will be identified and further developed as needed.

3. Status and available expertise within COSMO

Most COSMO NMSs have gathered experience and expertise in the handling of high impact weather forecasts which will be shared also through the Task work of PPAWARE. Below, some examples are given indicatively

In MeteoSwiss, a number of postprocessing techniques are in practice: Kalman filter, MOSMIX, on point-forecast at stations. The production chain includes man-machine interactions, mixing supervised values and model outputs. Recently, a PostprocVeri project was initiated aimed at postprocessing methods yielding spatial, probabilistic, multivariate, and seamless forecasts and also to refine the verification analysis at the forecast service of MeteoSwiss. The main elements of PostprocVeri are:

• Probabilistic postprocessing: well in line with NWP developments at MeteoSwiss and international developments in the field of postprocessing → Ensemble postprocessing routines, aiming at delivering calibrated ensemble predictions
• Spatial output: given the increasing importance of local forecast information, the postprocessing approaches aim at delivering output for any surface location of interest in Switzerland.
• Start with basic meteorological variables: introduce postprocessing for four basic meteorological variables (temperature, precipitation, wind, and cloud cover), build up knowhow to apply to derived variables later on
• Limit NWP data sources to COSMO and IFS ensembles (models operationally used in today’s forecast production), but ensure applicability to other NWP models

At IMGW-PIB (Polish National Weather Service) CW/HIW was the subject of interest and cooperation with road and air maintenance services. Starting from 2004, collaboration with roads maintenance services resulted in procedures of forecast of fog (or visibility range, VR), road icing (esp. occurrence of “black ice”) and intensive precipitation of snow, also in coincidence with strong wind that can blow snow on the roads. As far as aviation services are concerned, IMGW-PIB cooperates with PAZP (or NAA, National Aviation Agency). This cooperation includes – among others – forecasts of wind shear, runway visibility range (RVR), thunderstorms etc. The information delivered to PAZP contains not only absolute values, but also the probabilities (Bayesian approach) of occurrence of certain phenomena.

In RHM, many forecasters issue CW warnings based on different indices calculated from NWP production. For example, the thunderstorm probability is calculated using different instability indices, such as Whiting index (Richter H., 2004) taking into account the differences between temperature, dew point temperature, and dew point deficit at different isobaric heights or Peskov index estimating instability in the middle of convective cloud, humidity, vertical wind shear, and near-surface convergence (Guidelines, 1965). It should be noted that there exist about 26 such indices in the world. In work (Gubenko, 2016), a comparison of the accuracy of thunderstorm forecasts using these indices is given, as well as the quality estimates of different types of measurements.

4. Links to other projects

The PP AWARE is linked to the WMO Severe Weather Forecasting Demonstration Project and HIWeather Project. The mission of the latter is to promote cooperative international research to achieve a dramatic increase in resilience to high impact weather, worldwide, through improving forecasts for timescales of minutes to two weeks and enhancing their communication and utility in social, economic and environmental applications. There is a link to SURF (Study of Urban Rainfall and fog/haze) (experiment to study urban pollution and extreme precipitation in Beijing) through the collaboration of RHM. The PP AWARE is linked to the EUMETNET SRNWP EPS project, whose phase II ended in December 2018. The main aim of this project is to contribute to build very high-resolution ensemble systems in Europe, resolving the convection-permitting scale phenomena. The PP AWARE will benefit from the mutual exchange of experience with the EUMETNET SRNWP EPS project, in particular in what concerns the application task of the latter, where new products and methodologies for calibration of LAM ensembles for extremes and for probabilistic prediction of thunderstorms and fog are developed (calibration of ensemble outputs, mainly for extreme wind, precipitation and temperature by AEMET; products for probabilistic prediction of thunderstorms and fog, focus being on selected phenomena by CNMCA). Finally, in terms of verification software availability, PP AWARE is linked to PP CARMA that has the goal to provide COSMO community with the MEC-Rfdbk software developed by DWD, to perform traditional point verification both for the surface and the upper air using conventional methods, which arise from operational and research activities

5. Description of individual tasks

Task 0. Administrative Tasks

Due to the distributed nature of the project participation team, administrative activities will be included in this task, in order to maintain a good collaboration and information flow between all participants (regular web conferences, workshops, etc.). A mailing list for the project will be used in order to support communication and information exchange between project participants.

Task 1. Challenges in observing CW/HIW (WG5 and WG4 related)

This task is basic for the understanding of the nature of phenomena studied within the project. The task will consider which observations are necessary to verify HIW forecasts, as well as issues related to observation sparseness, quality, and thresholds. Furthermore, through the Task, work effort will be given to identify the dependence of HIW prediction improvements on dense observations, to identify observation requirements for monitoring the selected hazards and/or for assessing forecast accuracy, and quantify the role of observation uncertainty. The study on observation uncertainty was initiated within the INSPECT project (COSMO technical report 37, chapters 4.1.3 and chapter 5).

Task 1.1 Overview of CW/HIW observational data sources characteristics

Review of available sources, estimation methodologies, and associated error.

Task 1.2. Approaches to introduce observation uncertainty

Analysis of observation uncertainty contribution to verification scores focused on HIW forecasts. Quantification of observation uncertainty is important for forecasting all the hydrometeorological variables. For HIW events, which are often the rare ones, it is of extreme importance. Accounting for observation uncertainty can change verification results and it becomes even more important at present when the forecast quality tends to approach the level of observational errors.

Contributors to task 1:

• Task 1.1: Andrzej Mazur, FTE 0.1, Start 09.2019 – End 02.2020, Anastasia Bundel, FTE 0.05, Start 10.2019 – End 02.2020, Chiara Marsigli, FTE 0.1, Start 09.2019 – End 02.2020.
• Task 1.2: Anastasia Bundel, FTE 0.05, Start 10.2019 – End 04.2020
• Total Resources Task1: 0.3FTEs

Task 2: Overview of appropriate verification measures for HIW (WG5 related)

The verification of many HIW events requires metrics that remain useful for rare events. Their main characteristics are that must be less dependent on the base rate (climatology of the event), the dependency on spatial and temporal scales and sampling of observational data should be minimized and the dependency on the verification grid should be minimized. Hits and false alarms should be taken into account.

As no single score exists that addresses all these properties, the response of commonly used scores on HIW for these properties will be studied through this Task on selected test cases. Scores behavior for the evaluation of both the deterministic and ensemble forecasts of HIW (SEDI, EDS, EDI, SEEPS, CRPS) will be addressed. Global scores tuned over extreme events will be also tested.

Proper scoring rules for extremes, downscaling of precipitation extremes will be studied.

Task 2.1. Survey for assessment of proper verification of phenomena – continuous vs. discrete verification (occurrence vs. specific values).

This task should provide arguments for choosing a particular method that should be used to verify HIW phenomena. Continuous phenomena are easier to describe, and a mathematical apparatus for this purpose is huge and wide and easily accessible, however, it may be tempted to check whether other methods could be used in the verification (see task 3). One should also keep in mind that usually very sophisticated and complicated methods are not particularly useful (law of parsinomy, Ockham’s razor).

Task 2.2 Role of SEEPS and EDI-SEDI for the evaluation of extreme precipitation forecasts.

Combining data from a larger number of stations during the evaluation process of NWP forecasts can produce false skill if climatologically diverse regions are combined. In particular, when interest is driven by the presence and implications of heavy precipitation events, one must aggregate regions of similar climatology that will be reflected in the precipitation thresholds that constitute an ‘extreme’ event in the specific area. Consequently, it is important for HIW events to analyse the relative strengths and weaknesses of commonly used statistical measures but also to highlight the importance of threshold choice especially during the aggregation of results of stations with different climatological characteristics.

Stable Equitable Error in the Probability Space (SEEPS) uses the categories ‘dry’, ‘light precipitation’ and ‘heavy precipitation’ based on the climatological cumulative precipitation distribution (observation derived -30 year database). In addition, the supplementary application of categorical scores like the Symmetric External Dependency Index (SEDI) suitable for extreme events but adjusted to the climatological characteristics (model derived) of a specific region, can further contribute to the objectives of an operational meteorological service, including long term monitoring of model performance and assistance in the decision-making process when forecasting rare events.

Task 2.3 Extreme Value Theory (EVT) approach- Fitting precipitation object characteristics to different distributions

EVT is a branch of statistics which studies the properties of extreme values, and enables them to be fit with theoretical distributions (or probability models). The common approach to form a two-way contingency table for the joint distribution of forecasts and observations is of limited utility, because of sparse entries for extreme classes. This joint distribution could be represented by some parametric form, representing the behavior in the extreme tails. The adapted theory of extreme values is lately adopted and provide a family of parametric distributions with flexible tail behavior. Making use of the calibration-refinement factorization, the focus is on modeling the extreme tails of the conditional distribution of the weather observation given a forecast.

Contributors to task 2:

• Task 2.1: IMGW-PIB Andrzej Mazur, FTE 0.2, Joanna Linkowska, FTE 0.15, Start 10.2019 – End 05.2020.
• Task 2.2: HNMS: Flora Gofa, FTE 0.10, Dimitra Boucouvala, FTE 0.15, Start 12.2019 – End 08.2020.
• Task 2.3: RHM: Anatoly Muraviev, FTE 0.3, Start 09.2019 – End 08.2020.
• Total Resources Task1: 0.9 FTEs

Task 3: Verification applications (with a focus on spatial methods) to HIW (WG5 and WG7 related)

This task will make use of the analysis and outcomes of Task 2. It is also connected with and continued from PP-INSPECT (../../../model/documentation/techReports/cosmo/docs/techReport37.pdf) and MesoVICT project on the spatial methods (https://ral.ucar.edu/projects/icp/).

Task 3.1 Verification of forecasts of intense convective phenomena (thunderstorms w. lightning)

This subtask should supply with the extended knowledge of quality of HIW phenomena’ forecasts, in connection with selected spatial verification methods (this study should supplement task 2, subtask 2.1 – see above – by the application of spatial methods) and from the point of view of parameterization/forecasts methods (task 4, see below).

Task 3.2 Lightning potential index (LPI) in mountain regions

Strong convection, storms are potentially phenomena that can cause significant damages, which could be prevented by accurate weather forecasts. The outcome of this subtask should be the estimation of quality of the local LPI (Lightning potential index) forecasts received by MeteoSwiss end users. LPI will be integrated into the COSMO operational chain at MeteoSwiss and will be used to supply several products (See Task 4.3.1.)

Task 3.3. CRA (Contiguous rain area) and FSS analysis on intense precipitation

This task relates to two of the project key aspects: a. how well high-impact weather is represented in models, including systematic and stochastic errors, and their sensitivity to model resolution and b. the predictability, current predictive skill, and the user’s interpretation of forecast value in high-impact weather situations. This task continues the corresponding INSPECT task, but with a focus on intense precipitation events. In addition to precipitation data, the potential of using reflectivity will be explored.

Task 3.4 DIST methodology tuned on high-threshold events for flash floods forecast evaluation

The main goal of the methodology, an evolution of the DIST methodology (see Marsigli, C., Montani, A., and Paccagnella, T.: A spatial verification method applied to the evaluation of high-resolution ensemble forecasts, Meteorol. Appl., 15, 125–143, 2008), is to verify the products developed to estimate the QPF over catchment areas used operationally for issuing Civil Protection alerts. Verification results can be used directly to interpret how to use the forecast system and to decide in which situations one system is better than another.

The proposed methodology has been developed as a spatial method for the verification of heavy precipitation issued at high resolution. This task proposed to explore (or highlight) its suitability for verification of HIW.

In fact, it permits the use of a high-resolution rain-gauges network, but gridded observations, such as radar precipitation analysis, can be used as well. The main advantage of this approach is that no precipitation analysis is required and information about localized maxima of precipitation can be taken into account, as well as the variability of the precipitation field inside the area of interest.

Similarly all the grid points that belongs to the selected area are considered, in this way the ability of the model in reproducing high precipitation events, even if with some possible positioning errors, is evaluated.

Task 3.5. LPI verification and correlation of convective events with microphysical and thermodynamical indices.

The distribution analysis of a number of convective events in both space and time will allow lightning/thunderstorm regimes to be determined. On the other hand, lightning potential index (LPI), is a measure of the potential for charge generation and separation that leads to lightning flashes in convective thunderstorms and can be calculated from COSMO model. While the connection between cloud microphysics and lightning seems apparent, the common indices used for forecasting thunderstorms and the potential for lightning usually rely on stability and thermodynamical indices (e.g. CAPE). An effort will be given to correlate LPI with observed lightning. In this way, it will be evaluated if for Greek territory LPI can be useful parameter for predicting lightning as well as a tool for improving weather forecasting of convective storms and heavy rainfall. Statistical evaluation of precipitation forecasts with SAL spatial method on selected intense precipitation events will be also performed by comparing gridded observation datasets (satellite and radar precipitation estimates), as well as lightning data with model outputs (convective precipitation, LPI, CAPE).

Task 3.6. Work on the comparative verification of NWC and NWP results using spatial verification methods as part of the SINFONY project at DWD.

Germany is exposed to various kinds of high impact weather phenomena. Strong impacts are expected from convective events during summer time which happen to be especially hard to predict. The project SINFONY at DWD has the goal to improve forecasts of such events in the short range up to 12h. On the observation side, high-resolution reflectivities from the German radar network are used. Such reflectivities give the instantaneous state of the current precipitation situation, where HIW is correlated with high reflectivities. On the model side (COSMO-D2-EPS and later ICON-LAM) reflectivities are derived from the forward operator EMVORADO. One goal will be to compare the 1-moment with the 2-moment convection scheme. As the latter is able to produce higher reflectitivies, it is expected to better capture extreme events. Another goal will be to compare pixel and object-based reflectivity nowcasting for their potential to predict development of HIW. For the nowcasting, it is planned to perform life-cycle analyses and to use curved tracks, ensemble-nowcasting, etc. In the end, verification hopefully has contributed to the development of a combined product for the seamless prediction of the evolution of thunderstorms in summer. Hence, there will be a rich data base and tools that should be useful for the objectives of AWARE.

Contributors Task 3

• Subtask 3.1: IMGW-PIB: Joanna Linkowska, FTE 0.3, Andrzej Mazur, FTE 0.2 Start 09.2019 – End 08.2021
• Subtask 3.2: MCH: Daniel Cattani, FTE 0.1 Start 09.2019 – End 08.2021
• Subtask 3.3: RHM: Anastasia Bundel, FTE 0.3, Start 09.2019 – End 08.2020
• Subtask 3.4: ARPAE: Maria Stefania Tesini, FTE 0.1 Start 09.2019 – End 09.2020
• Subtask 3.5: HNMS: Dimitra Boucouvala, FTE 0.15, Flora Gofa, FTE0.15, Start 10.2019 – End 08.2020
• Subtask 3.6: DWD: Michael Hoff, FTE 0.1 Start 09.2019 – End 02.2021
• Total Resources Task3: 1.4FTEs

Task 4. Overview of forecast methods, representation and user-oriented products linked to HIW (WG4 related)

HIW forecasts are not always a direct model output (DMO) product. In particular, the forecasts of the following meteorological variables are often a result of postprocessing:

• Fogs and visibility
• Thunderstorms (heavy precipitation, lightning)
• Other severe phenomena (tornadoes, super-cell storms, CAT)
• Wind hazard forecasting, wind metrics that relate to impacts.

There are many methods and indices to predict these variables. This task will summarize them. Moreover, if not-HIW forecasts begin to be issued based on direct model output (DMO) in some weather services, human analysis is almost necessarily required when there is a possibility of HIW phenomena occurrence. Thus, the role of forecaster is considerable in relation to this task. The task will provide an overview of direct model output and postprocessing methods: describe the state of the art in predicting HIW; identify processes that lead to high impacts; make recommendations for targeted work to address weaknesses in understanding and predicting them. This task also touches on the question of decision-making based on CW/HIW forecasts.

This task relates to the following key project aspects: what models can provide and what warnings need to be made for (lightning, hail, wind gusts, fog, etc.) and the user’s interpretation of forecast value in high-impact weather situations (observed and/or forecast).

The output of this task is improvement in providing COSMO users with forecast/warnings of the most important HIW events.

Task 4.1. Postprocessing vs. direct model output for HIW

This task relates to the following key project aspect: How well high-impact weather is represented in postprocessing and modeling. In order to be able to choose the best method to predict HIW, we need to understand the state-of-the art in this field, both in direct modelling and processed model data. This task will provide an overview of the forecast methods for some of the most important HIW parameters. This task will also provide a comparison of post-processing techniques for HIW forecasts vs. direct model output and human forecasts, where possible.

Task 4.2 Improving existing post-processing methods

Exploring new approaches. Machine learning. Use of multi-linear regression (MLR), adaptive (recursive) least squares (A-RLS) and/or artificial neural networking (ANN) techniques. The feasibility and the usefulness of any particular method may be increased. Existing methods would profit from use of archive data. Especially, post-processing or hind-casting (back-testing) should improve current forecasts’ quality and reliability. Advantages and disadvantages may be easily assessed on the basis of training methods such as Adaptive Least–Squares Regression (ALSR), Multi-Line Regression (MLR) or Artificial Neural Networking (ANN) using archival data, and then it can be suggested to the present.

Task 4.3 QPF evaluation approaches

The evaluation of the amount of precipitation over catchment areas is one of the most important use of the QPF at ARPAE for hydrological purpose and for the issuing of Civil Protection alert for possible floods. To meet the needs end-users, such as hydrologists or forecaster, we have developed some tools that provide mean, maximum and some other percentiles values of the precipitation field over the catchment areas of the Emilia-Romagna region. Exceeding predefined thresholds can give useful indications for situations of intense precipitation possibly leading to floods.

Task 4.4 Representing and communicating HIW forecast for decision making

This subtask will describe approaches to representing and communicating HIW forecasts/warnings for a wide range of users (the kind of users depends on the country/institution) with examples (maps, test messages, symbols). It will compare data transfer channels to communicate HIW forecasts: internet sites, sms lists, e-mail, radio, mobile apps vs. “common” transfers. It will also study behavioral aspects of warnings.

Given the increasing number of HIW situations and the request to mitigate their impact, it is necessary to identify the best methods (media, applications, etc.) and forms (maps, test messages, symbols, etc.) to communicate the meteorological information to authorities, the public and specific users, depending on their specific needs.

Contributors Task 4

• Subtask 4.1: RHM, E.Tatarinovich, FTE 0.3; A.Bundel, FTE 0.2, Start 09.2019 – End 08.2021
• Subtask 4.2: IMGW-PIB, Andrzej Mazur, FTE 0.15, Grzegorz Duniec, FTE 0.1, Start 09.2019 – End 08.2021
• Subtask 4.3: ARPAE-SIMC, Maria Stefania Tesini, FTE 0.1, Start 09.2019 – End 12.2020
• Subtask 4.4: RHM, Anastasia Bundel, FTE 0.1, Inna Rozinkina FTE 0.1, Start 01.2020 – End 08.2021; NMA, Tudor Balacescu, FTE 0.1, Start 06.2020 – End 09.2021
• Total Resources Task4: 1.35FTEs

6. Risks

Small number of extreme observed event database. Definition of extreme events depends on local climatology. Dataset with observations and model outputs need to include rare events (various single test cases or long time series).

7. Project Participants

• ARPAE-SIMC: Maria Stefania Tesini
• DWD: Chiara Marsigli, Michael Hoff
• HNMS: Flora Gofa, Dimitra Boucouvala
• IMGW-PIB: Andrzej Mazur, Joanna Linkowska, Grzegorz Duniec,
• MeteoSwiss: Daniel Cattani
• NMA: Tudor Balacescu
• RHM: Anastasia Bundel, Ekaterina Tatarinovich, Anatoly Muraviev

8. References