Continuous deterministic verification

This document is a general description of the usage of this tool and the interpreation of the shown scores.


Select a score-file

Enter your logname in order to get you personal experiment verification files listed.
With default user "ffundel" mainly the routine verification score files are offered.
Select one of the scores files shown in the top left field and hit the "Load data" button (if existent).
The diplayed score files can be refined by one of the buttons above:
all - shows all existing score-files
monthly - shows score-files valid from first date to last (exisiting) date of a month
latest - shows scores valid for forecasts starting with the last change in the paralell routines P0 or P1
experiments - shows scores valid for all forecasts of an experiment and its reference
After having loaded a file the selection fields should be filled with all possible values and an initial plot will appear.
Note: Some apps using larger score-files (time-series and station based scores) (re)plot only after hitting the 'make plot' button.
If no such button exists the plots are refreshed after each change in the selection menue.
You can change the plot resolution by e.g. pressing "Strg" and turning the mouse wheel inside the browser.


Plot results

The user can now access stratified verification results. The selection options on the left allow for
numerous combinations. Some of the selections allow for one option only, some allow multiple choices.
To add a selection click on one of the entries in the drop down menue. Remove one entry by marking it and press delete or
by placing the cursor and press backspace.
Adding/removing an entry alters the plot by adding/removing e.g. a new line color, line type, column, row etc.
Some of the selections take order into account, e.g. changing the order of the model names changes the plot order.

Note: Not all combinations are valid, e.g. inital hour and valid hour 00 and lead time 012 results in no plot or an error message
(future versions might catch this, as for now this feature is not implemented for performance reasons).

Note: Some scores are calculated but should not be considered.
As for TCC (tendency correlation) it is sensible to look only at valid_hour = initial hour.
Correlation scores for wind direction should not be considered.

Score differences

In case of selecting exactely 2 models the option 'score difference' appears. By setting checkmark the difference of the score
between model 1 and model 2 is calculated and plotted.
If 'score difference' is selected the further option to calculate the relative difference ((mod1-mod2)/mod2) will be available.

Summary plots

If the score visulaization by lead-time is chosen, summary plots are offered (usually the 2nd tab). The plot show the relative difference (in RMSE, CRPS or ROC area) between the first
two selected models for differenct sub-domains and variables. SYNOP based scores will be shown for each lead-time, TEMP based scoreas will be shown for each level and as
an average over all lead-times except vv=0. The second model will serve as reference. The relative difference in scores is calculated as 200*(exp1-exp2) / (exp1+exp2).
Realtive difference greater than +-15% are cut off.

Save plot + link to app

To save a plot as .png right-click and select 'save image'. Some plots will increase in size and resoultion if the browser zoom function
is used (ctrl + mouse wheel)
The 'Direct Link' can be used to open the application with the current settings (e.g. send this link by e-mail or save as favorite).
This feature is not yet implemented in all apps. To see the plot after opening the link, the pre-chosen score-file needs to be loaded first via the load button.
Note: The browser address bar is never going to change, only the 'Direct Link' will be updated.

Data export

Most apps offer the possiblity to export the date from the data tab. This tab also has filter functionalities. The data can be copied to clipboard or in various data formats
for further use.

App source

The source tab shows the source code of the app. It is implemented for resaons of transparany and to give any inclined user the possibility to reproduce and possibly modify the shown plots.
The first script defines the organization of the app, the second script containes code filling the selection panels o´and the plots. R code for the plot or summary can be found
in the output$plot or output$summary entries of the second script.

Nomenclature

Upper Air Variables:
Z geopotential (m^2/s^2) - geopotential height for a pressure level
T upper air temperature (K) - upper air temperature
TD upper air dew point (K) - upper air dew point (calculated from T and RH)
U u-component of wind (m/s) - upper air or 10M zonal wind speed
V v-component of wind (m/s) - upper air or 10M meridional wind speed
FF wind speed (m/s) - calculated wind speed from u- and v-component
DD wind direction (degree) - direction of wind, only if wind speed >= 3 m/s
RH relative humidity (0..1) -
radio occ. bending angle (Rad) - bending angle of signal path between 2 GPS satellites

SYNOP Variables:
PS - surface pressure
T2M - 2 metre temperature
TD2M - 2 metre dew point
RH2M - 2 metre rel. hum.
RAD_GL_1h - global radiation last hour
RAD_DF_1h - diffuse radiation last hour
TMIN_12h - 2 metre temperature minimum of last 12 hours
TMAX_12h - 2 metre temperature maximum of last 12 hours
FF - wind speed
DD - direction of wind
N,N_[LMH] - total, low, medium and high cloud cover
GUST_Xh -maximum wind gust during the last x hours
RR_Xh -precipitation amount in the last x hours


Scores: wiki
ME - Mean error (bias)
MAE - Mean absolute error
SD - Standard deviation of error
RMSE - Root mean squared error
R2 - Correlation coeffiecient (explained variance)
TCC - Tendency correlation (like anomaly correlation using observation at initialization instead of an observation climatology)
LEN - Length of observation-forecast paires available to calculate the score (not valid for TCC)
CRPS - Continuous ranked probability score (assuming normally distributed eps members)
IGN - Ignorance score (assuming normally distributed eps members)
OUTLIER - Percentage of observations outside the range of the ensemble members (calculated from talagrand index)
SPREAD - Mean ensemble standard deviation
SKILL (RMSE) - RMSE of ensemble mean
Brier score - Magnitude of the probability forecast error (smaller is better).
Brier skill score - Brier score w.r.t. a reference probability forecast, usually sample climatology (greater is better).
Brier rel. - .Reliability component of the Brier score. mean, conditional bias of a probability forecast.
Brier res. - Resolution component of the Brier score. Gives the forecast ability to discriminate events (true value).
Uncertainty - Variability of observations, lower values show less often occuring events.
ROC area - Relative (or receiver) operatin characterisitics gives the probability that the forecast probability is greater for an event than for a non-event.

Domains:
NH - Northern hemisphere (all stations >20°)
SH - Southern hemisphere (all stations <-20°)
TR - Tropics (all stations between -20° and 20°)
SP - South-West Pacific (selection by WMO ID, no ships etc.)
NA - North and Central America (selection by WMO ID, no ships etc.)
SA - South America (selection by WMO ID, no ships etc.)
EU - Europe (selection by WMO ID, no ships etc.)
AF - Africa (selection by WMO ID, no ships etc.)
AA - Antarctica (selection by WMO ID, no ships etc.)
AS - Asia (selection by WMO ID, no ships etc.)
CEU - COSMO-EU (all stations in bounding box)
CDE - COSMO-DE (all stations in bounding box)
GER - Only stations within Germany
S-W Siberia - lon between (60,90) & lat between (50,60)
N-W Siberia - lon between (60,90) & lat between (60,70)
E Siberia - lon between (90,130) & lat between (50,70)
N Africa - lon between (-10,40) & lat between (0,30)
S Africa - lon between (10,40) & lat between (-35,0)
W Northamerica - lon between (-120,-100) & lat between (30,50)
E Northamerica - lon between (-100,-70) & lat between (30,50)
N Northamerica - lon between (-120,-70) & lat between (50,70)
N Southamerica - lon between (-80,-40) & lat between (-30,0)
S Southamerica - lon between (-80,-40) & lat between (-50,-30)
China - lon between (80,130) & lat between (20,50)
AUS,NZ,INDON - lon between (102.5,175) & lat between (-55,17)
Antarctic - lon between (-180,180) & lat between (-90,-60)
Kazakhstan - lon between (55,80) & lat between (35,50)
Arctic - lon between (-180,180) & lat between (70,90)

Models:
COSMO - COSMO


Further questions

Please direct any comments, questions, suggestions to felix.fundel[at]dwd.de

This is the place for the documentation of experiments