Issues in the

verification of

EPS

Åke Johansson

SMHI

Spread-Skill T2M from HIRLAM_K for JAN 2016

CU

The Commonly Used

Spread-Skill Relationship

MRMSEE

2

fo

MRMSEE E.Dev.StdE

2

fo 2

ff

CU SKILL – SPREAD RELATION

VAREEMSEE

.Dev.StdERMSEE

M

EM

CU SKILL – SPREAD RELATION

VAREEMSEE

.Dev.StdERMSEE

M

EM

CU Skill-Spread relation

is only valid for an

Ideal ensemble

4 reasons why

it is inappropriate for

Real World EPS

CU SKILL – SPREAD RELATION

VAREEMSEE M

Real World Models

usually have a

BIAS

VAREESDEE

VAREEMSEE

2

M

M

Real World Models

usually have members that

are not statistically equal

P

2

MP

2

M

VAREESDEE

VAREESDEE

Using U-Statistics for estimating

Skill and Spread

gives more realistic estimates

SKILL

SPREAD

SPREAD

K

1k

2

fofo

1K

1k

P ),t,t(f)k,t,t(f)1K(K

2SPRE

SPREAD

K

1k

2

fofo

1K

1k

P ),t,t(f)k,t,t(f)1K(K

2SPRE

U-statistic

A U-statistic is defined as the average –

across all combinatorial selections of the given size from the

full set of observations –

of the basic estimator applied to the sub-samples.

U-statistics, where the letter U stands for unbiased, arise

naturally in producing minimum-variance unbiased

estimators.

SPREAD

K

1k

2

fofo

1K

1k

P ),t,t(f)k,t,t(f)1K(K

2SPRE

U-statistic

A U-statistic is defined as the average –

across all combinatorial selections of the given size from

the full set of observations –

of the basic estimator applied to the sub-samples.

U-statistics, where the letter U stands for unbiased, arise

naturally in producing minimum-variance unbiased

estimators.

SPREAD

K

1k

2

fofo

1K

1k

P ),t,t(f)k,t,t(f)1K(K

2SPRE

2

fofoP )t,t(f)k,t,t(fVARE

PPPP VAREE2SPREEVARE

1K

K2SPRE

SKILL

2

PP

2

P MEMSESDE

SKILL

U-statistic

P

2

PSPREESDEE

U.UI SKILL – SPREAD CONDITION

P

2

PVAREESDEE

2

1

P

2

PSPREESDEE

U.UI SKILL – SPREAD CONDITION

P

2

PVAREESDEE

2

1

P

2

P

P

2

MP

VAREESDEE2

1

VAREESDEE

VAREESDEE

VAREEMSEE

2

M

M

CU U.UI

U

I

U

Example for T2M from HIRLAM_K for JAN 2016

Example for T2M from HIRLAM_K for JAN 2016

Why U-Statistics

Theoretical justification - Spread

aFE

22'FEaFE

2

2

ji

2

ji

2

ji

'FE2

'F'FE

'Fa'FaEFFE

Theoretical justification - Spread

aFE

22'FEaFE

2

2

ji

2

ji

2

ji

'FE2

'F'FE

'Fa'FaEFFE

Theoretical justification - Spread

22

ji

22

'FE2FFE

'FEaFE

Theoretical justification - Skill

aFE,0OE

222a'OEaOE

222

2

22

a'OE'FE

'O'FaE

'O)O(E'F)F(EEOFE

Theoretical justification - Skill

aFE,0OE

222a'OEaOE

222

2

22

a'OE'FE

'O'FaE

'O)O(E'F)F(EEOFE

Theoretical justification - Skill

222

222

a2

1'FE2aFE

a'FEaOE

Spread Skill

222

222

a2

1'FE2'FE2

a'FE'FE

Real World Models

are usually not verified against

the ”Truth”

(F, O)

Inflation = 1.0

Inflation = 1.1

Inflation = 1.16

Example for T2M from HIRLAM_K for JAN 2016

Conclusions

Over-dispersiveness – not under-dispersiveness

– seems to be the problem with EPS

Therefore one can wonder whether the following

really is necessary

• Parameter perturbations

• Stochastic physics

• Stochastic kinetic energy backscatter