Download presentation
Presentation is loading. Please wait.
1
confidence in classification
Paper 3 Technical guidance on achieving adequate confidence in classification CIS Working Group 2A ECOSTAT 1 July 2003
2
the Directive requires us to achieve an adequate level of confidence and to report this ...
Annex V, Section 1.3 and Section 1.3.4
3
consequence ... we need an estimate of the error in the values of metrics used to classify ... e.g. value (plus or minus 15%)
4
many quality elements ?
5
QE 1 QE 2 QE 3 QE 4 QE 5 QE 6 QE 7 QE 8 etc
6
high true false No QE is worse than High-Good limit QE 1 QE 2 QE 3
etc
7
high good mod poor bad true false true false true false true false
No QE is worse than High-Good limit true QE 1 QE 2 false good QE 3 No QE is worse than Good-Mod limit true QE 4 QE 5 mod false QE 6 etc true QE 7 false poor etc QE 8 true false etc etc bad true
8
one-out all-out
9
QE 1 QE 2 QE 3 QE 4 QE 5 QE 6 QE 7 QE 8 etc
10
Single element approach
metric 1 QE 1 metric 2 QE 2 metric 3 QE 3 metric 4 QE 4 metric 5 QE 5 metric 6 QE 6 metric 7 QE 7 metric 8 QE 8 etc etc Single element approach
11
Multi-metric approach
QE 1 metric 3 metric 4 metric 5 QE 1 metric 6 metric 7 metric 8 QE 3 metric 9 metric 10 Multi-metric approach
12
high good mod poor bad true false true false true false true false
No QE is worse than High-Good limit true metric 1 metric 2 QE 1 false good metric 3 No QE is worse than Good-Mod limit true metric 4 metric 5 QE 1 mod false metric 6 etc true metric 7 false poor etc metric 8 QE 3 true false metric 9 etc bad metric 10 true
13
effect of error from monitoring on these models
14
mean of 12 samples plus or minus 50%
15
number of taxa 12 ( )
16
principles apply to all metrics
17
leads to mis-classification 20% per QE
18
~ 20 % of sites site truly good is put wrongly into or high or mod
poor bad ~ 20 % of sites
19
wrong change of class 30%
20
between biological and
wrong difference between biological and chemical class 30%
21
lots of quality elements
each with 20% error one-out / all-out
22
100 fail 10% true waters
23
100 %reported %true number of QE’s
fail %reported fail %true waters number of QE’s
24
100 % reported % true number of QE’s
fail % reported fail % true waters number of QE’s
25
100 % reported % true number of QE’s
fail % reported fail % true waters number of QE’s
26
one-out / all-out ... is vulnerable to errors ...
27
extremely high high
28
controls ... 1 averaging 2 significance test 3 exclude QE’s
29
controls ... 1 averaging 2 significance test 3 exclude QE’s all needed
30
Multi-metric approach
1 averaging Multi-metric approach
31
high good mod poor bad true false true false true false true false
No QE is worse than High-Good limit true metric 1 metric 2 QE 1 false good metric 3 No QE is worse than Good-Mod limit true metric 4 metric 5 QE 1 mod false metric 6 etc true metric 7 false poor etc metric 8 QE 2 true false metric 9 etc bad metric 10 true
32
high good mod poor bad true false true false true false true false
No QE is worse than High-Good limit true metric 1 metric 2 QE 1 false good metric 3 No QE is worse than Good-Mod limit true metric 4 metric 5 QE 1 mod false metric 6 etc true metric 7 false poor etc metric 8 QE 2 true false metric 9 etc bad metric 10 true
33
8% high good mod poor bad true false true false true false true false
No QE is worse than High-Good limit true metric 1 metric 2 QE 1 false good metric 3 No QE is worse than Good-Mod limit true metric 4 metric 5 QE 1 mod false metric 6 etc true metric 7 false poor etc metric 8 QE 2 true false metric 9 etc bad metric 10 true
34
high good mod poor bad true false true false true false true false
No QE is worse than High-Good limit true metric 1 metric 2 QE 1 false good metric 3 No QE is worse than Good-Mod limit true metric 4 metric 5 QE 1 mod false metric 6 etc true metric 7 false poor etc metric 8 QE 2 true false metric 9 etc bad metric 10 true
35
100 %reported %true number of QE’s
fail %reported fail %true waters number of QE’s
36
limits to averaging ...
37
averaging will reduce mis-classification
hydrology averaging one-out, all-out nutrient averaging organic enrichment metrics grouped by pressure
38
But averaging can hide impacts ...
Sensitive metric
39
Composition and abundance undisturbed – no impacts
3 species of fish are present
40
abundance disturbed but composition unaffected
abundance has changed but 3 species are still present composition AND abundance must be no more than slightly changed for good status to be achieved
41
2 significance test
42
high No QE is significantly worse than High-Good limit true
43
high good mod poor bad QE 1 QE 2 QE 3 QE 4 QE 5 QE 6 QE 7 QE 8 etc
No QE is significantly worse than High-Good limit true QE 1 QE 2 false good QE 3 No QE is significantly worse than Good-Mod limit true QE 4 QE 5 mod false QE 6 etc true QE 7 false poor etc QE 8 true false etc etc bad true
44
100 %reported %true number of QE’s
fail %reported fail %true waters number of QE’s
45
100 %reported %true number of QE’s
fail %reported fail %true waters number of QE’s
46
at least 95% confidence? what is significant?
(for serious consequences)
47
consequence ... monitoring must produce an estimate of the error in the values of metrics used to classify ... e.g. value (plus or minus 15%)
48
monitoring where we cannot do this should not be used to classify ...
49
controls ... 1 averaging 2 significance test 3 exclude QE’s
50
Annex II, Section 1.3 Where it is not possible to establish reliable ... reference conditions for a quality element ... due to high ... natural variability ... then that element may be excluded ...
51
Annex V 1. 3. 2 Design of Operational Monitoring
Annex V Design of Operational Monitoring to assess the impact of ... pressure Member States shall monitor ... parameters indicative of the biological quality element, or elements, most sensitive to the pressures … parameters indicative of the hydromorphological quality element most sensitive to the pressure
52
exclude QE if ... no reliable estimate of reference conditions
QE not sensitive to the pressures pressure covered by other QEs
53
Exclude Quality Elements
QE 1 QE 2 QE 3 QE 4 QE 5 Exclude Quality Elements QE 6 QE 7 QE 8 etc
54
Exclude Quality Elements
high No relevant QE is significantly worse than High-Good limit true QE 1 QE 2 false good QE 3 No relevant QE is significantly worse than Good-Mod limit true QE 4 QE 5 mod Exclude Quality Elements false QE 6 etc true QE 7 false poor etc QE 8 true false etc etc bad true
55
Exclude Quality Elements
No relevant QE is significantly worse than High-Good limit high true metric 1 metric 2 QE 1 good metric 3 false true metric 4 etc metric 5 QE 1 mod false Exclude Quality Elements metric 6 etc true metric 7 false poor etc metric 8 QE 3 true false metric 9 etc bad metric 10 true
56
100 % reported % true number of QE’s
fail % reported fail % true waters number of QE’s
57
100 % reported % true number of QE’s
fail % reported fail % true waters number of QE’s
58
summary
59
100 %reported %true number of QE’s
fail %reported fail %true waters number of QE’s
60
controls ... 1 averaging 2 significance test 3 exclude QE’s
61
Exclude Quality Elements
high no relevant QE is significantly worse than High-Good limit true QE 1 QE 2 false good QE 3 No relevant QE is significantly worse than Good-Mod limit true QE 4 QE 5 mod Exclude Quality Elements false QE 6 etc true QE 7 false poor etc QE 8 true false etc etc bad true
62
significant? at least 95% confidence? (for serious consequences)
63
we need an estimate of the error in the values of metrics used to classify ...
e.g. value (plus or minus 15%)
64
100 % reported % true number of QE’s
fail % reported fail % true waters number of QE’s
65
confidence in classification
Paper 3 Technical guidance on achieving adequate confidence in classification CIS Working Group 2A ECOSTAT 1 July 2003
Similar presentations
