1. Public policy implications of accuracy assessment for fire and deforestation monitoring: Don’t forget the error bars A story of sin and redemption Foster Brown Woods Hole Research Center/ Universidade Federal do Acre fbrown@uol.com.br III LBA Scientific Conference Brasilia, DF, 28 July 2004

2. Premises for this presentation 1.Indicators are not the reality, but approximations to a reality, such are deforestation estimates and hot pixel data 2.All data have error, or less pejoratively, have uncertainty. 3.Binary logic can be overused, particularly for generalities, such as: reliable/unreliable, correct/wrong, Needs to be coupled the word ‘depends.’

3. Premises applied 1.Approximations –Classified satellite pixels or fractions of pixels to estimate area deforested. –Satellite pixels classified hot to estimate fires. 2. Uncertainty (error) –Is the uncertainty small enough so to be able the answer the research question? 3. Are the data of X, Y or Z reliable/correct? Answer: Depends on the question asked.

4. Explicit versus Implicit Uncertainty Explicit (error bars): –Reports a range (absolute or probabilistic) around a value: 35 ha ± 3.0 ha. –Reports a range: 1-3 Pg C/year, without central value. Implicit –Uncertainty is in the last digit –12.4 means a value between 12.35 and 12.45 or 12.4 ± 0.05 –Number of significant figures determines implicit uncertainty.

5. Examples of explicit uncertainty I: opinion polls Candidate A was preferred by 50 percent of those polled with a margin of error of plus or minus 3 percent. Uncertainty based on binomial distribution and is likely to be higher due to sampling factors.

6. Examples of explicit uncertainty II: weather forests Question: Will it rain today? Responses –1. There is a 10% chance –2. There is a 30% chance –3. There is a 60% chance –4. There is a 90% chance When to carry an umbrella becomes a user’s decision.

7. The danger of too many significant figures: World Bank estimate in 1988 for deforestation in Brazilian Amazonia (Mahar 1988) 598,921.7 km 2, 7 significant figures, implicit uncertainty of ±0.00005% Student in class: “The gringos know all about us.” If it is published, especially by the World Bank, then in must be true. Subsequent analysis: 415,000 km 2 Difference of approximately 50% of most likely value, a million-fold higher uncertainty than the implicit uncertainty. Right or wrong? Wrong in implying that the number is so accurate. Implications for public policy: World Bank is fallible.

8. If the World Bank can make mistakes, so can researchers, local institutions, governments. The World Bank would have been right if it had reported 600,000 km 2 ±35% Excessive use of significant figures, 598,921.7 km 2, means loss of credibility

9. Examples for public policy for Acre The data are good/bad, depending on the question. Zoning for Acre State, Brazil (Sassagawa and Brown ZEE, 2000) Data sources: INPE, Funtac, Ibama, Michigan State University. Question: Is the future of Acre in the forestry sector? –If more than two-thirds of the state has forest cover, it would be a good sign.

10. Percent of Acre State, Brazil deforested by 1996 Note: All values < 10% deforested >90% of Acre has forest in 1996. Good news for public policy.

11. But what about how much deforestation has occurred in Acre? The simple answer: “By 1996, estimates ranged from 5.0 to 9.0%.” Other questions: Where is it occurring? INPE couldn’t answer, it is a state-wide estimate. Need to use the others. Is the rate changing? Only INPE has yearly data for which yearly deforestation rates can be calculated.

12. Example of Deforestation Rates for Acre State

13. Zoning report for Acre – 2000, reconciling two different data 9.0% for INPE and it is put in a time perspective with deforestation rates during the 1990s. 7.4% from Funtac which shows that most of the deforestation occurs in eastern Acre. We needed to explain why we can use 9.0% and 7.4% in the same chapter for area deforested in Acre. Different methods and definitions can produce different areas of deforestation.

14. Implications for Public Policy Increasing deforestation rates for Brazilian Amazon have been of great concern (INPE website19jan04): –2000-2001 18,166.00 km 2 /yr –2001-2002 25,476.00 km 2 /yr This reported increase of 40% has influenced Amazonia Sustentavel and other public policy initiatives. Most recent (INPE website 28Jul03) –2000-2001 18,165 km 2 /yr –2001-2002 23,266 km 2 /yr –Increase28%

15. CENA 2000-2001 2001-2002 km 2 /year Orginal data recently revised. Two image scenes from Acre in red. Implicit uncertainty <0.05%. 2001-2002 revised

16. CENA 2000-2001 2001-2002 km 2 /year Official data rounded to ± 5 km 2 /year (2 significant figures) Implicit uncertainty <5%. 2001-2002 revised

17. Official data rounded to ± 50 km 2 /year (1 significant figure) Implicit uncertainty < 50%. CENA (Image)2000- 2001 2001 - 2002 Revised 2001-2002 002-67100700300 Revision >50% 001-67200500200 Revision >50%

18. Fires in Acre Questions important for public policy: –A) How many fires are occurring in Acre? –B) Where and when are the fires occurring in Acre?

19. “Error” Matrix for Hot Pixel Data and public policy implications Fire observed on ground No fire observed on ground Error of commission = B/(A+B) Fire observed by satellite A, good B, false fire/ comission error/ false positive No fire observed by satellite C, missed fire/ omission error/ false negative Error of omission = C/(A+C) BAD FOR ENFORCE- MENT BAD FOR FIRE CONTROL

20. Synthesis of indicators for fires in the State of Acre for 2001 – NOAA 12 – great underestimate for 2001 (1/10?) Selhorst and Brown (2003)

21. Need to choose between low omission and low commission errors To minimize commission errors typically means accepting higher omission errors. Costs of being wrong depends on the objective. –For enforcement, high cost of commission errors led to INPE to minimize this error to avoid sending crews on wild goose chases (looking for fires that are not there). –For fire control, the fire not detected could be very costly: decrease omission error and increase commission error (fewer missed fires and more wild goose chases).

22. Critical role of hot pixel maps in showing where burning is likely to be occurring

23. Concentration of hot pixels showing where burning is occurring (MODIS data June-September 2003)

24. Pontos MODIS (jun-set03) Imagem LandSat 1999 Pontos em Fazendas P. A. Tocantins (1997) P. C. Humaitá (1981)

25. Public Policy Implications Absolute number of hot pixels can be misleading, frequently underestimates fires (omission errors). Combining hot pixels from various satellites can be useful show relative concentrations. Maps of hot pixels showing their spatial location can be very useful to indicate where a closer look is needed. Maps need to be timely: available to those who can use the information shortly after observations.

26. Keynote address for the 1999 Simposium N.R. Chrisman - Speaking truth to power: an agenda for change. (Title based on Quaker religious saying)

27. The sins of unjustified significant figures for deforestation estimates (I have sinned, too) 1988. World Bank. 0.00005% went to 50% uncertainty: factor of 1,000,000. 1996. INPE+IBAMA+FUNTAC+MSU. 5% to 20+%. differences: factor of 4. 2003. INPE Prodes. 0,05% to 50%: factor of 1,000.

28. Why Do We Sin? - 1 The legacy of 19 th Century Physics and calculus – the myth that absolute knowledge exists and can be calculated. –Concept of Exact Sciences is alive and well. University in Israel and Brazil have divisions of exact sciences. –"One aim of the physical sciences has been to give an exact picture of the material world. One achievement of physics in the twentieth century has been to prove that aim is unattainable.“ - Jacob Bronowski (1973) – So why do we keep trying to attain it? –Habit and world view

29. Habit – World View “One of the most pervasive forces in technology is the tyranny of the way things have always been done… The research community…does not influence the practice of GIS until the research concepts change the routine ways of doing things.” N.R. Chrisman (1999).

30. Why Do We Sin? - II Fear: “Perhaps the worst nightmare of a natural resources manager is to appear ‘uncertain’ to the public, or to admit that there is ‘error’ in the decision process being presented.” H.T. Mowrer (1999) Accuracy (re)assurance: selling uncertainty assessment to the uncertain.

31. Why Do We Sin? - III The bookkeeping approach is the most common means to present data –Neat, tidy, and always two numbers to the right of the decimal point –123.34 – 12.22 – 1.12 –.02 –Data are commonly reported in newspapers in a similar fashion. Spreadsheets are designed with this structure in mind.

32. How do we seek redemption? Develop sensitivity analysis for uncertainty propagation (Monte Carlo approach) Compare various sources of data for consistency/inconsistency. Example Aster, Modis, Landsat. Make uncertainty explicit with error bars and ranges – Don’t sin like the World Bank.

33. Questions and recommendations from the workshop on accuracy assessment 26 July 2004 How good is good enough for public policy? - I For deforestation rate estimates, let us try to document ±20% or less uncertainty. –2000-2001: 18,000 ± 3,600 km 2 /yr –2001-2002: 25,000 ± 5,000 km 2 /yr Remember the difference between precision and accuracy.

34. Questions and recommendations from the workshop on accuracy assessment 26 July 2004 How good is good enough for public policy? – II Hot pixel data – tendency for omission errors. Acre: 40,000 small rural producers who burn most years. Hot pixels: NOAA-12 ~ 1,000, MODIS ~ 7,000. Use hot pixels as a qualitative indicator (low, medium, high) and as an alarm (near real time) accessible to society. Compare hot pixels from different satellites as a redundancy check for alarms as a measure of uncertainty.

35. Questions and recommendations from the workshop on accuracy assessment 26 July 2004 Involve users in validation exercises – research and government institutions and rural organizations. Document explicit uncertainties for deforestation estimates and hot pixels/fire counts