1. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station The Science/Policy Interface in Logic based Evaluation of Forest Ecosystem Sustainability Keith M. Reynolds, USDA Forest Service K. Norman Johnson, Oregon State University Sean N. Gordon, Oregon State University

2. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Acknowledgments USDA Forest Service Washington Office USDA Forest Service Washington Office National Forest System, Ecosystem Management National Forest System, Ecosystem Management Pacific Northwest Research Station Pacific Northwest Research Station Social and Economic Values RD&A Program Social and Economic Values RD&A Program

3. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Overview Introduction Introduction Knowledge bases and logic modeling Knowledge bases and logic modeling Analysis Analysis Model design issues Model design issues Lessons learned Lessons learned Recommendations Recommendations

4. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Introduction: indicators An indicator is any variable or component of the forest ecosystem … used to infer attributes of the sustainability of the resource and its utilization. Indicators should convey a single meaningful message. This single message is termed information. It represents an aggregate of one or more data elements with certain established relationships (Prabhu et al. 2001).

5. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Introduction: criterion A standard that a thing is judged by (Prabhu et al. 2001). A standard that a thing is judged by (Prabhu et al. 2001). Criteria are the intermediate points to which the information provided by the indicators can be integrated and where an interpretable assessment crystallizes. Criteria are the intermediate points to which the information provided by the indicators can be integrated and where an interpretable assessment crystallizes. Principles [e.g., sustainability] form the final point of integration. Principles [e.g., sustainability] form the final point of integration. A criterion should be treated as a reflection of knowledge. A criterion should be treated as a reflection of knowledge. It can be viewed as a large scale selective combination … of related pieces of information. It can be viewed as a large scale selective combination … of related pieces of information.

6. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Introduction: measurement endpoint Some indicators are simple. Some indicators are simple. Their definition suggests an obvious one to one correspondence between an indicator and a metric for that indicator. Their definition suggests an obvious one to one correspondence between an indicator and a metric for that indicator. Definitions of some indicators are more complex. Definitions of some indicators are more complex. They represent a synthesis of two or more data elements, which we refer to as measurement endpoints. They represent a synthesis of two or more data elements, which we refer to as measurement endpoints.

7. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Introduction: scales of application Purpose varies with scale (Castañeda 2001) Purpose varies with scale (Castañeda 2001) National and regional National and regional Policy instruments to evaluate laws, policy, regulations Policy instruments to evaluate laws, policy, regulations E.g., Montreal Process, NWFP, ICBEMP E.g., Montreal Process, NWFP, ICBEMP Management unit Management unit Evaluation and adjustment of management practices Evaluation and adjustment of management practices E.g., CIFOR, USDA FS IMI E.g., CIFOR, USDA FS IMI

8. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Introduction: objectives 1.Illustrate the value of a logic-based approach in designing a formal specification to evaluate the Montreal criteria and indicators. 2.Identify the roles of science and policy in this effort. 3.Highlight lessons learned from this process. 4.Suggest some general recommendations.

9. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Knowledge bases A form of meta database A form of meta database A formal logical representation of how to evaluate information A formal logical representation of how to evaluate information Networks of interrelated topics Networks of interrelated topics Mental map Mental map Advantages Advantages Interactive, graphic design (modularity) Interactive, graphic design (modularity) Numerous & diverse topics can be analyzed within a single integrated analysis Numerous & diverse topics can be analyzed within a single integrated analysis

10. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Knowledge bases: forms of uncertainty Probabilistic Probabilistic Uncertainty of events Uncertainty of events Linguistic (or lexical) Linguistic (or lexical) Lofti Zadeh, 1966 Lofti Zadeh, 1966 Uncertainty about the definition of the event Uncertainty about the definition of the event A proposition is the smallest unit of thought to which one can assign a measure of truth A proposition is the smallest unit of thought to which one can assign a measure of truth Truth value metrics Truth value metrics Indices that quantify the degree of support for a proposition provided by its premises Indices that quantify the degree of support for a proposition provided by its premises

11. Knowledge bases: networks of topics Concern 1 Ecostate AEcostate B Ecostate CEcostate DData link = topic Concern 2 Etc. Data

12. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Knowledge bases: topics Each typically evaluates a proposition Each typically evaluates a proposition Attributes of topics Attributes of topics Name Name Proposition Proposition Truth value: a measure of support for the proposition Truth value: a measure of support for the proposition Documentation Documentation Explanation, source, citations Explanation, source, citations

13. Knowledge bases: evaluation Concern 1 Ecostate A Ecostate BEcostate C Data link Get data requirements Evaluate data

14. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Knowledge bases: fuzzy logic

15. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Analysis: Montreal C&I The Montreal specifications provide relatively clear definitions of biophysical, socioeconomic, and framework attributes requiring evaluation (WGCICSMTBF 1995)... The Montreal specifications provide relatively clear definitions of biophysical, socioeconomic, and framework attributes requiring evaluation (WGCICSMTBF 1995)... But, design of evaluation procedures that allow interpretation of the Montreal C&I is one of the major technical issues that remain to be resolved (Raison et al. 2001). But, design of evaluation procedures that allow interpretation of the Montreal C&I is one of the major technical issues that remain to be resolved (Raison et al. 2001).

16. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Analysis: conceptual framework 1.Specified conditions or outcomes to be sustained (the indicators). 2.A measure for each condition or outcome. 3.Calculation of the level of the indicator over some time period using the selected measure. 4.A frame of reference for gauging sustainability. 5.Rules for deciding when sustainability has been achieved (sustainability check). 6.A monitoring program. 7.A formalism that supports requirements 1 to 6.

17. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Analysis: logic models as design frameworks Logic models (knowledge bases) provide a formal specification for organizing and interpreting information. Logic models (knowledge bases) provide a formal specification for organizing and interpreting information. NetWeaver kb developer system NetWeaver kb developer system Problem represented in terms of propositions about topics of interest and their interdependencies. Problem represented in terms of propositions about topics of interest and their interdependencies. Topics translated into propositions. Topics translated into propositions. Fuzzy logic to accommodate lexical uncertainty. Fuzzy logic to accommodate lexical uncertainty.

18. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Analysis: logic models as design frameworks (continued) Need for transparency (Prabhu et al. 2001) Need for transparency (Prabhu et al. 2001) Models embody important policy decisions. Models embody important policy decisions. Models depend on value judgments and critical assumptions that need clear documentation. Models depend on value judgments and critical assumptions that need clear documentation. Model development Model development Graphic representation is an effective basis for organizing discussion and for evolution of design. Graphic representation is an effective basis for organizing discussion and for evolution of design. Communication Communication Between scientists and policy makers. Between scientists and policy makers. With interested publics. With interested publics.

19. Design issues: model organization Basic organization of topics. For example, evaluation of criteria in the current prototype.

20. Design issues: model organization An alternative organization with very different emphasis on criteria.

21. Design issues: synthesis ADD operator: arguments evaluated as limiting factors. ADD operator: arguments evaluated as limiting factors. SUM operator: arguments contribute incrementally to evaluation and can compensate. SUM operator: arguments contribute incrementally to evaluation and can compensate.

22. Design issues: synthesis Another example, including the OR operator.

23. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Design issues: weighting Intrinsic weights Intrinsic weights Each topic in a NetWeaver logic model has an intrinsic weight attribute. Each topic in a NetWeaver logic model has an intrinsic weight attribute. E.g., set weight attribute on any topic to adjust its contribution of evidence to a proposition. E.g., set weight attribute on any topic to adjust its contribution of evidence to a proposition. Bad idea: part of specification, but not obvious. Bad idea: part of specification, but not obvious. Explicit weights Explicit weights Better, but add another layer of subjectivity. Better, but add another layer of subjectivity. Some valid purposes, however. Some valid purposes, however.

24. Design issues: reference conditions Each fuzzy node evaluates a measurement endpoint against reference conditions. Each fuzzy node evaluates a measurement endpoint against reference conditions. Lack of reference conditions is a basic problem for most measurement endpoints. Lack of reference conditions is a basic problem for most measurement endpoints.

25. Design issues: reference conditions Implementation of a fuzzy node to evaluate measurement endpoint against reference conditions.

26. Design issues: qualitative measures Outcomes evaluated on an ordinal scale.

27. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Design issues: reliability of data Reliability of data for evaluation of Montreal C&I. Reliability of data for evaluation of Montreal C&I. Stochastic, rather than lexical, uncertainty Stochastic, rather than lexical, uncertainty Formal representation of stochastic uncertainty is problematic in the context of a logic model. Formal representation of stochastic uncertainty is problematic in the context of a logic model. Not addressed in the current Montreal C&I prototype. Not addressed in the current Montreal C&I prototype. Possible solution Possible solution Adjusting topic weights with a normalized metric such as standard error of the mean. Adjusting topic weights with a normalized metric such as standard error of the mean. Problems: availability, unknown error correlations Problems: availability, unknown error correlations

28. Sequential OR (SOR) to specify multiple alternative pathways in order of preference. Design issues: precision of knowledge

29. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Lessons learned 1.Lexical uncertainty is an important issue in evaluation of Montreal criteria and indicators. 2.Many aspects of evaluating sustainability cannot be answered by science alone. 3.Acquiring data on sustainability is necessary but not sufficient for setting policy. 4.Evaluating the state of sustainability and deciding how to respond are separate but interdependent decision processes. 5.Evaluating sustainability is not the same as defining desired future conditions.

30. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Recommendations Assess the policy role in sustainability evaluation, and undertake a policy review of model organization and strategies for integrating sustainability information. Assess the policy role in sustainability evaluation, and undertake a policy review of model organization and strategies for integrating sustainability information. The clearest, and most critical, role of science is in development of reference conditions. The clearest, and most critical, role of science is in development of reference conditions. A major effort is needed to identify measurement endpoints for indicators of the institutional framework (criterion 7). A major effort is needed to identify measurement endpoints for indicators of the institutional framework (criterion 7).

31. USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station Authors Keith M. Reynolds Keith M. Reynolds USDA Forest Service, Pacific Northwest Research Stn. USDA Forest Service, Pacific Northwest Research Stn. kreynolds@fs.fed.us kreynolds@fs.fed.us kreynolds@fs.fed.us K. Norman Johnson K. Norman Johnson Oregon State University, College of Forestry Oregon State University, College of Forestry norm.johnson@orst.edu norm.johnson@orst.edu norm.johnson@orst.edu Sean N. Gordon Sean N. Gordon Oregon State University, College of Forestry Oregon State University, College of Forestry sean.gordon@orst.edu sean.gordon@orst.edu