1. Evaluating and Reporting Program Impacts: Contentious Public Issues Loretta Singletary Professor & Extension Educator University of Nevada Cooperative Extension

2. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Common Assumptions Can’t measure impacts of “process” programs because…  Handing out evaluations is awkward  Participants don’t like to complete forms  I don’t need to measure the impacts because I know when it is successful  I don’t want to report results of little to no impacts or negative impacts

3. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Common Assumptions Can’t measure impacts of “process” programs because…  These programs are just too complex  There is too much controversy  Some participants arrive angry, stay angry and leave angry  My role is limited to that of facilitator only  My role is that of educator only  My role is fact-finder and researcher only

4. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Dilemma All affected stakeholders do not see collaboration as a better alternative Stakeholders do not participate equally Participants have different skills, commitment, motives

5. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Dilemma Playing field is uneven Stakeholders lack sufficient time and receive varied “dosage” External forces are rushing the process Funding and time constraints are problems

6. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Required? LOGIC model is a systematic method for program planning that requires (handout) : 1. Situational Analyses 2. Priority Setting 3. Action Plan 4. Evaluation

7. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Benefits of Evaluation Serve to strengthen programs over the short and long term Evaluations determine strategies and methods that work best, under which conditions and for which purposes

8. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Benefits of Evaluation Enable comparisons with traditional decision-making processes to build public understanding of consensus- based processes

9. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluating Impacts of Programs that Address Contentious Public Issues WHAT? WHO? WHEN? HOW?

10. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 What to Evaluate? Learning objectives = desired changes Programs that address contentious public issues try to accomplish some basics:  Educate  Communicate  Explore options  Seek mutually satisfying agreements

11. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 What are Desired Changes? Contentious issue efficiently addressed Multiple options developed Multiple options considered Climate becomes less contentious Climate supports public discourse Conflict is resolved for mutual satisfaction

12. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Primary Impacts (Gray, 1989) Two objective criteria to measure impacts of a collaborative effort: 1. Negotiated agreement is reached 2. Agreement has been implemented

13. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impacts Subjective criteria show impacts of collaboration  Alter individuals’ skills to participate  Alter perceptions of others  Alter perceptions of issue  Alter attitudes  Alter behavior  Develop criteria to measure these changes

14. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impact Measures Improved…  Communication skills  Networking skills  Relationship building skills Increased hope for resolving the issue

15. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impact Measures Quality of Interaction  Volunteers learn how to share power  Treat one another fairly and with respect Operational Details  Meetings follow timeline  Meetings accomplish goals  Participants stay informed (Gray 1989)

16. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impacts… First order vs. second order effects Second order effects…  Increased knowledge about issues  Increased awareness of issues  Increased awareness of diverse views  New personal and working relationships  Stakeholders accept and understand scientific analyses  Respond to future conflict civilly (Innes, 1999) Best when retrospectively measured

17. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 When to Evaluate? Midcourse – partway through to assess progress and improvements End-of-process – at conclusion to assess learning, satisfaction and first order effects Retrospective – 12 to 18 months after process to identify value and stability of process and second order effects

18. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Midcourse Evaluation After a few months of process program to assess “how it is going”…  Are process criteria being met?  Are participants satisfied?  Are representatives communicating with their stakeholder groups?  Do all participants feel well informed?  Do all participants feel equally empowered?  Are participants engaged and interested?

19. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 End-of-Process Evaluation Within month or two of final meeting Experience still “fresh” in everyone’s mind Objectively assess accomplishments ð Participants feel time well spent? ð Participants plan to stay involved with implementing agreement? ð Participants would try this type of process again?

20. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Retrospective Evaluation Of single process takes place at least one year after process completed Examines immediate and longer term effects of program Traces over time program effects on participants and their relationships to one another

21. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Who to Evaluate? Program objectives determines WHO  Joint fact finding  Policy dialogue  Public-private partnerships  Regulatory negotiation  Advisory Committees  Public Forums  Seeking negotiated agreement

22. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Who to Evaluate? Program objectives determines WHO Process participants (stakeholders— those directly involved) Agencies not directly involved Citizens not directly involved Funding sources Program speakers, researchers, facilitators

23. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Focus Groups (least expensive) Face-to-face interviews Postal surveys Internet surveys

24. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Principles for Writing Questions  Simple wording, yet complete sentences  Avoid vague quantifiers  Avoid specificity (how many)  Equal scalar (positive and negative)  Provide “Don’t Know” category for each question *  State both ends of scale equally (Poor / Excellent)  Eliminate “check all that apply”  Avoid open ended questions  Offer ordered responses (scale)

25. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Close-ended questions with ordered responses provide participants with a response scale; Select one answer from a scale or fixed range of choices.

26. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? All citizens have a responsibility to conserve water on a daily basis.  Strongly disagree  Somewhat disagree  Neither agree nor disagree  Somewhat agree  Strongly agree

27. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? My ability to listen to others’ views has:  Improved very little  Improved little  Stayed about the same  Improved much  Improved very much

28. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? My understanding of the effect of salinity on the Lahonton Cutthroat Trout fishery has:  Improved very little  Improved little  Stayed about the same  Improved much  Improved very much

29. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Close-ended questions with unordered responses present answers in no particular order and participants must pick the one best response.

30. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Whose responsibility is it to see to it that watershed water quality improves? (check only one):  Area farmers  Environmental Protection Agency  My Regional Watershed Planning Group  Local elected officials  Environmental interest groups  Every citizen living in the watershed area

31. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? An example of a partially closed-ended question with unordered response is: How do you prefer to receive ABC Program information? (check all that apply)  Printed material mailed to me  Small group meetings  Large public forums  Newspaper articles  Other (please describe) ________________________________

32. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Quantitative Impacts Ranked means using Likert scale Percentages collapsing highs/low Likert scale Regression analyses using participant characteristics to find impact patterns Several other tests for relationships

33. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Qualitative Impacts Frequency count by key words  Verbs  Nouns  Direct objects Categorize comments by class  Liked most  Liked least

34. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Impacts Citizen participants Participating agencies Your administrators Land-grant universities (academic body) Funding sources

35. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Impacts 4-page fact sheets In-depth Extension publications Power point presentations to citizens Professional conferences (ANREP, NACAA, NACDEP, ESP, Galaxy) Refereed journal articles (JOE) News articles Radio and television interviews

36. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Bibliography Bickman, L., Rog, D.J. & Hedrick, T.E. 1998. Applied Research Design: A Practical Approach in L. Bickman and D.J. Rog (eds.) Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications. Bickman, L., Rog, D.J. & Hedrick, T.E. 1998. Applied Research Design: A Practical Approach in L. Bickman and D.J. Rog (eds.) Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications. Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, 2nd edition. New York: John Wiley and Sons. Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, 2nd edition. New York: John Wiley and Sons. Dillman, D. A. 1978. Mail and Telephone Surveys: The Total Design Method. New York, NY: John Wiley and Sons. Dillman, D. A. 1978. Mail and Telephone Surveys: The Total Design Method. New York, NY: John Wiley and Sons. Gray, B. 1989. Collaborating: Finding Common Ground for Multiparty Problems Gray, B. 1989. Collaborating: Finding Common Ground for Multiparty Problems

37. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Bibliography Heron, J. 1996. Co-operative Inquiry: Research into the Human Condition. Thousand Oaks, CA: Sage Publications. Heron, J. 1996. Co-operative Inquiry: Research into the Human Condition. Thousand Oaks, CA: Sage Publications. Innes, J. E. 1999. Evaluating Consensus Building. Thousand Oaks, CA: Sage Publications. Innes, J. E. 1999. Evaluating Consensus Building. Thousand Oaks, CA: Sage Publications. Patton, M.Q. 1978. Utilization-Focused Evaluation. Beverly Hills, CA: Sage Publications. Patton, M.Q. 1978. Utilization-Focused Evaluation. Beverly Hills, CA: Sage Publications. Salant, P. & Dillman, D.A. 1994. How to Conduct Your Own Survey. New York: John Wiley and Sons. Salant, P. & Dillman, D.A. 1994. How to Conduct Your Own Survey. New York: John Wiley and Sons. Singletary, L. 2004. Mail Surveys, in L. Singletary (Ed.) Conducting Community Situational Analysis: A Field Guide to Dynamic Extension Programming. Reno, NV: UNCE EB-04-02,available online at www.unce.unr.edu. Singletary, L. 2004. Mail Surveys, in L. Singletary (Ed.) Conducting Community Situational Analysis: A Field Guide to Dynamic Extension Programming. Reno, NV: UNCE EB-04-02,available online at www.unce.unr.edu.www.unce.unr.edu

38. Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Staci Emm University of Nevada Cooperative Extension 314 5th Street P.O. Box 810 Hawthorne, NV 89415 (775) 945-3444 fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary, Ph.D. Professor and Extension Educator University of Nevada Cooperative Extension Phone: (775) 463-6541 FAX: (775) 463-6545 Email: singletaryl@unce.unr.edusingletaryl@unce.unr.edu