Presentation on theme: "M&E Working Group and Evaluation Team GEOSS and Users: User Engagement in the Evaluation Process Input from User Interface Committee Lawrence Friedl (USA-NASA)"— Presentation transcript:
M&E Working Group and Evaluation Team GEOSS and Users: User Engagement in the Evaluation Process Input from User Interface Committee Lawrence Friedl (USA-NASA) Evaluation Kick-off Meeting Geneva, Switzerland January.2010
2 Group on Earth Observations Users & Evaluation Process Sections of Presentation I.Users II.UIC work with users & user engagement III.Mechanisms to reach users Discussion
Group on Earth Observations I. Users 3 UIC Working Definition of the GEOSS User The user may be any person that uses GEO data or information at various levels of sophistication. They may be considered as entities along a value chain that includes, for example, the raw data collector, data archivist, modeler, scenario builder, program manager, policy analyst, and political decision maker. Users will include the specialist technician, scientist, politician and lay public. The user may be an expert in the sense that they have prior knowledge of how web portals may work and intuitively know how to find the data. And, the user may be a novice, such as an individual of the lay public, who may wish to explore information related to an unusual environmental event that he or she has just witnessed.
Group on Earth Observations I. Users 4
5 Novice User Science-to-policy analysts, decision makers, public officials, and the public May be interested in specific issues that fall within one or more SBAs Not GEOSS- or SOA-experienced Looking for any and all observational data relevant to their issues Want to easily find it and view it
Group on Earth Observations I. Users 6 GMES Network of Users – User Typology Users may be classified along various dimensions: Position on value-adding chain (provider, intermediate, end user) Political level they operate in (international, national, regional, local) Field of work (e.g., GEO SBA) Role in projects Activity level - operational, active, passive, nominal, statistical user Reception of products - how likely to give feedback and be vocal about use Further and other ones are possible Approach: User types may be defined in two or more-dimensional matrices
7 Group on Earth Observations Users & Evaluation Process Sections of Presentation I.Users II.UIC work with users & user engagement III.Mechanisms to reach users Discussion
8 Group on Earth Observations II. UIC and Users Current UIC User-oriented Approaches & Activities A.User/Demand-side view of Earth observation priorities (Task US-09-01a) B.User Types Analysis C.User-based Call for Proposals D.Usability Testing of GCI & Portals E.Communities of Practice F.GEO Task Sheet Analysis
9 Group on Earth Observations Task US-09-01a GEO Task US-09-01a: Establish a GEO process for identifying critical Earth observation priorities common to many GEOSS societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems requirements development processes. UIC is using an approach that engages representatives of key user groups and geographic regions to identify observation priorities listed in user-oriented documents.
10 Group on Earth Observations Task US-09-01a Analysts and Advisory Groups include people from Australia, Austria, Canada, China, Costa Rica, Denmark, Germany, Finland, France, Ghana, India, Iran, Italy, Kenya, Japan, Mexico, Norway, Paraguay, Russia, USA, Senegal, South Africa, Thailand, Tunisia, CEOS, DIVERSITAS, ECMWF, ESA, FAO, GCOS, IEEE, UNESCO, WMO, and others.
Agriculture SBA User Types Forest Protection Officers and Firefighters Urban Foresters and Arborists Microbiologists Watershed Manager Conservationists Biologist Ecologist Land Use Planners Health Specialists and Surveyors Land Use Managers Land Developers Natural Resource Extraction (Loggers, Miners, Oilman) Protected Area Managers Park Ranger Plant Pathologist Recreation and Tourist Specialists Tourists 11
Biodiversity SBA User Types Fish & Wildlife Biologists/Managers Managers Protected Area Managers Botanists Taxonomists Conservationists Ecologists Land Use Planners Land Use Managers Land Developers Natural Resource Extraction (Loggers, Miners, Oilman) Policy Makers (National, State, and Local Governments; NGOs) Academicians Researchers Community Based Monitoring Groups Concerned Citizens (e.g., bird watchers/groups) Private Land Owners 12
Climate SBA User Types PRIMARY USE OF CLIMATE OBSERVATIONS Modeling and Scenario Development Climate modelers Integrated assessment modelers (e.g., IMAGE, DICE, MINICAM, MERGE) Meteorologists Intergovernmental Panel on Climate Change (IPCC) of the WMO and UNEP by way of the international research community contributing to these periodic assessments SECONDARY USE OF CLIMATE OBSERVATIONS Examples in Resource Science and Management Model and scenario results in turn used by: Marine biologists Hydrologists Ecologists Energy researchers Geographers Earth system scientists Environmental scientists Conservationists Etc. POLICY USE OF CLIMATE OBSERVATIONS Examples of Decision Makers Science and management results in turn used by: National and sub national governments and international efforts in policy design for GHG stabilization, verification, monitoring, and adaptation Infrastructure managers, planners and investors (such as the World Bank, Army Corps of Engineers, energy planners) Etc. 13
Disasters SBA User Types Research and Forecasting -University Professors -Graduate Students -Researchers at national research centers -Researchers at government research centers -Weather Forecasters and Modelers -Flood Forecasters and Modelers -Landslide Forecasters and Modelers -Earthquake Forecasters and Modelers -Seismologists -Geologists -Hydrologists Emergency Management and Public Outreach -Risk Mitigation Manager -National and Local Emergency -Planning Managers -Weather Service Providers -Media: – Anchormen – Weathermen – Journalists -Local Law Enforcement/ Fire Department Officers Infrastructure Civil Engineers Earthquake Engineers Environmental Engineers Water Resource Manager City Planners and Zoners Regional Planners Insurance Assessors Risk Assessors 14
Ecosystems SBA User Types Water resources managers Natural resources managers Wildlife managers Coastal planners and developers Land use planners Agricultural managers Policymakers (local, state, national, Environmental Protection Departments) Researchers (climate, ecosystem function, biological invasion, disease, future change) Climate impacts analysts 15
Energy SBA User Types Researchers -Resource Assessment (e.g., at a national lab, studying solar energy potential) -Technology Development (e.g., wind turbine designer) -Energy model developer (e.g., for analysis of likely future biomass scenarios) Energy Facility Developers and Operators -e.g., Biofuels refinery planner, for plant siting and operations logistics -e.g., Wind farm developer, for plant siting and operations logistics -e.g., State utility, developing plans for how to meet a renewable portfolio standard -e.g., Water resources manager, deciding how to operate a hydropower facility Energy Policy Developers -e.g., Energy Department official studying renewable portfolio standard options Energy Regulators -e.g., Energy Commission official responsible for overseeing grid integration Private Users -e.g., Rural village leader, utilizing renewable energy tools, to optimize village energy system including options such as micro-hydropower -e.g., Citizen determining whether to purchase a solar water heater Electric Grid Operators -e.g., Independent System Operator (ISO) trying to predict how much wind power they will need to integrate into their regional grid system on a given day Building Designer or Engineer -e.g., American Society of Heating, Refrigeration, and Air Conditioning Engineer (ASHRAE), to minimize building heating/cooling costs 16
Human Health: Aeroallergens SBA User Types Air quality model and systems developers/providers Air quality scientists Health care providers Health outreach and education professionals Health risk analysts Interested/aware/concerned members of the public, including sensitive populations Public health organizations Public health policy organizations Public health managers and decision-makers Public health policy makers Non-governmental organizations (NGOs) and advocacy groups Health scientists (e.g., epidemiologists, immunologists) Weathercasters 17
Use of Assessments and Forecasts to Promote Health Protection -National and Sub national Policy Makers in Environmental and Public Health Sectors -National and Sub national Air Quality Managers -National and Sub national Public Health Managers and Decision-makers -Health Care Providers -Weathercasters -Health Outreach and Education Professionals -Interested/Aware/Concerned Members of the Public -Businesses Supporting Local Clean Air Programs -NGOs and Advocacy Groups Direct Use of Earth Observations (EO) -Air Quality Researchers -Environmental Scientists Use of EO to Develop and Populate Integrative Systems, Analytical Tools, and Models -Air Quality Model and Systems Developers/Providers -Public Health Specialists Use of Tools and Models to Assess and Forecast Air Quality -Air Quality Scientists -Environmental Process Modelers and Researchers -Public Health Specialists 18 Human Health: Air Quality SBA User Types
ENVIRONMENTAL CHANGE*HEALTH PROTECTION Direct Use of Earth Observations (EO) Climate Scientists* Environmental Scientists (Ecologists, Land Use/Cover Specialists, etc.)* International Public Health Organizations (e.g., WHO Global Health Observatory) National and Sub national Public Health Policy Makers Use of EO to Develop and Populate Integrative Systems, Analytical Tools, and Models Environmental Process Modelers and Systems Developers* Public Health Risk Modelers and Systems Developers Environmental Benefits Valuation Specialists Use of Tools and Models to Assess and Forecast Disease Incidence and Trends Health Risk Analysts Public Health Scientists (e.g., Epidemiologists, Infectious Disease Specialists) Use of Assessments and Forecasts to Promote Health Protection National and Sub national Policy Makers in Environmental Sectors* NGOs and Advocacy Groups* Infrastructure Managers, Planners and Investors* International Public Health Policy Organizations National and Sub national Public Health Policy Makers National and Sub national Public Health Managers and Decision-makers Health Care Providers Health Outreach and Education Professionals Interested/Aware/Concerned Members of the Public NGOs and Advocacy Groups 19 Human Health: Infectious Disease SBA User Types
Water SBA User Types Climate Scientist Climate Forecaster Weather Forecaster Hydrologist Flood Forecaster Drought Forecaster Water Resource Manager Water Res. Allocator Urban Water Supplier Agronomist/Farmer Hydro Power Engineer Energy (other) Engineer Heating/cooling Engineer Irrigation Scheduler Manager Conservationist Ecologist Environ. Engineer Fire Prevention Planner Fire Fighter Environmental Protection Manager Nat. Disaster Manager Drought Monitor Drought Mitigation Manager Flood Control Manager Flood Control Planner Nat. Hazards Risk Assessor Insurance purveyor Land Use Planner Epidemiologists Disease Outbreak Forecaster Pollution Forecaster Water Quality Manager Urban Planners City Development Zoner Environmental Impact Assessor 20
Weather SBA User Types Public organizations Civil protection agencies and organizations: Warning systems operator Crisis / disaster / emergency manager Risk manager Fire department officer Police officer -National meteorological and hydrological services (NMHS): Weather forecaster Meteorologist (e.g. scientific applications, systematic climatologic monitoring) Climate modeler Flood modeler, flood forecaster -Miscellaneous: Water resource manager Water resource scientist Private companies -Aviation: Airport / aerodrome operator Air traffic controller / control tower operator Pilot -Civil engineering: Building engineer (e.g., optimization of heating technology) -Commercial shipping and offshore industry: Captain Nautical engineer Offshore facility engineer Ship owner -Energy industry: Renewable energy consultant (e.g., assessment of available energies) Renewable energy project manager (e.g., planning of photovoltaic or wind installations) -Health care: Health resort manager Health clinic manager Doctor (e.g. dermatologist) -Media: Weatherman -Meteorological service providers: Weather forecaster Meteorologist -Telecommunication: Telecommunication engineer (e.g., calculation of microwave telecommunication signal losses) -Tourism: Hotel keeper Travel agent Event manager Private users -Agriculture: Farmer -Aviation: Pilot Balloonist -Shipping: Yachtsman -Miscellaneous private users: Hobby meteorologist Scientific organizations -Universities and research centers: Scientists from various geoscientific and earth system science disciplines 21
22 Group on Earth Observations CFP for Decision Support Projects Overview CFP seeks proposals for projects to apply Earth observations to improve policy decisions and management decisions. CFP seeks to identify and enable practical applications of Earth obs. Strong focus on Developing Countries and supporting end-users Showcase the value and benefits of Earth obs. and GEOSS through successful applications examples. Topics & GEO Tasks Focus on 4 SBAs: – Agriculture (including Forests) – Energy – Health (including Air Quality) – Water CFP supports GEO Tasks: CB (Mobilising Resources) US-09-01b (Communities of Practice) Project Types: I. New users to Earth obs. II.Some experience using Earth obs III.Lots of experience with Earth obs
23 Group on Earth Observations CFP for Decision Support Projects Type I & II Proposals (by SBA) Agriculture32 Energy3 Health18 Water57 Total:110 Majority were Type II
Group on Earth Observations GEOSS Portal/GCI Usability Testing GEO Web Portal Usability Testing GEO UIC set up computer workstations at locations where range of users could get onto GEOSS system and provide feedback Four rounds: ISRSE Conference (May 2009), electronically (Sept. 2009), GEO Plenary VI (Nov. 2009), ESIP Winter meeting (Jan. 2010) Two main types of questions: 1. USER INFORMATION QUESTIONS How would you classify yourself professionally? What best describes your GIS and geospatial experience? Identify the geographic area of interest (AOI) for your search. 2. GEO WEB PORTAL OPERATIONAL QUESTIONS 24
Group on Earth Observations Community of Practice Communities of Practice (US-09-01b) A Community of Practice (CoP) is a user-led community of stakeholders, from providers to the final beneficiaries of Earth observation data and information, with a common interest in specific aspects of societal benefits to be realized by GEOSS implementation. Each Community of Practice should include representatives from GEO Members, Participating Organizations, and any other stakeholders that have similar interests, goals, and or objectives. The CoP members should work closely together to create a forum for efficient and effective intelligence and advice to be provided to GEO for the successful implementation of GEOSS. Both developing and developed countries should be represented. 25
Group on Earth Observations Task Sheet Analysis – User Engagement Task Sheet Analysis UIC has a responsibility within GEO to address the user element in all of the tasks in the Work Plan. Some tasks have a significant, inherent user-engagement dimension to them, and some tasks do not (though eventually all might). UIC pursued analysis focused on assessing the quality of the user-engagement component in the task sheets. Analysis looked especially for tasks that an inherent need for user engagement and a high probability of success in engaging users. Analysis sought to identify key needs/ways to improve Task Leads understanding of user engagement 5 major findings and 5 major recommendations 26
Group on Earth Observations Task Sheet Analysis – User Engagement Note: Analysis used the task sheets submitted in December 2008 & March
Group on Earth Observations Task Sheet Analysis – User Engagement User Engagement Aspects of Task Sheets 28
Group on Earth Observations Task Sheet Analysis – User Engagement 29
Group on Earth Observations Task Sheet Analysis – User Engagement User Engagement Aspects of Task Sheets task sheets had some text in the UE section (63 had nothing). Of the 51 task sheets that had text: - 15 task sheets (29%) had substantive, thoughtful statements - 25 sheets (49%) had some level of thought - 11 sheets (22%) had minimal response. Thus, overall, only 13% of all task sheets (15 of 114) gave substantive attention to user engagement in the UE section.
Group on Earth Observations Task Sheet Analysis – User Engagement User Engagement Aspects of Task Sheets 31 Some of the recommendations: - GEO/UIC should develop materials on user engagement Document and publish a range of user engagement techniques and examples as a resource to the GEO community. - GEO/UIC should utilize its User Type Analysis to support task teams to understand, design, and implement user engagement approaches. UIC should publish user types for GEOs use and encourage task teams to employ the user types to identify potential user groups they can engage in their tasks.
32 Group on Earth Observations Users & Evaluation Process Sections of Presentation I.Users II.UIC work with users & user engagement III.Mechanisms to reach users Discussion
Group on Earth Observations III. Mechanisms to Reach Users 33 Multiple Approaches to Reaching Users 1.Go to where the users are Usability testing Key conferences where cross-section of user types or specific user- types of particular interest gather. Surveys and feedback on actual experiences 2.Solicit input and ideasCall for Proposals Request information from broad to specific user types Work with key associations that have established networks 3.Work through credible representativesUS-09-01a Representatives able to consolidate and present community-wide views
Group on Earth Observations III. Mechanisms to Reach Users 34 Multiple Approaches to Reaching Users (cont.) 4.Establish on-going foraCoP Fora for people to gather (physically or virtually) and identify common interests and action to pursue 5.Utilize existing statementsTask Sheet Analysis Gather information from explicit statements of implied opinions and information In addition: Self-assessmentUser Types Review of your approach; perform gap-analyses to identify biases
Group on Earth Observations III. Mechanisms to Reach Users 35 Other Considerations Protecting from bias: Large sample vs random sampling Open up broad surveys to get sheer numbers of responses; however, this opens up to bias by not having random sampling Qualitative vs. Quantitative input UIC has been focused on sheer numbers to begin with, which potentially opens a bias (relies on connecting with most vocal users) UIC is just moving into more quantitative input
36 Group on Earth Observations Task US-09-01a Discussion Lawrence Friedl, UIC Member NASA Earth Science Division Director (Acting), Applied Sciences Program