Measuring and reporting outcomes for BTOP grants: the UW iSchool approach Samantha Becker Research Project Manager U.S. IMPACT Study 1UW iSchool evaluation framework

Communities Connect Network Study of Community Technology in Washington State In 2007, UW conducted a phone survey across 211 agencies in the state identified as community technology providers. 47of these agencies completed the survey, and 7 sites were visited for more in-depth study. This was combined with earlier data to provide a snapshot of CT in Washington State 2UW iSchool evaluation framework

Findings from the CCN survey indicated that CT was having an impact on communities. Three levels of benefits were identified – Individual – Family – Community Six domains were found to be important – Employment/economic – Academic skills and literacy – Social inclusion and personal growth – Independence – Access to information and resources – Communication From 2007 CCN study 3UW iSchool evaluation framework

This lead to the development of the community technology impact analysis framework Context Analysis Situated Logic Model Outcome Measurement Validation and Reflective Practice Used to frame the Washington Community Technology Opportunity Program goals Six domains were found to be important – Employment/economic – Academic skills and literacy – Social inclusion and personal growth – Independence – Access to information and resources – Communication – A similar process was used to frame the current WA BTOP evaluation framework 4UW iSchool evaluation framework

Identify major policy issues confronting the PCC’s community, such as: Workforce development Education Poverty Civic engagement And link them to activities used to address these issues, such as: Access to technology and information Skills building Development of local content Step 1: Policy issue mapping Context Analysis 5UW iSchool evaluation framework

Identify other stakeholders concerned with the policy goals – individuals – groups – organizations – institutions Collect data about what they do and how they experience PCC services Include stakeholders who are working to achieve similar policy goals or who are affected by the PCC Step 2: Stakeholder analysis Context Analysis 6UW iSchool evaluation framework

Step 3: Develop a policy or stakeholder logic model Situated Logic Model Inputs Facilities Technology Knowledge Relationships Activities Technology access Training and support Awareness building Outcomes Effective and efficient service delivery Improved decision making Gain employment Impacts Improved workforce Reduction of poverty Improved community health Policy Issue: Workforce Development 7UW iSchool evaluation framework

Step 4: Develop a PCC Logic Model to link goals with measurable indicators Situated Logic Model Inputs Facilities Technology Knowledge Relationships Activities Technology access Training and support Awareness building Outputs Hours of access to technology Number of clients participate in training Impacts Clients get jobs Clients earn GEDs Clients are able to use technology independently Clients engage in their communities 8UW iSchool evaluation framework

Sidebar: what’s a measurable indicator? Indicators need to meet certain utilitarian standards. Beyond the actual content of the indicator, they should also be: specific, unique and unambiguous; observable, practical, cost effective to collect, and measurable; understandable and comprehensible; relevant (measures important dimensions, appropriate and related to the program, that are of significance, predictive and timely); time bound; and valid, providing reliable, accurate, unbiased, consistent, and verifiable data (Hatry, 2006) UW iSchool evaluation framework9

Step 4: Bridge the logic models to show how the CTC supports larger policy goals Situated Logic Model Inputs Facilities Computers Internet connection Software Staff Activities Open technology access Computer classes Tutoring Outputs Hours of access to technology Number of clients participate in training Number of clients looking for jobs Impacts Clients get jobs Clients earn GEDs Clients are able to use technology independently Clients engage in their communities Inputs Facilities Technology Knowledge Relationships Activities Technology access Training and support Awareness building Outputs Citizen technology access Increased citizen knowledge Use of technology to support social services Impacts Improved workforce Reduction of poverty Improved community health Example: Workforce Development  Community technology 1.Workforce development clients use community technology to look for and apply for jobs 2.Clients get jobs 3.Workforce is improved 1 23 Workforce Development Community Technology 10UW iSchool evaluation framework

Identify the most important outcomes to measure: Tie activities to immediate policy goals in the PCC community Link to the larger community policy context to evaluate overall impact on stakeholders and the community Step 6: Measure outcomes Outcome Measurement Step 7: Report outcomes Report outcome measures in the context of the situated logic model to show contribution to community policy goals. 11UW iSchool evaluation framework

Re-examine relationship between outcomes and policy issues: Validate measures Challenge assumptions Interview stakeholders Step 8: Validate Outcome Measures Validation and Reflective Practice Step 9: Reflect on performance Use outcome/impact measures to inform your work: Establish the value of your work Improve effectiveness Understand your organization’s contribution to the community you service 12UW iSchool evaluation framework

Measuring and reporting outcomes for WA BTOP 1UW iSchool evaluation framework

WA BTOP’s reporting system relates to a community technology logic model InputsActivitiesOutputsImpacts SRs are asked to keep track of and report inputs, activities, outputs, and impacts on a quarterly basis for the duration of the grant in order to: Show the value of the services offered Show the changes in use as a result of the grant Evaluate the impact of the grant on the outcomes of clients 2 Measuring and Reporting Outcomes UW iSchool evaluation framework

Impact types for WA BTOP were informed by the CCN study, BTOP policy goals, and PCC self-identified anticipated outcomes Computer skills enhancement Help clients gain computer and Internet skills to enable them to independently use technology Can be at any skills level and includes multimedia training Employment skills and opportunities Help clients learn how to use computers and the Internet to strengthen their ability to become employed Provide access to employment opportunities Education enhancement Provide supplemental learning support for students challenged by limited learning resources Access to information and services Enable clients to use digitally accessible information for personal needs like managing health problems Enable clients to access government information, benefits, or services Life skills and social inclusion Includes special services for people with disabilities Help clients learn to use technology for managing personal relations Help clients learn about and use technology for money management, finding support, interacting with the community, and daily living skills (e.g. getting bus schedules) Teach and empower clients to use digital technology to express themselves and participate in their communities 4UW iSchool evaluation framework

SRs are not expected to have outcomes to report for every category or indicator. SRs were asked to review the list of possible outcomes and choose those they wish to track. No single client survey can accommodate all PCCs, but samples and advice are provided for SRs for designing instruments and tracking sheets. Tracking outcomes 4UW iSchool evaluation framework