1. Program Evaluation Tools and Strategies for Instructional Technology

2. 2 2 zjsun@sun-associates.com y978-251-1600 ext. 204 zomunist@sun-associates.com zwww.sun-associates.com/necc2006www.sun-associates.com/necc2006 yThis presentation will be linked to that site

3. 3 3 Where Do We Stand? zWho’s working on an actual project? yCurrent? yAnticipated? zYour expectations for today

4. 4 4 Workshop Goals zTo review the key elements of effective program evaluation as applied to instructional technology evaluations zTo consider evaluation in the context of your actual projects (via example projects)

5. 5 5 Why Evaluate? zTo fulfill program requirements yNCLB and hence Title IID carry evaluation requirements yMost other state and federal proposals require an evaluation component xAnd not simply a statement that “we will evaluate” xBut actual info on who will evaluate, the evaluation questions, and methodologies zProject sustainability zGeneration of new and improved project ideas zOthers?

6. 6 6 By Definition, Evaluation… zIs both formative and summative zHelps clarify project goals, processes, products zShould be tied to indicators of success written for your project’s goals zIs not a “test” or simply a checklist of completed activities zQualitatively, are you achieving your goals? zWhat adjustments can be made to your project to realize greater success?

7. 7 7 A Three-Phase Evaluation Process zEvaluation Questions yTied to original project goals yIndicator rubrics yAllow for authentic, qualitative, and holistic evaluation zData Collection yTied to indicators in the rubrics zScoring and Reporting yRole of the evaluation committee pg 5 in workbook

8. 8 8 Who Evaluates? zCommittee of stakeholders (pg 10) zOutside facilitator? zTask checklist (pg 6) zOther issues… yPerspective yTime-intensive

9. 9 9 Project Sample

10. 10 An Iterative Process zEvaluation breaks your vision down into increasingly observable and measurable pieces.

11. 11 Goals Lead to Questions zWhat do you want to see happen? yThese are your goals yRephrase goals into questions zAchieving these goals requires a process that can be measured through a formative evaluation

12. 12 …And Then to Indicators zWhat is it that you want to measure? yWhat are the conditions of success and to what degree are those conditions being met? yBy what criteria should performance be judged? yWhere should we look and what should we look for to judge performance success? yWhat does the range in the quality of performance look like? yHow should different levels of quality be described and distinguished from each other?

13. 13 zIndicators should reflect your project’s unique goals and aspirations yRooted in proposed work yIndicators must reflect your own environment...what constitutes success for you might not for someone else yIndicators need to be highly descriptive and can include both qualitative and quantitative measures zYou collect data on your indicators

14. 14 Try it on a Sample zUsing the Evaluation Logic Map, map your: yProject purpose/vision yGoals yObjectives yActions zWe’ll take 20 minutes for this…and then come back for indicators

15. 15 1.Logic mapping the sample - 20 min 2.Writing questions and indicators - 30 min 3.Discuss with partner team - 20 min 4.Debrief with everyone - 20 min

16. 16 To Summarize... zStart with your proposal or technology plan zLogic map the connections between actions, objectives, and goals zFrom your goals/objectives, develop evaluation questions zQuestions lead to indicators yIndicators are organized into rubrics zData collection flows from that rubric

17. 17 Evidence/Data Collection zClassroom observation, interviews, and work- product review yWhat are teachers doing on a day-to-day basis to address student needs? zFocus groups and surveys yMeasuring teacher satisfaction zTriangulation with data from administrators and staffTriangulation yDo other groups confirm that teachers are being served?

18. 18 Data Collection Basics zReview Existing Data yCurrent technology plan yCurriculum yDistrict/school improvement plans zwww.sun-associates.com/eval/samplewww.sun-associates.com/eval/sample zCreate a checklist for data collection

19. 19 Surveys zCreating good surveys yLength yDifferentiation (teachers, staff, parents, community, etc..) yQuantitative data yAttitudinal data yTiming/response rates (getting returns!) zwww.sun-associates.com/eval/samples/samplesurv.htmlwww.sun-associates.com/eval/samples/samplesurv.html

20. 20 Online Survey Tools zOnline yVIVEDVIVED yProfilerProfiler yLoTiLoTi yZoomerangZoomerang ySurveyMonkey.comSurveyMonkey.com

21. 21 Survey Issues  Online surveys produce high response rates  Easy to report and analyze data  Potential for abuse  Depends on access to connectivity

22. 22 Focus Groups/Interviews zFocus Groups/Interviews yTeachers yParents yStudents yAdministrators yOther stakeholders

23. 23 Classroom Observations zUsing an observation template zUsing outside observers

24. 24 Other Data Elements? zArtifact analysis yA rubric for analyzing teacher and student work? zSolicitation of teacher/parent/student stories yThis is a way to gather truly qualitative data yWhat does the community say about the use and impact of technology?

25. 25 Dissemination zCompile the report zDetermine how to share the report ySchool committee presentation yPress releases yCommunity meetings

26. 26 Conclusion zBuild evaluation into your technology planning effort zRemember, not all evaluation is quantitative zYou cannot evaluate what you are not looking for, so it’s important to — zDevelop expectations of what constitutes good technology integration

27. 27 More Information zjsun@sun-associates.comjsun@sun-associates.com y978-251-1600 ext. 204 zomunist@sun-associates.com zwww.sun-associates.com/evaluationwww.sun-associates.com/evaluation zwww.edtechevaluation.comwww.edtechevaluation.com