1. Comp 20 - Training & Instructional Design Unit 6 - Assessment This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003.

2. Learning Objectives Identify an instructional problem Plan and implement an instructional needs assessment Analyze learner, task, and situational characteristics By the end of this unit learners will be able to: Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 2

3. Evaluation & Learner Assessment in Instructional Design Analysis Design Develop Implement Evaluate Component 20/Unit 63 Health IT Workforce Curriculum Version 2.0/Spring 2011

4. Purpose of Program Evaluation Decision-making to improve the program Revisions to program goals and objectives Budget requests and resource allocation Progress reports to management Assessment determines the extent to which a program achieves its stated goals and objectives. The results can be used to inform: Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 4

5. The Evaluation Plan How well are the stated training goals being met? How will the information be collected? How will the data be used to improve the program? Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 5

6. The Evaluation Plan Cont’d Program goals Intended outcomes Data gathering & interpreting methods Plan for using data as evidence to inform program improvement When and how the evaluation will be conducted The assessment plan should outline: Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 6

7. Develop a Logic Model Inputs Resources Contributions Investments Outputs Activities Services Events Products Outcomes Results or changes for participants Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 7

8. Simple Logic Model Get Pills INPUTS Take Pills OUTPUTS Feel Better OUTCOMES Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 8

9. Logic Model: EHR Training Workshop INPUTS What we invest TrainersFunds Computers, software Training curriculum OUTPUTS What we do Lecture Individual & group work Program Evaluation OUTPUTS Who we reach ParticipantsPatientsManagement OUTCOMES Short-term to long- term Increase knowledge of software Increase office efficiency Increase heath outcomes Component 20/Unit 69 Health IT Workforce Curriculum Version 2.0/Spring 2011

10. Purpose of the Logic Model Match evaluation to the program Determine what and when to measure Decide on the type of evaluation, process and/or outcomes Determine the testing instruments and procedures Prioritize where to spend evaluation resources The logic model helps us: Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 10

11. Assessment & Evaluation Definitions Process evaluation: determines if activities are delivered to the target population as intended Outcome evaluation: determines if the program successfully produced the desired changes or outcomes Formative evaluation: gathers data to inform program improvement Summative evaluation: makes summary judgments about a program’s performance Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 11

12. Writing Assessment Plans Evaluation questions Collection methods Instruments used for data collection For each evaluation objective, we need to identify: Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 12

13. Logic Model: EHR Training Workshop Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 13 INPUTS What we invest TrainersFunds Computers, software Training curriculum OUTPUTS What we do Lecture Individual & group work Program Evaluation OUTPUTS Who we reach ParticipantsPatientsManagement OUTCOMES Short-term to long- term Increase knowledge of software Increase office efficiency Increase heath outcomes

14. Process & Formative Evaluation Questions Questions What amount of money and time were invested? Were all the sessions delivered? How many persons were trained? Were the intended population trained? What was the level of participation? How well were the sessions delivered? Were participants satisfied? Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 14

15. Process Evaluation Plan QuestionsCollection Methods Instruments Amount of money & time ReceiptsProposed budget Timesheets Number of sessions delivered Training curriculum Program requirements Number of persons trained Attendance records Population trainedAttendance records Job descriptions Component 20/Unit 615 Health IT Workforce Curriculum Version 2.0/Spring 2011

16. Process Evaluation Plan Cont’d QuestionsCollection Methods Instruments How well were the sessions delivered? End of workshop evaluation Self-report pencil and paper of online survey What was the level of participation? Number & quality questions asked Computer activity Observation rubric Were participants satisfied? End of workshop evaluation Self-report pencil and paper or online survey Component 20/Unit 616 Health IT Workforce Curriculum Version 2.0/Spring 2011

17. Outcome Evaluation Questions Questions To what extent did skills improved? To what extent did behaviors improve? Which participants demonstrated a change in skills and behaviors? Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 17

18. Outcome/Learner Assessment Focus on student performance Outline measurable & observable tasks State how you will know that the objective has been reached Learner assessments start with good learning objectives. Create learning objectives that: Component 20/Unit 618 Health IT Workforce Curriculum Version 2.0/Spring 2011

19. Outcome/Learner Assessment Cont’d By end of the workshop, participants will be able to recall the steps of the intake process. Can assess this objective with an oral test, or Observe the participants’ performance as they actually use the computer to complete the task But, multiple assessment methods are best to authentically measure student learning Component 20/Unit 619 Health IT Workforce Curriculum Version 2.0/Spring 2011

20. Types of Learner Assessments Tests or Quizzes Specified increase in the number of correct responses as compared to a pretest or a specified number of students who are able to attain a targeted level of correct responses. ObservationsSpecific student behaviors, e.g. to accomplish a specific task Student WritingSpecific content knowledge or structural characteristics such as organization, development, or other specified features. OralLevel of participation, specific content knowledge, specific thinking skills, or other indications of student progress as indicated by student comments. Self Assessment Student comments, written statements, visual indicators, or other indications from students regarding their progress towards the learning goals. Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 20

21. Using Rubrics A rubric is a set of criteria used for assessing a piece of work or performance Includes levels of achievement for each criterion There will often be numerical scores for each level of achievement, and a summary score for the may be produced by adding the scores May also include space to describe the reasons for each score Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 21

22. Sample Rubric The Performance Task requires students to collect and process information, using it for an authentic purpose. (Scarborough, 2006 [Based upon White & Scarborough, 2004]) The task incorporates a variety of information gathering techniques and information resources. Students are required to interpret and synthesize information and accurately assess the value of information gathered. They are required to collect the right information for an authentic purpose, e.g. solve a problem, apply or use in a complex project, etc. The task requires students to gather and synthesize information, but the value of the information gathered is not assessed. Information may not be used for a purpose. The task requires the students to gather information, but not to interpret it. The task requires no gathering or processing of information. Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 22

23. Outcome Evaluation Questions Questions To what extent did skills improved? To what extent did behaviors improve? Which participants demonstrated a change in skills and behaviors? Component 20/Unit 623 Health IT Workforce Curriculum Version 2.0/Spring 2011

24. Outcome Evaluation/Learner Assessment Plan QuestionsCollection MethodsInstruments To what extent did knowledge of the system improved? Written quizzes Participation in discussions Test rubrics Discussion protocol To what extent did skills & behaviors improve? Performance task assessed in workshop, and later on the job Performance task rubric Which participants demonstrated a change in skills and behaviors? Compile performance tasks results and group by job description and department Personnel records Performance task rubric Component 20/Unit 624 Health IT Workforce Curriculum Version 2.0/Spring 2011

25. How to Use Evaluation Results Results can open up a discussion about a program’s strengths and weaknesses and inform changes to improve its effectiveness Compare program objectives with the outcome data, and determine the degree to which each objective was met or not met Understanding the reasons why a program succeeded or failed can lead to recommendations Component 20/Unit 625 Health IT Workforce Curriculum Version 2.0/Spring 2011

26. Communicating Evaluation Results Evaluation report should outline: The objectives and outcomes How data was collected and identify sources (e.g. participants) Barriers Recommendations Component 20/Unit 6 Health IT Workforce Curriculum Version 2.0/Spring 2011 26

27. Decision-making to improve the program Revisions to program goals and objectives Budget requests and resource allocation Progress reports to management Program Evaluation & Learner Assessment Summary Component 20/Unit 627 Health IT Workforce Curriculum Version 2.0/Spring 2011