1. Training & Instructional Design Assessment This material (Comp20_Unit6) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003.

2. Assessment Learning Objectives 2 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Identify an instructional problem Plan and implement an instructional needs assessment Analyze learner, task, and situational characteristics By the end of this unit learners will be able to:

3. Evaluation & Learner Assessment in Instructional Design 3 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Analysis Design Develop Implement Evaluate

4. Purpose of Program Evaluation 4 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Decision-making to improve the program Revisions to program goals and objectives Budget requests and resource allocation Progress reports to management Assessment determines the extent to which a program achieves its stated goals and objectives. The results can be used to inform:

5. The Evaluation Plan 5 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment How well are the stated training goals being met? How will the information be collected? How will the data be used to improve the program?

6. The Evaluation Plan (Cont.) 6 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Program goals Intended outcomes Data gathering & interpreting methods Plan for using data as evidence to inform program improvement When and how the evaluation will be conducted The assessment plan should outline:

7. Develop a Logic Model 7 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Inputs Resources Contributions Investments Outputs Activities Services Events Products Outcomes Results or changes for participants

8. Simple Logic Model 8 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Get Pills INPUTS Take Pills OUTPUTS Feel Better OUTCOMES

9. Logic Model: EHR Training Workshop 9 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment

10. Purpose of the Logic Model 10 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Match evaluation to the program Determine what and when to measure Decide on the type of evaluation, process and/or outcomes Determine the testing instruments and procedures Prioritize where to spend evaluation resources The logic model helps us:

11. Assessment & Evaluation Definitions 11 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Process evaluation: determines if activities are delivered to the target population as intended Outcome evaluation: determines if the program successfully produced the desired changes or outcomes Formative evaluation: gathers data to inform program improvement Summative evaluation: makes summary judgments about a program’s performance

12. Writing Assessment Plans 12 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Evaluation questions Collection methods Instruments used for data collection For each evaluation objective, we need to identify:

13. Logic Model: EHR Training Workshop 13 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment

14. Process & Formative Evaluation Questions 14 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Questions What amount of money and time were invested? Were all the sessions delivered? How many persons were trained? Was the intended population trained? How well were the sessions delivered? What was the level of participation? Were participants satisfied?

15. Process Evaluation Plan QuestionsCollection Methods Instruments Amount of money & time ReceiptsProposed budget Timesheets Number of sessions delivered Training curriculum Program requirements Number of persons trained Attendance records Population trainedAttendance records Job descriptions Table 1.1 15 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment

16. Process Evaluation Plan (Cont.) QuestionsCollection Methods Instruments How well were the sessions delivered? End of workshop evaluation Self-report pencil and paper of online survey What was the level of participation? Number & quality questions asked Computer activity Observation rubric Were participants satisfied? End of workshop evaluation Self-report pencil and paper or online survey Table 1.2 16 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment

17. Outcome Evaluation Questions 17 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Questions To what extent did skills improved? To what extent did behaviors improve? Which participants demonstrated a change in skills and behaviors?

18. Outcome/Learner Assessment 18 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Focus on student performance Outline measurable & observable tasks State how you will know that the objective has been reached Learner assessments start with good learning objectives. Create learning objectives that:

19. Outcome/Learner Assessment (Cont.) 19 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment By end of the workshop, participants will be able to recall the steps of the intake process. Can assess this objective with an oral test, or Observe the participants’ performance as they actually use the computer to complete the task But, multiple assessment methods are best to authentically measure student learning

20. Types of Learner Assessments Tests or Quizzes Specified increase in the number of correct responses as compared to a pretest or a specified number of students who are able to attain a targeted level of correct responses. ObservationsSpecific student behaviors, e.g. to accomplish a specific task Student WritingSpecific content knowledge or structural characteristics such as organization, development, or other specified features. OralLevel of participation, specific content knowledge, specific thinking skills, or other indications of student progress as indicated by student comments. Self Assessment Student comments, written statements, visual indicators, or other indications from students regarding their progress towards the learning goals. Table 1.3 20 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment

21. Using Rubrics 21 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment A rubric is a set of criteria used for assessing a piece of work or performance Includes levels of achievement for each criterion There will often be numerical scores for each level of achievement, and a summary score for the may be produced by adding the scores May also include space to describe the reasons for each score

22. Sample Rubric 22 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment The Performance Task requires students to collect and process information, using it for an authentic purpose. (Scarborough, 2006 [Based upon White & Scarborough, 2004]) The task incorporates a variety of information gathering techniques and information resources. Students are required to interpret and synthesize information and accurately assess the value of information gathered. They are required to collect the right information for an authentic purpose, e.g. solve a problem, apply or use in a complex project, etc. The task requires students to gather and synthesize information, but the value of the information gathered is not assessed. Information may not be used for a purpose. The task requires the students to gather information, but not to interpret it. The task requires no gathering or processing of information.

23. Outcome Evaluation Questions 23 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Questions To what extent did skills improved? To what extent did behaviors improve? Which participants demonstrated a change in skills and behaviors?

24. Outcome Evaluation/Learner Assessment Plan QuestionsCollection MethodsInstruments To what extent did knowledge of the system improved? Written quizzes Participation in discussions Test rubrics Discussion protocol To what extent did skills & behaviors improve? Performance task assessed in workshop, and later on the job Performance task rubric Which participants demonstrated a change in skills and behaviors? Compile performance tasks results and group by job description and department Personnel records Performance task rubric Table 1.4 24 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment

25. How to Use Evaluation Results 25 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Results can open up a discussion about a program’s strengths and weaknesses and inform changes to improve its effectiveness Compare program objectives with the outcome data, and determine the degree to which each objective was met or not met Understanding the reasons why a program succeeded or failed can lead to recommendations

26. Communicating Evaluation Results 26 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Evaluation report should outline: The objectives and outcomes How data was collected and identify sources (e.g. participants) Barriers Recommendations

27. Assessment Summary 27 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment Decision-making to improve the program Revisions to program goals and objectives Budget requests and resource allocation Progress reports to management Program Evaluation & Learner Assessment

28. Assessment References References 1.Clark, D.R. (2004). Bloom’s Taxonomy [monograph on the Internet]. Big Dog & Little Dog’s Performance Juxtaposition. Retrieved on June 21st,2010 from: http://www.nwlink.com/~donclark/hrd/sat.html.http://www.nwlink.com/~donclark/hrd/sat.html 2.Clark, D.R. Instructional System Design (ISD). (2004). Big Dog & Little Dog’s Performance Juxtaposition. Retrieved on June 21st, 2010 from: http://nwlink.com/~donclark/hrd/ahold/isd.html.http://nwlink.com/~donclark/hrd/ahold/isd.html 3.Carkhuff, R.R., & Fisher, S.G. (1984). Instructional systems design: volumes I & II. Amherst, MA: Human Resource Development Press. 4.Carliner, S. Training Design. (2003).Danvers, MA: American Society for Training and Development. 5.Chatterji, M. (2003). Designing and Using Tools for Educational Assessment. Massachusetts: Allyn & Bacon 6.Gagne, R.M., Wager, W.W., & Golas, K. (2004). Principles of Instructional Design (5th ed.). California: Wadsworth Publishing. 7.Reigeluth, C.M. Instructional-design Theories and Models: A new paradigm of instructional theory. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. Charts, Tables & Figures 1.1 & 1.2 Table: Zimmerman, J. (2010). Process evaluation. Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. 1.3 Table: Zimmerman, J. (2010). Types of learner assessments. Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. 1.4 Table: Zimmerman, J. (2010). Outcome evaluation/learner assessment plan. Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. 28 Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Assessment