Presentation on theme: "Measuring Outcomes Geoffrey T. Miller"— Presentation transcript:
1 Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum DevelopmentDivision of Pehospital and Emergency HealthcareGordon Center for Research in Medical EducationUniversity of Miami Miller School of Medicine
2 Session aimsDiscuss the importance of outcomes evaluation and challenges to traditional assessmentsDiscuss the importance of validity, reliability and feasibility as it relates to assessmentDiscuss types of assessments and their application in healthcare education
3 A little terminology…Assessment and evaluation are often used interchangeablyHowever for our purposes…Assessment = learner outcomesEvaluation = course/program outcomes
5 Because… assessment: “Drives learning” Allows measures of individual and programmatic progressFundamental to outcomes- or competency-based educationAssures public that providers are competentCredentialing, privileging, licensure, board certification – high stakes for practitioner and patient/societyAll involve assessment of competence
6 Formula for the effective use of simulation TrainingResourcesTrainedEducatorsCurricularInstitutionalizationEffectiveSimulation-basedHealthcareEducationXX=Issenberg, SB. The Scope of Simulation-based Healthcare Education. Simulation in Healthcare
7 Formula for effective outcomes measurement DefinedOutcomesInstruments& TrainedEvaluatorsAppropriateSimulatorEffectiveOutcomesMeasurementXX=
8 What are some challenges to traditional methods of assessment for healthcare providers?
9 Challenges in traditional assessments Ethical issues: “using” real pts (substitutes)Invasive procedures (patient safety)Sensitive tasks (cultural concerns, pt modesty)Problems using cadaveric tissue modelsAnimal welfare issues
10 Challenges in traditional assessments Real patients for evaluation of physical exam skillsFeasibility issues for large-scale examinationsStandardized, perceived fairness issues in high-stakes settingsStandardized patients (SPs) improve reliability, but validity issues exist: cannot mimic many physical findings
11 Challenges in traditional assessments Wide range of clinical problems, including rare and critical eventsAvailabilityCostReliability, validity, feasibility
13 “Any road will get you there, when you don’t know where you are going”
14 Curricula development AnalysisDefine expected outcomesDesignDevelopmentImplementationEvaluation
15 Defining outcomesLearners are more likely to achieve competency and mastery of skills if the outcomes are well defined and appropriate for the level of skill trainingDefine clear benchmarks for learners to achievePlain goals with tangible, measurable objectivesStart with the end-goal in mind and the assessment metrics, then the content will begin to develop itself
16 Curricula/assessment process Teaching andLearningCurricularDevelopmentDefineOutcomes+/- RefinementAssessment andEvaluation
17 Use of assessments in healthcare simulation InformationDemonstrationPracticeFeedbackRemediationMeasurementDiagnosisRosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare.Simulation in Healthcare 3:2008;33–41.
18 Preparing assessments What should be assessed?Every aspect of curriculum considered essential and/or has significant designated teaching timeShould be consistent with learning outcomes that are established as the competencies students should master/perform at a given phase of study
19 Blueprinting X 5, 23 6, 19, 20 Global Objective Recognize a potential terrorist incident and initiate incident operationsUM-ERTModule Obj.2.3Recognize and describe scene hazards and appropriate personal protective measuresFloridaObjective(s)Tier 1: I (L), III (D), (F), (N), IV (J), V (A), (D), VI (B)LearningOpportunityLectureTabletopVideoExerciseSkillOSCEXAssessmentPreMCQPost5, 236, 19, 20
20 Clinical competence and performance “Competent performance” = requires acquisition of basic knowledge, skills & attitudesCompetence =Application of specific KSAsPerformance =“Translation of competence into action”“Can they do it? Do they do it?”
21 Possible outcome competencies Patient careMedical knowledgePractice-based learning and improvementInterpersonal and communication skillsProfessionalismSystems-Based PracticeKnowledgeSkillsAttitudes
29 Assessment of “Knows”DoesShowsKnows HowKnowsFactual Tests
30 The Tools of “Knows” Multiple Choice Questions (MCQs) Short Answer True / FalseMatching (extended)Constructed Response Questions
31 Information Input (Facts) Factual Output (Answers) Example - MCQFACT“Wheezes are continuous, musical, whistling sounds during difficult breathing such as in asthma, croup and other respiratory disorders.”Learning OpportunityInformation Input (Facts)Factual Output (Answers)AssessmentQ. Whistling sounds associated with an asthmatic patient are called?Rales B. RhonchiC. Wheezes D. VesicularANSWER
32 Click on picture to play video Computer-based modelChoose the best description of the patient’s finding:A. MyoclonusB. Partial SeizureC. TicD. FasciculationsE. TremorClick on picture to play video
34 Assessment of “Knows how” DoesShowsKnows HowKnowsClinical ContextBased Tests
35 The Tools of “Knows How” Multiple-choice questionEssayShort answerOral interview
36 Example – Clinical Context MCQ Which of the following is most likely the patients problem?A. MigraineB. Myasthenia gravisC. Multiple SclerosisD. Ischemic StrokeE. Cerebral aneurysm64-year-old manNo past medical Hx1 week of intermittentHeadacheDouble visionR pupil dilated
38 Assessment of “Shows” Performance Assessment Does Shows Knows How
39 The Tools of “Shows” Objective Structured Clinical Examination (OSCE) Standardized Patient-based
40 Variables in Clinical Assessment ExaminerStudentClinicalAssessmentPatientControl as many variables as possible
41 Teaching and Learning - “Does” ShowsKnows HowKnowsLearningOpportunityExperience
42 Assessment of “Does”PerformanceAssessmentDoesShowsKnows HowKnows
43 The Tools of “Does”Undercover / Stealth / Incognito Standardized Patient-basedVideoPortfolioService ratings (customer satisfaction)
44 Influences on clinical performance DoesSystem relatedIndividual relatedCompetenceCambridge Model for delineating performance and competenceRethans JJ, et al. The relationship between competence and performance: implications for assessing practice performance, Medical Education, 36:
45 Assessments types Choose the appropriate assessment method: Formative SummativeSelfPeer
46 Assessment Formative Assessment Summative Assessment Lower stakes One of several, over time of course or programMay be evaluative, diagnostic, or prescriptiveOften results in remediation or progression to next levelSummative AssessmentHigher stakesGenerally final of course or programPrimary purpose is performance measurementOften results in a “Go, No-Go” outcome
48 Assessments - selfEncourages responsibility for the learning process, fosters skills in making judgments as to whether work is of an acceptable standard – it improves performance.Most forms of assessment can be adapted to a self-assessment format (MCQs, OSCEs, and short answers)Students must be aware of standards required for competent performance.
50 Assessments - peerEnables learners to hone their skills in their ability to work with others and professional insightEnables faculty to obtain a view of students they do not seeAn important part of peer assessment is for students to justify the marks they award to othersJustification can also be used as a component when faculty evaluates attitudes and professionalism.
51 Assessments - standard setting Should be set to determine competenceEnables certification to be documented, accountable and defensibleAppropriately set standards for an assessment will pass those students who are truly competentStandards should not be two low (false positives) to pass those who are incompetent, nor too high (false negative) to fail those who are competent.
52 Assessments - standard setting Those responsible in setting standards must also have a direct role in teaching students at the level being examined and assist in providing examination material
53 Assessments - standard setting Standards should be set around a core curriculum that includes the knowledge, skills and attitudes required of all studentsWhen setting a standard the following should be considered:What is assessed must reflect the core curriculumStudents should be expected to reach a high standard in the core components of the curriculum (For instance an 80-90% pass mark of for the important core and 60-80% for the less important aspects.)Students should be required to demonstrate mastery of the core in one phase of the curriculum before moving on to the next part of the curriculum
54 Choosing appropriate assessment methods When choosing the assessment instrument, the following should be answered:Is it validIs it reliableIs it feasible
55 Assessments - validity Are we measuring what we are supposed to be measuringUse the appropriate instrument for the knowledge, skill, or attitude you are testingThe major types of validity should be considered (content, predictive, and face)
56 Assessments - reliability Does the test consistently measure what it is supposed to be measuringTypes of reliability:Inter-rater (consistency over raters)Test-retest (consistency over time)Internal consistency (over different items/forms)
57 Reliability as Consistency Archer 1 hits bulls eye every time.Archer 2 hits outer ring in same spot every time.NB: same analogy used in validity sectionBoth archers are equally reliable.
58 Inter-rater Reliability Multiple judges code independently using the same criteriaReliability = raters code same observations into same classificationExamplesMedical record reviewsClinical skillsOral examinations
59 Factors Influencing Reliability Test lengthLonger tests give more reliable scoresGroup homogeneityThe more heterogeneous the group, the higher the reliabilityObjectivity of scoringThe more objective the scoring, the higher the reliability
60 Validity is accuracy Both archers are equally reliable Archer 1 hits bulls eye every timeArcher 2 hits outer ring in same spot every timeThe problem in educational measurement is that we can seldom “see” the target, e.g., attitudes, knowledge, professionalism as targets are “invisible targets” and validity claims are based on inferences and extrapolations from indirect evidence that we have hit the target.Both archers are equally reliableValidity = quality of archer’s hits
61 Reliability and Validity Reliable and ValidReliable, not validNot reliable, not valid
62 Improving reliability and validity Base assessment on outcome/objectives- event triggers- observable behavior- behavioral rating-assess against competenceDefine:Low-medium-high performanceUse of rubric or rating metricUse (video) training examples of performanceEmploy quality assurance/improvement system
63 Assessments - feasibility Is the administration and taking of the assessment instrument feasible in terms of time and resourcesThe following questions should be considered:How long will it take to construct the instrument?How much time will be involved with the scoring process?Will it be relatively easy to interpret the scores and produce the results?Is it practical in terms of organization?Can quality feedback result from the instrument?Will the instrument indicate to the students the important elements within the course?Will the assessment have a beneficial effect in terms of student motivation, good study habits and positive career aspirations?
64 Practicality Number of students to be assessed Time available for the assessmentNumber of staff availableResources/equipment availableSpecial accommodations
66 Assessments - instruments Be aware of the types of assessment instruments available as well as the advantages and disadvantages of eachIt is important, if feasible, to use more than one assessment instrument and more than one assessor when looking at skills and attitudes
70 Assessment Metrics Procedural or Check List assessment Global Rating assessment
71 Assessment Metrics Procedural or Check List assessment BCLS Y N Open AirwayCheck BreathingBCLSYNOpen Airway (< 5 sec of LOC)Check Breathing (< 5 sec of Airway)BCLSYNOpen AirwayCheck BreathingARating Score+1-1*Assist
72 Assessment Metrics Global Rating assessment Code Blue P F CPR and ACLS CPR <1(low) - 5(Hi)> pointsACLS <1(low)- 5(Hi)> pointsPts.Code BlueHMCPRACLSLRating Score+1-1
73 Review Assessment drives learning Clearly define the desired outcome, ensure that it can be measuredConsider the “threats” to the effectiveness of the measurementFeedback to individual candidatesFeedback to training programs