Presentation on theme: "How Does CRTD Support Software Learning? Criterion Referenced Objectives as Foundation for Evaluation Julianne O’Neil, M.Ed. Siemens Healthcare."— Presentation transcript:
How Does CRTD Support Software Learning? Criterion Referenced Objectives as Foundation for Evaluation Julianne O’Neil, M.Ed. Siemens Healthcare
Goals Recognize criterion-referenced test development (CRTD) integration within overall evaluation framework Describe components of criterion-referenced objectives Write criterion-referenced objectives with those components and consider use of mind mapping Identify possible solutions to challenges implementing performance-based tests
4 Levels of Evaluation Level 1 (Reaction) Learner satisfaction/reaction following training Customer Satisfaction Level 2 (Learning) Level 2 (Learning) Knowledge acquisition, skill development, attitude change Measurement of learning acquisition Level 4 (Results) Level 4 (Results) Organizational results attributed to training Reduced costs, reduced support time, increased profits, improved productivity Level 3 (Behavior) Level 3 (Behavior) Behavior change demonstrated on the job after training “Training Transfer”
CRTD in Evaluation Framework Level 1 (Reaction) Learner satisfaction/reaction following training Customer Satisfaction Level 2 (Learning) Level 2 (Learning) HOW DO YOU KNOW WHAT THEY HAVE LEARNED? Criterion-referenced objectives/tests Level 4 (Results) Level 4 (Results) Organizational results attributed to training Reduced costs, reduced support time, increased profits, improved productivity Level 3 (Behavior) Level 3 (Behavior) Behavior change demonstrated on the job after training “Training Transfer”
Objectives’ Focus on Performance Traditional objectives and tests focus on declarative knowledge Knowledge not guarantee of performance improvement What are learners going to DO on job after training? Want to improve performance to drive organizational success
CRTD in Evaluation Framework Goal of Siemens Healthcare Evaluation Framework: To ultimately evaluate Levels 1 thru 4 Level 2 Evaluation Focus: Identify desired performance Establish/maintain standards for objectives/tests Restructure objectives format criterion-referenced Declarative knowledge performance improvement Level 3 Evaluation Focus: Data collection by written/phone survey Survey based on CRTD objectives
Components of Criterion-Referenced Objectives A – Audience Learner/role B – Behavior Desired performance/skill C – Condition Circumstances under which behavior is performed D – Degree Extent to which the new behavior is to be demonstrated
Template for a CRTD Objective Given, The Will Correctly.
Examples of CRTD Objectives Given an out-of-balance Patient Census in which the new Month-to-Date amount is not equal to the Actual Month-to-Date Total, the financial analyst will write the procedures required to balance the patient census with 100% accuracy. Audience:Financial Analyst Behavior:Write procedures to balance patient census Condition:Out-of-balance patient census Degree:100% Accuracy
Examples of CRTD Objectives The reporting analyst will track the progress of report output when presented with the Output Job Status utility to effectively reprint reports 100% of the time. Audience:Reporting Analyst Behavior:Track progress of report output and print reports Condition:Output Job Status Utility Degree:Effectively reprint 100% of time
Examples of CRTD Objectives Using 3 sample pairs of sterile examination gloves, the nurse will demonstrate the proper technique for donning sterile gloves without tearing or touching the outside of the gloves at least 1 out of 3 tries. Audience:Nurse Behavior:Demonstrate proper technique for donning sterile gloves Condition:Using sample sterile examination gloves Degree:Without tearing or touching the outside of the gloves 1 out of 3 tries
Defining 2 Types of Objectives Topic complexity determines: Depth/breadth of objectives What to test Standard for what is to be tested (Siemens standard: Test terminal objectives) Terminal Objectives: Determine performance desired upon training completion Include at beginning of course material Describe what learners are to demonstrate Include all content to be tested Describe learning “above memory level” Enabling Objectives: Enable achievement of terminal objectives. Define “teachable” content at chapter or modular level Describe prerequisite knowledge/skills needed to satisfy terminal objectives
Siemens-Defined Standards for CRTD CRTD OBJECTIVES: Terminal objectives Must be tested Should be performance-based whenever possible Must be defined using CRTD format Enabling objectives May be written if needed May optionally be tested May optionally be defined using CRTD format Audience, Condition and Degree may be defined once for all objectives in group (vs. stated for each objective) YOUR STANDARDS MAY BE DIFFERENT!
CRTD Objectives Activity 1.Instructional Goal: Select one from list in handout 2.Write 1 terminal objectives in CRTD format to address chosen goal CRTD Template: Given, The Will Correctly.
CRTD Objectives Activity 1.Write 2-3 enabling objectives to support the terminal objective just written 2.NOTE: These enabling objectives do not have to be written in CRTD format.
Siemens-Defined Standards for CRTD CRTD END-OF-COURSE TESTS: Test items Must test terminal objectives Optionally test enabling objectives (depending on complexity) For performance-based objectives, practice opportunities must be provided during class Test item types allowed Performance-based When “traditional” performance-based can’t be used: Multiple-choice Matching Sequencing NO: Fill-in Essay True/False YOUR STANDARDS MAY BE DIFFERENT!
Corresponding Test Items Performance-based Test Item: Can test one or more objectives just by producing an end product/service. SAMPLE TEST ITEM: Download a model report, modify the report, and add it as a custom report: Download report from the Report Catalog. Save report as “LA _ORE_Report_Catalog.rpt,” where is your Student Number. Open report in Crystal and change report header name to “Operational Reports Report Catalog List for.” Add new report to the Report Catalog using the following values: Report Name: “LA-,” where is your Student Number Template: Your LA _ORE_Report_Catalog.rpt Crystal Report Description: Learner Assessment Filter: Create a new filter file using the available report parameters Create a sub-folder under the SCR Class Reports folder with name “Folder_,” where is your student number and add your new report to “Folder_ ” Locate your report output. Create an Export Package called “LA _Assessment_,” where is your student number. Be sure the Export Package contains your report.
Corresponding Test Items Multiple-Choice Item: NOTE: This is also performance-based… 1) The Stored Procedure used by the model Dietary Report is ORE_0100 DIETARY_REPORT_SP ORE_0105 ORE_0030 The Dietary Report does not use a Stored Procedure
Sample Instructional Objectives MindMap
Terminal and Enabling Objectives
Considerations for Performance-Based CRTD Implementation at Siemens Considerations Test writer resistance to change (standards restricting test item types) Manual test scoring for “traditional” performance-based (non multiple choice) Automated scoring for multiple-choice (Questionmark® Perception®) Possible costs to maintain product software/server environments
Benefits of Performance-Based CRTD Implementation to Siemens ….and Possibly to YOU! Benefits Provides consistency of objectives/tests across modules Facilitates cleaner test item construction (based on CRTD objectives) Allows higher fidelity testing (closer to actual job task performance) Permits clearer focus for instructional design Improves measure of learning success at end of training Sets performance measures foundation for post-training assessment