1. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 18 July 2003 The Workforce Learning Community Bill Bewley and Roxanne Sylvester UCLA / CRESST The ONR Workplace Education Project  2003 Regents of the University of California

2. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 1 Funded by an ONR grant to UCLA CRESST –“Research-Based Distance Learning to Support Navy Workplace Education” The Workplace Education Project

3. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 2 The goal –Create a framework for application of guidelines to courseware evaluation by real users The process –Expert review –Select guidelines, recast as questions –Test, revise, test—an iterative process –Base technical quality judgment on instrument and rater reliability –Report and deliver Guidelines Framework

4. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 3 Web-based instruction context –Accuracy of course content, instructional design, operations, format, technical standards Other guidelines –FSU “Evaluating the Course”: 27 categories, 119 questions –ASTD “Courseware Certification”: 19 categories, 78 questions –USMC “Interactive Multimedia Instruction Style Guide”: 12 categories, 50 pages Guidelines Framework

5. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 4 The challenges –Balancing technical quality with usability –Creating sufficient user support to ensure rater reliability while not overburdening raters with information Guidelines Framework

6. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 5 KMT Guidelines: Value Added –Based on rigorous, up-to-date research –Examination of technical quality of instrument; it may be possible to evaluate courseware well with fewer questions –Examination of rater reliability –Examination of scoring training issues related to calibration of rater scores Guidelines Framework

7. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 6 How technical quality is established –Representative sample population, various course types; variability of raters and courses –Qualitative data on rater interactions with framework and course components –Quantitative data on instrument reliability and rater reliability –Development of rater scoring training materials Guidelines Framework

8. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 7 Initial work on KMT guidelines –Reviewed guidelines in relation to formative evaluation of course components Originally 58 guidelines, reduced to 52 –Selected guidelines supporting formative evaluation of course components From 52 to 14 guidelines –Turned guidelines into questions From 14 guidelines to 45 questions –Applied guidelines questions to three courses, reviewed results, revised guidelines selections and questions Guidelines Framework

9. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 8 Usability Study 1: Questions Courseware Evaluation Questions Usability Study –Videotaped think-aloud –Review of Systems Thinking lesson –Use of paper-based Guidelines questions –Graduate students and temporary workers –August-December 2003 –Revision of questions; addition of definitions & examples to enhance rater agreement Guidelines Framework

10. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 9 Usability Study 2: Prototype E-Tool Prototype E-Tool Usability Study –Think-aloud screen and audio capture –Examine e-tool functionality –Examine interplay between e-tool and courseware –Begin to examine rater reliability –Graduate students, instructional designers, SMEs –January to April 2004 –Determine all features required –Use results to program final e-tool Guidelines Framework

11. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 10 Usability Study 3: Final E-Tool Final E-Tool Usability Study –Think-aloud screen and audio capture –Refine e-tool –Examine rater reliability –Develop rater training procedures –Graduate students, instructional designers, SMEs –June to September 2004 –Deliver October 2004 Guidelines Framework

12. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 11 Continue formative evaluation –Evaluate additional WLC courses –Continue research on validity of framework and rater reliability –Disseminate framework for review –Develop worked examples –Develop rater scoring training materials –Conduct rater and trainer training 04/05 Tasks

13. CRESST ONR/NETC Meetings, 17-18 July 2003, v1 12 Bill Bewley –310-825-7995 –Bewley@ucla.edu Roxanne Sylvester –310-206-5672 –Sylvester@cse.ucla.edu For More Information