USEFUL LESSONS FROM ASSESSMENT IN AN ADULT LEARNING ENVIRONMENT: INSTRUCTIONAL STRATEGIES IN DISTANCE LEARNING Harry O’Neil University of Southern California & National Center for Research on Evaluation, Standards, and Student Testing (CRESST) CRESST Conf 07 v.2 CRESST Conference 2007 Los Angeles, CA January 22, 2007

C R E S S T / U S C Page 2 CRESST Conf 07 v.2 Instructional Design Issues in Distance Learning Models of Learning Analytic Approaches to Assessment Empirical Approaches to Assessment

C R E S S T / U S C Page 3 CRESST Conf 07 v.2 CRESST Model of Cognitive Learning Content Understanding Learning Communication Collaboration Problem Solving Self-Regulation

C R E S S T / U S C Page 4 CRESST Conf 07 v.2 Affective/Motivation Model of Learning Play Affective Learning Effort Test Anxiety Academic Goals Self-Efficacy

C R E S S T / U S C Page 5 CRESST Conf 07 v.2 Social Capital Trust Collective Efficacy Networks Collective Effort Transparency Learning Organization Teamwork Skills

C R E S S T / U S C Page 6 CRESST Conf 07 v.2 Analytic Approach: Check Validity of Instructional Strategy in e-Learning Often inductive discovery approach —Usually doesn’t result in learning [Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based learning. Educational Psychologist, 41, 75-86] What Works in Distance Learning –Good instructional practices that can be applied to e- learning

C R E S S T / U S C Page 7 CRESST Conf 07 v.2 What Works in Distance Learning * Develop a core set of research-based DL Guidelines –Instructional Design (Clark) –Multimedia (Mayer) –Learning Strategies (Dembo) –Assessment (Baker) –Management Strategies (Kazlauskas) –Self-Regulation and Motivation (O’Neil) Lessons based on guidelines * O’Neil, H. F. (Ed.). (2005). What works in distance learning: Guidelines. Greenwich, CT: Information Age Publishing Inc. *O’Neil, H. F. (Ed.). (2003). What works in distance learning (Report to the Office of Naval Research). Los Angeles: University of Southern California, Rossier School of Education. Available at

C R E S S T / U S C Page 8 CRESST Conf 07 v.2 Guidelines Guidelines: Team Training –Team training/assessment Jan Cannon-Bowers –Team motivation Dick Clark –Lessons based on Guidelines Pit stop in a race –To be published as a book Team Training Guidelines and Lessons (O’Neil, in preparation)

C R E S S T / U S C Page 9 CRESST Conf 07 v.2 A Worked Example for Multimedia Dr. Richard Mayer, University of California at Santa Barbara

C R E S S T / U S C Page 10 CRESST Conf 07 v.2 What Works Documentation A computer-based tool based on individual and team Guidelines –Existing courseware evaluation tool –Software supplied by Bill Bewley/CRESST –Results from Think Aloud Study indicate useful tool that could use revision

C R E S S T / U S C Page 11 CRESST Conf 07 v.2 Tool Development

C R E S S T / U S C Page 12 CRESST Conf 07 v.2 Guideline Status Individual and team guidelines and lessons submitted as deliverable to ONR – Individual guidelines published as book (O’Neil, H. F. (Ed.). (2005). What works in distance learning: Guidelines. Greenwich, CT: Information Age Publishing Inc.) – Team guidelines and lessons are in editing, to be published as a book Self and team assessment guidelines for distance learning are in development

C R E S S T / U S C Page 13 CRESST Conf 07 v.2 Self and Peer Assessment Distance Learning (DL) has minimal human instructional interventions for performance feedback Individual or team needs to perform this function Train self and peer assessment skills –Based on “ What Works ”— Dembo, O ’ Neil, & Chuang –Lessons learned — ONR/NEW Wendy Anson –Design work — Dembo & O ’ Neil –Development — Kazlauskas & O ’ Neil –Formative evaluation –Experimental Study at CSUSB

C R E S S T / U S C Page 14 CRESST Conf 07 v.2 Self-Assessment Model Process –Need clear goal/objective –Evidence about present state of learning-self-monitoring/self evaluation –Means of closing gap –Measure process and outcomes Similar Theories –Formative assessment (Black & Wiliam, 1998) –Gap analysis techniques (Clark & Estes 2002) Embedded in Content –Prototype coursework Domain-Independent Instructional Strategy

C R E S S T / U S C Page 15 CRESST Conf 07 v.2 Self-Assessment Guidelines Strategies Based on Goal-Setting and Planning Strategies Based Help-Seeking Strategies Based on Peer Interaction Strategies Based on Self-Monitoring Strategies Based on Self-Questioning Strategies Based on Self-Evaluation

C R E S S T / U S C Page 16 CRESST Conf 07 v.2 Self and Peer Assessment (cont.) Courseware Design –Explicit information regarding the techniques –Worked examples provided –Extensive use of (video) models for effective and ineffective use of skills –Techniques for self-regulation (e.g., self-monitoring) –Practice in techniques

C R E S S T / U S C Page 17 CRESST Conf 07 v.2 Self and Peer Assessment (cont.) Evaluation contexts –Embedded within CSUSB remedial math –Embedded within SWOS simulated navigation scenarios for USC Center for Cognitive Technology

C R E S S T / U S C Page 18 CRESST Conf 07 v.2 CRESST Web Site or any search engine: type CRESST

C R E S S T / U S C Page 19 CRESST Conf 07 v.2 BACK-UP SLIDES

C R E S S T / U S C Page 20 CRESST Conf 07 v.2 Control Elaboration Effort Math Achievement Worry Self-Efficacy Mastery Goal Orientation Gender Ethnicity Instrumental Motivation Maladaptive Help Seeking Performance Avoidance Prior Knowledge Language Use Cultural Practices Performance Approach Opportunity to Learn Minus SEM FOR CSUSB RESEARCH

C R E S S T / U S C Page 21 CRESST Conf 07 v.2 What Could Be Done Next on Guidelines With Additional Funds? Create Guidelines Management Training System –Guidelines –Reach back Lessons instantiating guidelines Create tools from guidelines –Formative evaluation