1. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 6 February 2003 ONR/NSF Technology Assessment of Web-Based Learning Eva L. Baker UCLA/CRESST Harry O’Neil USC/CRESST Assessment of Distance Learning

2. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 1 Why Are Some Organizations Successful in Using Knowledge? Focus on learning Constant use of appropriate information (formal and informal) Focus on feedback and change Public display and exchange Community pride in outcomes of learners Select knowledge to foster these ends

3. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 2 Assessment in Support of Evaluation Distance Learning Intellectual Capital Social Capital + Learning Organization

4. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 3 Goals for CRESST Model-Based Assessment (MBA) Assessment components share a common framework. Model-based assessment starts with intellectual capital and applies it to content domains to support –Coherent, sustained learning –Spiral design—common language –Transfer (application to new situations) –Multipurpose

5. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 4 CRESST Model-Based Assessment (MBA) Features Research-based Focus on cognition and learning Abstracted in models based on key learning elements—principles guiding test design and instruction Operationalized in templates Reusable and cost-sensitive design/training/scoring

6. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 5 Model-Based Assessment Families of Cognitive Demands Content Understanding Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning

7. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 6 Expert Model—Deep Understanding of Content (Domain Independent) Came from syntheses of research on learning Principles or themes (big ideas) Use of prior knowledge Explicit relationships Avoiding misconceptions Expert performance-based scoring

8. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 7 Model-Based Assessment Design Models to templates (to specification) to tests Template contains domain-independent (transfer) and domain-specific (strategy and knowledge) components Templates that allow common domain- specific design approaches to be used, e.g., primary sources in history Scoring requirements

9. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 8 Template Task(s) Format(s) Prompt(s) and requirements Scoring Directions Sample

10. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 9 One Template for the Model of Deep Content Understanding Graphical representation of relationships

11. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 10 Template Knowledge Representation Prompts Key aspects of ideas, supporting facts and views and their relationships Relationship is explicit Organizational options –Core and peripheral –Hierarchical –Cause-and-effect –Chronological Expert scoring

12. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 11 Genetics

13. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 12 Bicycle Pump

14. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 13 Economics

15. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 14 Brief History of MBA in LAUSD Content understanding and problem- solving models Explanation templates 4 subjects, 3 grade levels, 2 languages –Math, science, history, literacy Purposes: (1) to clarify expectations; (2) to provided instructionally embedded assessment; (3) to get a measure of performance CRESST-managed instructor involvement

16. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 15 CRESST Validation Studies Score reliability Task and rater generalizability Stability of student performance over time Relationships among measures Instructional sensitivity Opportunity to Learn (OTL) Cut-score modeling

17. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 16 Model-Based Assessment of Social Capital Trust EfficacyNetworks EffortTransparency Learning Organization

18. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 17 Context for Success of Knowledge-Based Reform Measures of intellectual and social capital Psychometrics of team-based simulation Infrastructure and stability Capacity to investigate—6 Sigma organizational Learning Congruence or peace with external mandates

19. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 18 Summary of Assessment Knowledge Requirements Knowing why Knowing what to assess: intellectual and social capital components Knowing how: transfer (application to other topics and situations) Applying MBA to distance learning

20. ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 19 CRESST Web Site http://www.cse.ucla.edu