1. C R E S S T / U C L A Model-Based Assessment: Why, What, How, How Good, and What Next? Eva L. Baker National Research Council, Board on Testing and Assessment Bridging the Gap Between Classroom and Large-Scale Assessment Workshop January 23-24, 2003 Washington, DC UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing

2. C R E S S T / U C L A Why? Assessment Knowledge: Usable and Useful Usable Knowledge In a form that can be understood In a form that can be applied Timed appropriately May cause rethinking of the problem Useful Knowledge Rethinking indicates a new solution path Adapted to situation Sufficient to guide solution Improved outcomes occur as a result

3. C R E S S T / U C L A Why Are Some Schools Successful in Using Assessment Knowledge?  Focus on learning (students and adults)  Constant use of appropriate information (formal and informal)  Focus on feedback and change  Public display and exchange  Community pride in outcomes of students and place

4. C R E S S T / U C L A Goals for CRESST Model-Based Assessment (MBA)  Assessment components share a common framework. MBA starts with thinking skills and applies them to content domains to support Coherent, sustained learning Spiral teaching-common language Transfer (application to new situations) Multipurpose Learning organization

5. C R E S S T / U C L A CRESST Model-Based Assessments (MBA) Features  Research based  Focus on cognition and learning  Abstracted in models based on key learning elements—principles guiding test design and instruction  Operationalized in templates  Reusable and cost-sensitive design/training/scoring

6. C R E S S T / U C L A Model-Based Assessment Cognitive Families Content Understanding Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning

7. C R E S S T / U C L A Model-Based Assessment Design  Models to templates (to specification) to tests  Template contains domain-independent (transfer) and domain-specific (strategy and knowledge) components  Templates that allow common domain-specific design approaches to be used, e.g., primary sources in history  Scoring requirements

8. C R E S S T / U C L A Expert Model—Deep Understanding of Content (Domain Independent)  Principles or themes (big ideas)  Use of prior knowledge  Explicit relationships  Avoiding misconceptions  Expert performance-based scoring

9. C R E S S T / U C L A Specifications for Large-Scale Use  Standards reference  Place in sequence  Content domain (what’s in and out)  Proportion of effort  Format options  Interpretation rules  Time

10. C R E S S T / U C L A Template  Task(s)  Format(s)  Prompt(s) and requirements  Scoring  Directions  Sample

11. C R E S S T / U C L A Three Templates for the Model of Deep Content Understanding  Explanation  Explanation with explicit knowledge  Graphical representation of relationships

12. C R E S S T / U C L A Deep Content Understanding  Primary source materials in each domain  Student required to integrate prior knowledge and principles to succeed  Scored by using expert model in subject matter

13. C R E S S T / U C L A Content Understanding Template #1 Explanation  An array of primary source materials  A prompt that asks for an explanation in context  Constructed (written) answer  Evaluated by means of a scoring rubric that operationalizes learning model

14. C R E S S T / U C L A Hawaiian History Writing Assignment: Bayonet Constitution Be sure to show the relationships among your ideas and facts. Your essay should be based on two major sources: 1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution. 2. What you have learned from the readings yesterday. Imagine you are in a class that has been studying Hawaiian history. One of your friends, who is a new student in the class, has missed all the classes. Recently, your class began studying the Bayonet Constitution. Your friend is very interested in this topic and asks you to explain everything that you have learned about it. Write an essay explaining the most important ideas you want your friend to understand. Include what you have already learned in class about Hawaiian history, and what you have learned from the texts you have just read. While you write, think about what Thurston and Liliuokalani said about the Bayonet Constitution, and what is shown in the other materials.

15. C R E S S T / U C L A EXCERPTS from HAWAIIAN HISTORY PRIMARY SOURCE DOCUMENTS LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they became wealthy, their greed and their love of power increased. Although settled among us, and drawing their wealth from resources, they were alien to us in their customs and ideas, and desired above all things to secure their own personal benefit. Kalakaua valued the commercial and industrial prosperity of his kingdom highly. He sought honestly to secure it for every class of people, alien or native. Kalakaua’s highest desire was to be a true sovereign, the chief servant of a happy, prosperous, and progressive people. And now, without any provocation on the part of the king, having matured their plans in secret, the men of foreign birth rose one day en masse, called a public meeting, and forced the king to sign a constitution of their own preparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.

16. C R E S S T / U C L A Explanation Scoring Rubric  General impression of content quality  Principles or concepts  Prior knowledge  Examples  Misconceptions  Argumentation

17. C R E S S T / U C L A Template #2 Prior Knowledge and Explanation  Explicit measurement of knowledge domain before explanation  Uses short answer or selected response  Helps interprets explanation performance

18. C R E S S T / U C L A Template #3 Knowledge Representation  Same prompts  Key aspects of ideas, supporting facts and views and their relationships  Relationship is explicit  Organizational options Core and peripheral Hierarchical Cause-and-effect Chronological  Expert scoring

19. C R E S S T / U C L A History

20. C R E S S T / U C L A Genetics

21. C R E S S T / U C L A Bicycle Pump

22. C R E S S T / U C L A Brief History of MBA in LAUSD  Content understanding and problem- solving models  Explanation templates  4 subjects, 3 grade levels, 2 languages  Purposes: (1) to clarify expectations; (2) to provided instructionally embedded assessment; (3) to get a measure of school performance  CRESST-managed teacher involvement

23. C R E S S T / U C L A LAUSD Process  Teacher design teams  LAUSD standards first  Adapted to success standards  Training cadre of scorers  Training trainers  Supervising scoring

24. C R E S S T / U C L A LAUSD Process (cont’d)  Shift in four-topic focus (capacity based) to two and then to one, now back to two  Continual assaults  Curriculum mandates  Accountability pressure (API)  Long-term embedded approach resurfacing

25. C R E S S T / U C L A Present LA Situation  Administered in 2, 3, 4, 5, 6, 7, 8, 9  Purpose added regarding promotion  Teacher scored with an audit reported to school  Local sub-districts managing activity  Ongoing validity studies  District review of alternative assessments

26. C R E S S T / U C L A

27. CRESST Validation Studies  Score reliability  Task and rater generalizability  Stability of student performance over time  Relationships among measures  Instructional sensitivity  Opportunity to Learn (OTL)  Effect of school composition on performance  Cut-score modeling

28. C R E S S T / U C L A LAUSD Grade 7 Students’ Achievement Levels: Comparison of 2002 California Standards Test and Performance Assignment Scores Evidence of Predictive Validity

29. C R E S S T / U C L A LA Scale-Up  Cost and time driven  Maintained by board and union support  Transfer of responsibility  Reduction in technical quality  Reduction in range of measures  Positive evaluation from independent group focusing on changing teaching practices

30. C R E S S T / U C L A Continuing R&D Areas  New contexts  Trade-offs (limited number of templates vs. wide range of formats)  Performance over time  Scalability in the long run  Authoring systems to support teacher- developed assessments linked to large- scale assessment

31. C R E S S T / U C L A Summary of Assessment Knowledge Requirements  Knowing why  Knowing what to assess: content plus cognitive demands (problem solving, communication, learning to learn, teamwork, content knowledge)  Knowing how: transfer (application to other topics and situations)  Reflecting: applying MBA to teaching

32. C R E S S T / U C L A Model-Based Assessment Cognitive Families Content Understanding Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning

33. C R E S S T / U C L A Context for Success of Knowledge-Based Reform  Local ownership of knowledge  Infrastructure and stability  Capacity to investigate  Learning  Congruence or peace with external mandates

34. C R E S S T / U C L A Usable Knowledge and Support May Get to Useful Knowledge  For assessment knowledge to be useful, it depends upon the context, capacity, and communication of the teaching system  For assessment knowledge to be useful to students, it must go to the heart of why, what, and how they learn