Download presentation
Presentation is loading. Please wait.
1
2009 AERA Annual Meeting, San Diego
Principle-based and Process-based Multidimensionality and Rater Effects in Validation of the Carbon Cycle Learning Progression Jinnie Choi Yong Sang Lee Karen Draney University of California at Berkeley 2009 AERA Annual Meeting, San Diego April 14, 2009
2
Outline of Presentation
Learning Progression of Carbon Cycle Complications in Validation Principle- and Process-based Multidimensionality Rater Effect Modeling Multidimensionality and Rater Effect Understanding the Results Next Steps 2009 AERA Annual Meeting
3
Outline of Presentation
Learning Progression of Carbon Cycle Complications in Validation Principle- and Process-based Multidimensionality Rater Effect Modeling Multidimensionality and Rater Effect Understanding the Results Next Steps 2009 AERA Annual Meeting
4
Learning Progression of Carbon Cycle (1)
The Carbon Cycle Project A part of the Environmental Literacy Project funded by National Science Foundation The goal is to integrate Environmental Science Literacy into contemporary K-12 curriculum Big Idea: Environmental Literacy Environmentally literate students are expected to be able to apply fundamental principles to processes in coupled human and natural systems Systems: coupled human and natural systems Principles for scientific accounts: scale, conservation of matter (both mass and atoms), conservation of energy and energy degradation Processes: generation, modification, and oxidation of organic carbon 2009 AERA Annual Meeting
5
Learning Progression of Carbon Cycle (2)
Carbon Cycle Learning Progression Framework Learning progressions are “descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another as children learn about and investigate a topic over a broad span of time” (Duschl, Schweingruber, & Shouse, 2007) Lower anchor, intermediate, and upper anchor understandings that define levels Two principle-based dimensions that define progress variables Six process-based dimensions 2009 AERA Annual Meeting
6
Learning Progression of Carbon Cycle (3)
Levels of achievement (performances) Levels Characteristics of Responses (Food chain example) 4: Qualitative model-based accounts Responses describe food chains at an atomic-molecular level, identifying key organic and inorganic materials that flow between organisms, and key processes responsible for changes in matter 3: “School science” narratives Responses attempt to explain food chains at atomic-molecular level by either identifying materials that connect or move between organisms, but does not consistently trace matter at a-m level and may confuse matter and energy flow 2: Force-dynamic with hidden mechanisms Responses construct food chains that connect organisms by materials such as "food" or organisms being eaten by other organisms. 1: Force-dynamic narratives Responses recognize that living organisms are related because they all exhibit qualities of being alive and depend on each other. Upper Anchor Lower Anchor 2009 AERA Annual Meeting
7
Learning Progression of Carbon Cycle (3)
Carbon Cycle Learning Progression Framework Photosynthesis Digestion/ Growth Cellular Respiration Decompo-sition Combustion Cross Process Upper Anchor (Process Dimension) 4: Qualitative model-based accounts 3: “School science” narratives 2: Force-dynamic with hidden mechanisms 1: Force-dynamic narratives Tracing Matter / Tracing Energy (Principle Dimension) Lower Anchor 2009 AERA Annual Meeting
8
Outline of Presentation
Learning Progression of Carbon Cycle Complications in Validation Principle- and Process-based Multidimensionality Rater Effect Modeling Multidimensionality and Rater Effect Understanding the Results Next Steps 2009 AERA Annual Meeting
9
Complication in Validation (1)
Principle-based Multidimensionality Tracing Matter and Tracing Energy Progress variables are described separately and used as scoring rubrics Process-based Multidimensionality Photosynthesis, Digestion/growth, Cellular respiration, Decomposition, Combustion, and Cross-process More like item groups, but crossed with Principle dimension Rater Effect Multiple raters rated different combinations of groups of people and sets of items 2009 AERA Annual Meeting
10
Complication in Validation (2)
Validation of a Learning Progression Levels of performances How are the students distributed across levels of performances? Do the items capture students’ lower level performances as well as higher level performances? Dimensional structure Do dimensions exist? Are the dimensions statistically distinguishable? What are the correlations between students’ performances on different latent dimensions? How consistently/differently do students perform on different sets of items that measured different latent dimensions? Is a set of items more difficult or easier when measuring one dimension than others? 2009 AERA Annual Meeting
11
Outline of Presentation
Learning Progression of Carbon Cycle Complications in Validation Principle- and Process-based Multidimensionality Rater Effect Modeling Multidimensionality and Rater Effect Understanding the Results Next Steps 2009 AERA Annual Meeting
12
Modeling Multidimensionality and Rater Effects
Multidimensional Random Coefficient Multinomial Logit Model (MRCML; Adams, Wilson & Wang, 1997) Confirmatory analysis of between-item multidimensionality Multifaceted Item Response Model (Linacre ,1994) Examine variation in the harshness or leniency of raters Examine the fit (or consistency) of individual raters with other raters 2009 AERA Annual Meeting
13
Outline of Presentation
Learning Progression of Carbon Cycle Complications in Validation Principle- and Process-based Multidimensionality Rater Effect Modeling Multidimensionality and Rater Effect Understanding the Results Next Steps 2009 AERA Annual Meeting
14
Understanding the Results (1)
Levels of Performances 2009 AERA Annual Meeting
15
Understanding the Results (2)
Principle-based Multidimensionality Relative fit test Model Deviance # of parameters Unidimension Multidimension 110 112 Difference 37.66 2 Chi2(37.66, 2) = 5.272e-18 2009 AERA Annual Meeting
16
Understanding the Results (3)
Principle-based Multidimensionality Item difficulty estimates 2009 AERA Annual Meeting
17
Understanding the Results (4)
Principle-based Multidimensionality Person ability estimates Dimension Matter Energy 1.231 0.959 1.335 EAP Reliability 0.623 0.614 2009 AERA Annual Meeting
18
Understanding the Results (5)
2009 AERA Annual Meeting
19
Understanding the Results (6)
Comparison of rater harshness and weighted fit By checking reliability of scoring between pairs of raters, the rating difference was resolved. This graph shows the rater effects before the rating difference was resolved. 2009 AERA Annual Meeting
20
Outline of Presentation
Learning Progression of Carbon Cycle Complications in Validation Principle- and Process-based Multidimensionality Rater Effect Modeling Multidimensionality and Rater Effect Understanding the Results Next Steps 2009 AERA Annual Meeting
21
Next Steps More measurement questions…
How can we model responses if a rater rates differently for items that measure different dimensions? How can we model responses if a rater rates differently for items that requires different scoring criteria? How can we model responses when an item is measuring one process dimension but doubly scored based on principle dimensions? How do we model responses when a set of item is differently perceived by different groups of people? Informing and revising the four building blocks (Wilson, 2004) … Construct map Items design Outcome space Measurement models 2009 AERA Annual Meeting
22
For more information… Environmental Literacy research groups
Michigan State University Long Term Ecological Research (LTER) Network University of California at Berkeley University of Michigan Northwestern University AAAS Project 2061 Visit our websites at… Environmental Literacy website Berkeley Evaluation and Assessment Research (BEAR) Center Thank You! 2009 AERA Annual Meeting
Similar presentations
