1. C R E S S T / U C L A Alicia M. Cheak, Gregory K. W. K. Chung, Eva L. Baker, Cecile H. Phan, and Linda F. de Vries American Educational Research Association, New Orleans, LA April 2002 UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing An Investigation of the Feasibility of an Automated Approach to Classifying Open-Ended Concept Map Links

2. C R E S S T / U C L A Overview of Presentation è Concept map definition è Past research è Current research study n Purpose of study, method, and results

3. C R E S S T / U C L A Concept Maps è Node-Link representation (nodes = concepts, links = relationships) è Typically used for instructional purposes è Sometimes used for assessment è Automated scoring possible with fixed set of concepts and links sunlight surface warming leads to

4. C R E S S T / U C L A Concept Map Sample

5. C R E S S T / U C L A Previous CRESST Work è Use concept mapping for assessment purposes n Predefined terms and links allow for automated scoring n Expert maps used as scoring criterion n Tested with 4th-graders to adult learners n Map scores correlate with essay measures of content understanding (r ~.70)

6. C R E S S T / U C L A Purpose of Our Study è Investigate feasibility of an automated approach to classifying open-ended links n Identify “root” form of links used in open-ended concept maps in two content areas n Develop software to convert links into root forms n Examine quality of conversion n Test approach with existing map data è Long-term goal is automated scoring of open- ended links

7. C R E S S T / U C L A Method è Analyze distribution of links in two content areas n Highly skewed distribution would be a favorable situation for an automated classification system è Approach n Raters convert links to root form  Example: is a, is an, is one of, are => is a  Example: are part of, are (is) a part of, is a part of the, is part of, is part of the, part of, part of process => part of n Identify most common root form links across two content areas (physiology and marine mammals)

8. C R E S S T / U C L A Method è Automated approach n Use part-of-speech (POS) tagger and custom-developed software to convert links to root form n Examine quality of conversion  With respect to student-generated links  With respect to rater-converted links n Test overall system with existing map data

9. C R E S S T / U C L A Results: Distribution of Links è Very skewed distribution of links è Marine Mammals: n 898 links converted to 134 root form links n Most used: has, is, part of, eats (43%); top 10 links account for 58% è Physiology: n 464 links converted to 92 root form links n Most used: part of, is, causes, has (36%); top 10 links account for 60% è Common links n Has, is, part of

10. C R E S S T / U C L A Results: Quality of Automated Conversion vs. Student-Generated Links

11. C R E S S T / U C L A Results: Quality of Automated vs. Rater-Converted Links

12. C R E S S T / U C L A Test of Automated Approach Propositions (57) Exact match (28) No exact match (29) Synonyms Match (8)No match (21) Correct (12)Incorrect (5)Correct (4)Incorrect (3) Could not convert (7) Correct (28) ? (1)? (4)

13. C R E S S T / U C L A Summary è Limitations n Small sample of links, only 2 content areas, age of sample (4th-graders) n Analysis of mismatches indicates insensitivity to prepositions and other integral parts of speech è Skewed distribution of link types favorable for automated processing è Automated conversion yields links similar to student-generated and rater-converted links è Test of automated approach promising

14. C R E S S T / U C L A For More Information è Greg Chung è E-mail: greg@ucla.edu è Web: http://www.cse.ucla.edu/

15. C R E S S T / U C L A Categories of Link Types

16. C R E S S T / U C L A Distribution of Links