1. 1 Free software that scores biology field reports and classroom essays Dr. Roy B. Clariana – Penn State U. RClariana@psu.edu http://www.personal.psu.edu/rbc4 T3 International Conference Denver, CO February 25th, 2006

2. 2 of 24 Presentation Goals This presentation will demonstrate how to download and use a software tool called ALA- Reader (analysis of lexical aggregates) to score students’ biology essays My goal: To recruit action researchers from this audience

3. 3 of 24 Presentation flow Quick ALA-Reader demo – to see it work Brief look at its genesis and how it works Step-by-step demo – how you set it up and use it in your classroom Monday morning Review the data files produced One more run Final questions, Show of hands

4. 4 of 24 Quick ALA-Reader Demo Place student essays, terms file, expert referent essay, and ALA-Reader software into the same folder (click)click Double click ALA-Reader Click the Mark L button Type in the file name of a student’s essay Observe scores

5. 5 of 24 Genesis of ALA-Reader We are interested in concept maps, and developed software called ALA-Mapper to score concept maps Then we noticed that teachers frequently associate concept maps and essays (i.e., concept maps are a good organization tool) For example, Lomask et al. derived concept maps from student’s essays in order to score the Connecticut statewide science assessment So we developed ALA-Reader to score essays based on ALA-Mapper

6. 6 of 24 Concept map validity Concept maps are generally considered to be valid and reliable measures of science content knowledge (Ruiz- Primo, Schultz, Li, Shavelson, CREST in California, Rye and Rubba (2002), Jonassen, Beissner, & Yacci, 1993...). essays multiple-choice tests

7. 7 of 24 Types of knowing Declarative knowledge (semantic and episodic) Procedural knowledge “Structural knowledge” – knowledge of the interrelationships of the domain elements, may be most essential for higher-order learning structural knowledge seems to be part of both declarative and procedural knowledge (so it may be a manifestatio of the other two) Both concept maps and essays (esp. compare/contrast prompts) capture aspects of all three depending on what components of the map or essay you measure

8. 8 of 24 “Structural knowledge” Vygotsky (in Luria, 1979); Miller (1969) card- sorting approaches Deese’s (1965) ideas on the structure of association in language and thought Recent neural network representations (e.g., Elman, 1995) Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysis (Pearson); ETS; Vantage Learning Corp. Concept maps (Jonassen…)

9. 9 of 24 measuring it … One measure of structural knowledge consists of the “links” in a concept map A Link can also can be considered to be a ‘proposition’ that includes the two concepts that are joined by the link Links are one component of concept maps that are easy to measure

10. 10 of 24.. map w/ proposition array Most studies use only link (edge) information, usually called “propositions”. Proposition Array lungs oxygenate deoxygenate pulmonary artery pulmonary vein left atrium right ventricle Map (n 2 -n)/2 pair-wise comparisons

11. 11 of 24 Convert raw map data into scores Currently, we use a data reduction and comparison approach called Pathfinder network representation (UNM, Schanveldt, 1990) to turn counts into scores PFNets describe the least weighted path to connect the terms in the data proximity array Scores are established by comparing the participant’s PFNet to a referent (expert) PFNet, and calculating the number of common links (the intersection)

12. 12 of 24 Concept map & traditional test scores Example: During a lesson on the heart, students created concept maps while reading, then took a test. The correlations between the concept map link scores from ALA-Mapper and terminology multiple-choice test scores are: Taricani & Clariana (2006), r =.78 Poindexter & Clariana (2006), r =.77

13. 13 of 24 We use the same approach for ALA-Reader … an electrical signal starts the heartbeat, by causing the atrium to contract. The blood then flows through the pulmonary valve into the pulmonary artery and then into the lungs. Once inside the lungs, the blood gives up the carbon dioxide (cleansed) and receives oxygen. This oxygenated blood … atrium contract lungs cleansedoxygenated P artery P valve TextPFNet Proposition array

14. 14 of 24 Step-by-step 1. Create the writing prompt 2. Students write essays, when done, they save it in a specific format 3. Teacher creates key terms with synonyms and saves as terms.txt 4. Teacher creates one or two expert referent essays 5. Put all into the same folder and run ALA-Reader 6. Collect scores from the report file

15. 15 of 24 Step 1. Create the writing prompt For this demo, we used: In a 300-word essay, describe the structure and function of the human heart and circulatory system.

16. 16 of 24 Step 2. Student essay file format The Dance of Life Student 5 The heart is the major component of the circulatory system. Its function is to circulate blood, which is comprised of both blood cells and plasma, throughout the body. Two types of blood vessels are used to circulate blood. Arteries are used to carry oxygen rich blood throughout the body, while veins are used to carry oxygen deficient blood to the heart. The heart is one of the most complex organs in the body. One of its primary jobs is to oxygenate blood for the body. Here is how it works. As stated earlier, veins carry oxygen deficient blood to the heart. The blood flows into the right atrium, then the right ventricle, and then the pulmonary artery pumps blood to the lungs for oxygen. Once the blood is oxygenated, pulmonary veins carry blood back to the heart, where it flows from the left atrium to the left ventricle. The aorta pumps oxygen rich blood throughout the body, then the veins carry oxygen deficient blood back to the heart, where the cycle begins again. Quite literally, the heart is “the heart” of the circulatory system. It is amazing to think that such a seemingly simple organ is so complex. 30 max

17. 17 of 24 Students write essays Students use Notefolio app* to write their essay on a TI calculator (TI-83 Plus family, TI-84 Plus family, TI-89, TI-89 Titanium, TI- 92 Plus and Voyage™ 200). Teacher uses NoteFolio™ Creator Software* on the teachers laptop to organize and view the students’ essays (TI connectivity kit or TI- Navigator wireless and TI Connect software* to import and export)NoteFolio™ Creator Software (*free downloads)

18. 18 of 24 Step 3. create terms.txt file aorta aortic_valve aortic_v body cleanse clean carbon contract contrac diastolic diastol endocardium endocar epicardium epicar inferior_vena_cava inferior left_atrium left_atri left_ventricle left_vent lungs lung membranes membran mitral_valve mitral bicusp atrioventricular myocardium myocard oxygen pericardium pericard pulmonary_artery pulmonary_art pulmonary_valve pulmonary_val pulmonary_veins pulmonary_vei relax right_atrium right_atri right_ventricle right_vent superior_vena_cava superior systolic systol tricuspid tricusp 30 terms maximum Space to separate synonyms _ to show spaces 30 max full term pattern match mitral_valve mitral bicusp atrioventricular

19. 19 of 24 Step 4. create expert1.txt file Heart Essay Expert 1 Blood flows from the extremities towards the inferior vena cava and superior vena cava. Blood enters the right atrium from the inferior vena cava and superior vena cava. Flow passes from the right atrium through the tricuspid valve to the right ventricle. From the right ventricle, blood flows oneway through the pulmonary valve onto the pulmonary artery. The blood flows through the pulmonary artery to the lungs where it is cleansed and oxygenated. Blood flows from the lungs back to the heart through the pulmonary veins. Blood then enters the left atrium from the pulmonary vein. Flow passes from the left atrium through the mitral valve into the left ventricle. Blood then flows out of the heart through the aortic valve into the aorta. The blood in the aorta flows to all parts of the body. contract and systolic diastolic and relax membranes endocardium and myocardium epicardium and pericardium 30 max

20. 20 of 24 Step 5. Run ALA-Reader Place student essays, terms file, expert referent essay, and ALA-Reader software into the same folder (click)click Double click ALA-Reader Click the Mark L button Type in the file name of a student’s essay Use scores

21. 21 of 24 Step 6. Use scores See report.txt or L_report.txtreport.txtL_report.txt

22. 22 of 24 ALA-Reader future Current beta version forms proposition array based on sentence-level and also linear associations Future: Add other ways to parse the text, e.g., pure linear, reverse linear, double-chunk, etc.double-chunk Add a graphing utility to the tool Download from my web site at www.personal.psu.edu/rbc4

23. 23 of 24 Questions ALA-Reader can mark any essay (you provide the expert comparison essay and the list of important terms) Any Questions Show of hands… One-on-one after the session, also email me with any questions, etc.

24. 24 of 24 Our papers on concept maps and essays Koul, R., Clariana, R. B., & Salehi, R. (2005). Comparing several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32 (3), 261-273. linklink Taricani, E. M. & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 54, 61-78. Clariana, R.B., Koul, R., & Salehi, R. (2006). The criterion- related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33 (3), in press. Poindexter, M. T., & Clariana, R.B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), in press.