1. Assessment and Concept Maps Rose M. Marra David Jonassen Bressanone – December 2007 1

2. Concept Maps Are used in many ways.. Greatest potential benefit – student creation. Benefits ◦ Supports knowledge construction ◦ Clarifies relationships ◦ Often identifies – even to the student – what she does and doesn’t know 2

3. Assessing with Concept Maps Creating concepts maps engages critical and creative thinking skills Assessing maps can assess higher order skills. ◦ Evaluation, analyzing, connecting, synthesizing, elaborating, imagining. 3

4. Assessing with Concept Maps Assess learners’ cognitive structures. Patterns of relationships among concepts in long term memory. Change in learner knowledge structures. 4

5. Using Concept Maps as Assessment Formative assessment ◦ To help learners self assess domain and structural knowledge ◦ Peer to peer ◦ Teacher to student ◦ Ongoing task of updating concept map  Emphasis on revision and descriptive links  Accompany with reflection activities  How has your map changed?  How is your map different from Mario’s map? Why?  Whose is “right”?  Student must be able to justify his or her map ◦ See list of link labels 5

6. Using Concept Maps as Assessment Assessing Domain knowledge ◦ Graded task ◦ Assign points, etc. ◦ Again, can be individual or group task Create concept map as reflection tool ◦ Before and after an experiment ◦ Before - predict how concepts are related ◦ After – verify relationships; update 6 Rubric will vary based on domain and task.

7. Concept Mapping Assessment Methods Built in statistics provided by some tools. ◦ See Semantica 7

8. 8

9. Assessing Learners’ Concept Maps Relevant questions: ◦ Are all important concepts represented in the map as nodes? (completeness). ◦ Are the relationships between nodes (links) descriptive? For instance, XX “is related to” YY is a meaningless link. Links should be descriptive and accurate. ◦ How well connected is the map? Try to eliminate orphans or dead-ends. ◦ What are the most important concepts (most interrelated) in the map? Does this align with how the knowledge domain is actually structured? Also - Student self – assessment 9

10. Assessing Concept Maps -- Rubrics 10 BreadthNumber of nodes in the network. Embeddedness Ratio of instances to concepts; shows how well integrated the concepts in the domain are. Centrality of nodes Centrality is a measure of the importance of concepts in a domain. Number of direct links (concepts linked directly to it) and indirect links (concepts linked to other concepts directly linked to it). Are the most “central” concepts the most important to the map? Depth (hierarchicalness ) Measured by the levels of nodes represented. Further, is each subordinate concept more specific than concept above? Concept map rubric criteria (adapted from Jonassen (2000)

11. Assessing Concept Maps -- Rubrics 11 Link direction Does the direction of links with arrows convey hierarchical or causal relationship between nodes in propositions (McClure & Bell, 1990)? Number of links Links should be used economically. If six different links will describe all of the relationships in the net, then do not use more than six (i.e., don't use three different links that mean the same thing, e.g., "attribute of," "property of," and "characteristic of"). Salience Number of valid links in map divided by total number of links in map (Hoz, Tomer, & Tamir, 1990) Deadend linksThe number of "dead-end" nodes (linked to only one other concept)

12. Assessing Concept Maps -- Rubrics 12 AccuracyAre learners making meaningful connections? Is the text in nodes correct? That is, is the information in the net correct? This part of rubric will vary based on content domain.

13. Rubric Example 13 ElementExcellentUnsatisfactory Concepts are descriptive of content domain (breadth of net) Map includes all important concepts; describes domain on multiple levels Map includes most important concepts; describes domain on limited number of levels Map includes numerous concepts; important concepts missing; describes domain on only one level Map includes minimum concepts with many important concepts missing Embeddedness and interconnectedness All concepts interlinked with several other concepts Most concepts interlinked with 3 or more other concepts Several concepts linked to only one or two other concepts Most concepts linked to only one or two other concepts Links are descriptiveLinks succinctly, accurately describe all relationships Links are descriptive and valid for most relationships Some links unclear or vague; some invalid or unclear Links are vague; show inconsistent relationships Links are efficient (parsimonious, no more nor less than necessary) Each link type is distinct from all others, clearly describes relationship; used consistently Most links are distinct from others; discriminate concepts; present variety of relationships; used fairly consistently Several links are synonymous; don't discriminate concepts well; don't show a variety of relationships; used inconsistently Most links synonymous or vaguely describe relationships and aren't distinct from other links

14. Concept Map Peer Assessment Activity Learners create maps individually or in small groups. Learners look at each others’ maps and ask questions, and analyze the maps for accuracy and organization Activity goal ◦ Explain your maps organization and content ◦ Encourage discussion of content ◦ Instigate changes / additions to maps. 14

15. Concept Map Peer Questions Why are these two concepts connected? Why don’t you have these concepts connected directly? Where is concept ____ in your map? I don’t agree with this link label. I labeled that link _________. Why did you use this link label? The textbook says ______ and your map does not agree with that. Why? 15

16. Your Turn – Peer Concept Map Assessment Do a peer assessment of another map created by one of your workshop colleagues. Use the questions just discussed. Create your own questions. What did you learn from this activity? 16

17. End 17

18. Assessing Structural Knowledge Said differently, it is how the person represents their understanding of a domain or discipline. (1) relevant domain concepts, (2) the presence and/or nature of relationships between those concepts, and (3) the strength of those relationships. 18

19. Assessing Structural Knowledge Path finder nets Pathfinder nets are built by eliciting semantic relatedness ratings. In this study semantic relatedness ratings were captured on paper, and assessed using the computerized program KNOT™. Participants’ ratings were entered into the software by the researcher, and all evaluations were conducted using the assessment features of KNOT™. Need example 19

20. Assessing Structural Knowledge -- Analogies Describes links – but not strengths of the links. Bach : Composing :: Monet : ◦ a. painting ◦ b. composing ◦ c. writing ◦ d. orating "Bach is to (:) Composing as (::) Monet is to (:) _______." 20

21. Assessing Structural Knowledge -- Analogies Bach : Composing :: Monet : ◦ a. painting ◦ b. composing ◦ c. writing ◦ d. orating "Bach is to (:) Composing as (::) Monet is to (:) _______." a. painting Bach is most known for composing music, Monet is most known for his painting.BachMonet 21

22. Assessing Structural Knowledge -- Analogies DOCTOR : HOSPITAL :: ◦ (A) sports fan : stadium ◦ (B) cow : farm ◦ (C) professor : college ◦ (D) criminal : jail (E) food : grocery store CUB : BEAR :: ◦ (A) piano : orchestra ◦ (B) puppy : dog ◦ (C) cat : kitten ◦ (D) eagle : predator (E) fork : utensil 22

23. Assessing Structural Knowledge -- Analogies BIRD : NEST :: ◦ (A) dog : doghouse (B) squirrel : tree ◦ (C) beaver : dam ◦ (D) cat : litter box (E) book : library 23

24. Assessing Structural Knowledge – Essay Tests Essay tests or writing tasks 24 Compare / contrast the presidencies of Ronald Reagan and George W. Bush. Discuss the ways that voltage and current are related. Which process is evaporation more closely related to? Condensation or precipitation? Why?

25. Assessing Structural Knowledge Path finder nets Citation Teaching of Psychology 1984, Vol. 11, No. 2, Pages 99-101 True-False Tests That Measure and Promote Structural Understanding 25

26. Concept Mapping Tools Computer-based tools for organizing, representing and assessing knowledge organization and understanding. Maps: ◦ Concepts or nodes usually enclosed in circles or boxes of some type. ◦ Links (labeled) between concepts to define relationships between the two. Enables easier addition to and changes to maps… and assessment! 26

27. Concept Map Rubric Criteria 27 BreadthNumber of nodes in the network. EmbeddednessRatio of instances to concepts; shows how well integrated the concepts in the domain are. Centrality of nodes Centrality is a measure of the importance of concepts in a domain. Number of direct links (concepts linked directly to it) and indirect links (concepts linked to other concepts directly linked to it). Are the most “central” concepts the most important to the map? Depth (hierarchicalness) Measured by the levels of nodes represented. Further, is each subordinate concept more specific than concept above? Link directionDoes the direction of links with arrows convey hierarchical or causal relationship between nodes in propositions (McClure & Bell, 1990)?

28. Concept Map Criteria 28 Number of links Links should be used economically. If six different links will describe all of the relationships in the net, then do not use more than six (i.e., don't use three different links that mean the same thing, e.g., "attribute of," "property of," and "characteristic of"). SalienceNumber of valid links in map divided by total number of links in map (Hoz, Tomer, & Tamir, 1990) Dead-end links The number of "dead-end" nodes (linked to only one other concept) AccuracyAre learners making meaningful connections? Is the text in nodes correct? That is, is the information in the net correct?

29. 29

30. How can assessing with Concept Maps make a difference in learning? Concept maps can make external learners’ internal knowledge representations. Research has shown that students with better-structured knowledge are better thinkers, rememberers and problem solvers. Supports formative assessment via identification of mis-conceptions. Reinforces expectation with students of requiring synthesis and connection-making. Tools can provide rich individual or class level metrics (e.g. depth of nets). These also facilitate “grading” of nets. 30

31. Concept Maps Applications Organizational activity before a writing task. Reflection tool after completing a science lab Create a comprehensive and continually updated map of their understanding of how the key concepts in the domain fit together. 31 All can provide artifacts for assessment!

32. Concept Mapping Assessment Methods NOTE: All methods can be used effectively as either formative feedback or for a “grade”. 32