2. Diane Ebert-May Department of Plant Biology Michigan State University www.first2.org Disciplinary Research Strategies for Assessment of Learning

3. Question 1 Please respond on a scale of 0 - 100 in increments of 10: How important is it to use multiple kinds of data to assess student learning?

5. Question 2 Please respond on a scale of 0 - 100 in increments of 10: How often do you use data to make instructional decisions?

7. True or False? Assessing student learning in science parallels what scientists do as researchers.

8. Assessment in Teaching Parallels Assessment in Research â We ask questions and develop hypotheses to solve problems and make predictions about learning. â Our questions are based on current knowledge and theories, are creative, original and relevant to the investigator. â Research designs and methods we use to collect data are logical processes to answer questions. â Instruments/techniques we use are valid, repeatable measures of learning. â Assessment (results) help us understand student thinking. â Results drive our next questions and decisions about a course. â Our ideas are peer reviewed - informally or formally.

9. What is assessment? Data collection with the purpose of answering questions about… »student understanding »students’ attitudes »students’ skills »instructional design and implementation Use data to make predictions about student learning

10. Graduate Education Often excellent at preparing individuals to design and carry out disciplinary research.

11. Graduate Education Often inadequate and haphazard in preparing future faculty/professionals to take on the increasingly complex demands of the professoriate. Teaching is not mentored, peer reviewed, or based on accumulated knowledge.

12. Solution: IRD model Intergenerational research development teams (IRDs) in cooperative academic environments »Who: senior faculty, junior faculty, postdoctoral and graduate students. »What: scholarship of science teaching and learning is fully integrated into the professional culture along with discipline-based activities. Assessment is critical to both practices.

13. Collaborators Janet Batzli - Plant Biology (Director-Biocore, U of Wisconsin) Doug Luckie - Physiology (Associate Professor) Scott Harrison - Microbiology (Graduate student) Tammy Long - Plant Biology (Assistant Professor) Jim Smith - Zoology (Associate Professor) Deb Linton - Plant Biology (Postdoctoral Fellow) Heejun Lim - Chemistry Education (Postdoctoral Fellow) Duncan Sibley - Geology (Professor) National Science Foundation, Hewlett Foundation

14. Recognizing and Rewarding Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics (2003) »National Research Council »www.nap.edu/catalog/10024.htmlwww.nap.edu/catalog/10024.html

15. What are central questions about learning? 1.What do we want our students to know and be able to do? 2.What knowledge or misconceptions do our students bring to the course? 3.What evidence will we accept that students know and can do? 4.How does our instruction help learning?

20. What Type of Learning? Bloom (1956) 6 major categories in the Cognitive Domain of Educational Objectives

21. Cognitive Levels Knowledge - remember Understanding and Application - grasp meaning, use, interpret Critical Analysis - original thinking, open-ended answers, whole to parts, parts to whole, evaluation

22. What type of data do we gather? Depends on the evidence we will accept that students have learned what we want them to learn. Data must be aligned with the course goals. Measures of knowledge, attitudes, and skills. »tests, extended responses, concept maps, »research papers, teamwork, communication Use Bloom’s as a tool to categorize cognitive domains

24. Research Question How can diagnostic assessment questions help us understand students’ prior understanding and progressive thinking about the carbon cycle over time?

25. Prediction Diagnostic, robust questions about the carbon cycle integrated into the biology course instructional design will provide the same results about student learning regardless of the teacher.

26. Theoretical Background Conceptual change theory »Force Concept Inventory (David Hestenes, Physics Dept., ASU)

27. Carbon Cycle = Rich Problem Why? Integrates many biological concepts at multiple scales - ecosystems to molecules. Instruction can return to elements intrinsic in the carbon cycle - bioenergetics, metabolism. Several documented student misconceptions associated with the carbon cycle. Real-world applied consequences if students continue to misunderstand.

28. Some Common Misconceptions about Photosynthesis & Respiration Concept 1: Matter disappears during decomposition of organisms in the soil. Concept 2: Photosynthesis as Energy: Photosynthesis provides energy for uptake of nutrients through roots which builds biomass. No biomass built through photosynthesis alone. Concept 3: Thin Air: CO 2 and O 2 are gases therefore, do not have mass and therefore, can not add or take away mass from an organism. Concept 4: Plant Altruism: CO 2 is converted to O 2 in plant leaves so that all organisms can ‘breathe’. Concept 5: All Green: Plants have chloroplasts instead of mitochondria so they can not respire.

29. Instructional Design Active, inquiry-based learning »Cooperative groups »Questions, group processing, large lecture sections (2 class meetings @80 minutes), 2 discussion sections, multi-week laboratory investigation »Homework problems including web-based modules Different faculty for two courses »One graduate/8-10 undergraduate TAs per course

30. Experimental Design Two introductory courses for majors: »Bio 1 - organismal/population biology (faculty A) »Bio 2 - cell and molecular biology (faculty B) Three cohorts: »Cohort 1 Bio 1 »Cohort 2 Bio1/Bio2 »Cohort 3 Other/Bio2

32. Assessment Design Multiple iterations/versions of the carbon cycle problem Pretest, midterm, final with additional formative assessments during class Administered during instruction »Semester 1 - pretest, midterm, final exam »Semester 2 - final exam

33. Multiple choice question (pre-post) The majority of actual weight (dry biomass) gained by plants as they progress from seed to adult plant comes from which one of the following substances? a. Particle substances in soil that are take up by plant roots. (15%). b. Molecules in the air that enter through holes in the plant leaves (4%). c. Substances dissolved in water taken up directly by plant roots. (28%). d. Energy from the sun (29%). N=138

34. Radish Problem (formative) Experimental Setup: Weighed out 3 batches of radish seeds each weighing 1.5 g. Experimental treatments: »1. Seeds placed on moistened paper towels in LIGHT »2. Seeds placed on moistened paper towels in DARK »3. Seeds not moistened (left DRY) placed in light

35. Radish problem (2) After 1 week, all plant material was dried in an oven overnight (no water left) and plant biomass was measured in grams. Predict the biomass of the plant material in the various treatments. »Water, light »Water, dark »No water, light

36. Results: Weight of Radish Plants 1.46 g1.63 g 1.20 g Write an explanation about the results.

37. Whale Problem (midterm Bio 1) Two fundamental concepts in ecology are “energy flows” and “matter cycles”. In an Antarctic ecosystem with the food web given above, how could a carbon atom in the blubber of the Minke whale become part of a crabeater seal? Note: crabeater seals do not eat Minke whales. In your response include a drawing with arrows showing the movement of the C atom. In addition to your drawing, provide a written description of the steps the carbon atom must take through each component of the ecosystem Describe which biological processes are involved in the carbon cycle.

38. Grandma Johnson Problem (final, Bio 1) Hypothetical scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.

39. Spider Monkey Problem (final, Bio 2) Deep within a remote forest of Guatemala, the remains of a spider monkey have been buried under an enormous mahogany tree. Although rare, jaguars have been spotted in this forest by local farmers. Use coherently written sentences and clearly labeled drawings to explain how a carbon atom in glucose contained within muscle cells of the spider monkey might become part of a cell within the stomach lining of a jaguar. (Note:The jaguar does not dig up the monkey and eat the remains!) Include in your answer descriptions of the key features (not complete biochemical pathways!) of the organismal and cellular processes that explain how the carbon atom of the monkey’s corpse could become a part of the jaguar’s body.

40. Analysis of Responses Used same scoring rubric for all three problems - calibrated by adding additional criteria when necessary, rescoring: Examined two major concepts: Concept 1: Decomposers respire CO 2 Concept 2: Plants uptake of CO 2 Explanations categorized into two groups: Organisms (trophic levels) Processes (metabolic)

41. Trace Carbon from Whale to Seal (Bio1 students, n=141) Organism Process Concept 1 Decomposers respire CO 2 Concept 2 Plants uptake CO 2 Respiration Release CO 2 Primary produces Through AirThrough RootGlucose Photosynthesis 0 20 40 60 80 100 % Decomposers

42. Cellular Respiration by Decomposers (Bio1/Bio2 students, n=63)  2 (2) = 20.16, p < 0.01 Q 1 WhaleQ 2 Grandma JQ 3 Spider Monkey 0 20 40 60 80 100 % Concept 1: Decomposers respire CO 2

43. Pathway of Carbon into Primary Producer (Bio1/Bio2 students, n=63)  2 (2) = 4.778, p =.092 0 20 40 60 80 100 Q 1 WhaleQ 2 Grandma JQ 3 Spider Monkey % Air Root Concept 2: Plants uptake CO 2

44. Trace Carbon from Spider Monkey to Jaguar Respiration NA  2 (1) = 14.59, p <.01 0 20 40 60 80 100 % Bio1/Bio2 (n=63) 0ther + Bio2 (n=40) Concept 1: Decomposers respire CO 2

45. Pathway of Carbon into Primary Producer Concept 2: Plants uptake CO 2 0 20 40 60 80 100 Bio1/Bio2 (n=63) 0ther + Bio2 (n=40) % Air Root NA  2 (1) = 8.89, p < 0.05

46. So What? Problem sets about major concepts: »Diagnostic re: what students understand/misconceptions »Methods: parallel to process in disciplinary research »Learn what prior knowledge students bring to course, what students gained »Make predictions re: student responses about difficult concepts »Unveil new misconceptions »Influenced our teaching for understanding

47. So What? (2) Curricular changes: »Bacteria/Archaea metabolism - often omitted »Primary production - models in lab »Source/Sink and carbon flux »‘Spiral’ major concepts - over/over/over Use of technology: CTOOLS (concept mapping java applet ctools.msu.edu)

48. Multiple Choice … … Concept Maps … … Essay … … Interview high Ease of Assessment low low Potential for Assessment of Learning high Theoretical Framework Ausubel 1968; meaningful learning Novak 1998; visual representations King and Kitchner 1994; reflective judgement National Research Council 1999; theoretical frameworks for assessment Assessment Gradient

50. The real world without C-TOOLs

51. The ideal world with C-TOOLS

52. Which scenario would you bet money on?

55. The Grandma Johnson Problem Hypothetical Scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. She loved the idea that she'd become part of the wonderful wilderness and live on through the wildlife that lived there. Think to yourself and begin to trace the path of a carbon atom from Grandma Johnson's (GJ) remains to where it could become part of a coyote (NOTE: the coyote WILL NOT dig up Grandma and consume any of her remains). What fundamental pathways and processes of biology will be involved in the transit of GJ's carbon atoms to that of the wild coyote in Utah?

56. Task: Create a concept map that illustrates your understanding of the relationship between these 10 concepts in the context of the Grandma Johnson problem. You may add up to 5 extra concepts if you need them to explain the problem more clearly. photosynthesisrespirationcarbon cycle decomposersprimary producersconsumers carbon dioxideglucoseenergy oxygen 1. Work on the problem individually first, save it in CTOOLS, and print a hard copy. 2. Work on the problem with a partner. Both of you can retrieve your concept maps, discuss, revise and produce the best final map to which both of you have contributed. 3. Submit all three maps - yours, your partner's and the FINAL MAP you completed together - Please put the final map on top, with both of your names. Staple them together (5 pts off if not stapled).

64. C-TOOLS Research Question: Is there a correlation between students’ concept map and their written explanation of a problem? Methods: 1. Develop a diagnostic problem 2. Build into instructional design 3. Student complete as homework 4. Develop coding scheme and analyze

65. The Plant Adaptation Problem In this course, we have claimed that land plants were the first truly terrestrial organisms. However, most biologists contend that the saturated soils on land were undoubtedly teaming with bacteria, archaea, and protists long before land plants evolved. In light of this, what does the phrase "truly terrestrial” mean? To answer this question follow these instructions:

66. Task: Make a concept map using the following concepts: adaptationdispersalfitness flowersfruitleaves photosynthesisreproductionroots seedsvascular tissue 2. PRINT one copy for yourself then SUBMIT a copy electronically. 3. Then, using your concept map, write a short response to answer the question by explaining the problems plants had to overcome to live on land and explain the adaptations that allowed plants to overcome those problems. 4. After you finish you short response, print it out and use a highlighter to highlight the statements (propositions) you used directly from your concept map. You should have elaborated (explained) further upon these statements in your written response. 5. Hand in hard copy of (a) concept map and (b) written, highlighted answer.

67. Key Concepts - Plant Adaptation Problem Key Concepts in Extended ResponseLinks on Concept Map 1. Adaptation increases fitness.adaptation / fitness 2. Fitness “is” reproduction.fitness / reproduction 3. Vascular tissue is an adaptation for getting water on land. vascular tissue / roots 4. Leaves are and adaptation for photosynthesis on land. leaves / photosynthesis 5. Flowers are an adaptation for reproduction on land. flowers / reproduction 6. Seeds are an adaptation for dispersal on land. seeds / dispersal

68. Coding Criteria Each key concept coded correct [C] incorrect [I] non-informative [N] absent [A] for both concept map and extended response. Each key concept counted as a Match:same coding on map and writing or No match: different coding on map and writing

69. Matches Between Concept Map & Extended Response Plant Adaptation Problem Upper 25% of Students n=44

70. Concept Present: Both C-Map & Extended Response Plant Adaptation Problem Upper 25% of Students n=44

71. Non-matches: Concept Included on C-map or Extended Response Only Plant Adaptation Problem

72. So Who? IRD teams in the disciplines -- Intergenerational Research Development teams Use the process they know best to gather meaningful data about student learning to guide the direction of undergraduate education.