Research Strategies for Assessment of Learning in Large Classes Diane Ebert-May Department of Plant Biology Michigan State University /home.html

Anonymous (Change Magazine, 2001): “I believe we would all agree that the absolute best teaching learning-assessment model is the one-on-one Socratic apprenticeship model with unlimited time with the student. But ever since Socrates took on two students rather than only one (to double his income), teachers have had to make compromises in teaching.”

Question 1 How important is it to use multiple kinds of data to assess student learning? Please respond on a scale if in increments of 10:

How important is it to use multiple forms of data to assess student learning? % Relative Importance n=127

Question 2 How often do you use data to make instructional decisions? Please respond on a scale of in increments of 10:

How often do you use data to make instructional decisions? n=127 Frequency %

True or False? Assessing student learning in science parallels what scientists do as researchers.

1. Description: -What is happening? 2. Cause: -Does ‘x’ (teaching strategy) affect ‘y’ (understanding)? 3. Process or mechanism: -Why or how does ‘x’ cause ‘y’? Parallel: ask questions

We collect data to find out what our students know. Data helps us understand student thinking about concepts and content. We use data to guide decisions about course/curriculum/innovative instruction Parallel: collect data

Quantitative data - statistical analysis Qualitative data break into manageable units and define coding categories search for patterns, quantify interpret and synthesize Valid and repeatable measures Parallel: analyze data

Ideas and results are peer reviewed - formally and/or informally. Parallel: peer review

Q3. What is assessment? Data collection with the purpose of answering questions about… students’ understanding students’ attitudes students’ skills instructional design and implementation curricular reform (at multiple grainsizes)

Research Methods

Why do assessment? Improve student learning and development. Provides students and faculty substantive feedback about student understanding. Challenge to use disciplinary research strategies to assess learning. (Ebert-May et al 2003)

Data collection approaches

Multiple Choice … … Concept Maps … … Essay … … Interview high Ease of Assessment low low Potential for Assessment of Learning high Theoretical Framework Ausubel 1968; meaningful learning Novak 1998; visual representations King and Kitchner 1994; reflective judgment National Research Council 1999; theoretical frameworks for assessment Assessment Gradient

Tools for Data Collection Items Students Purchase: Course Pack - at Ned's and the College Store on Hagadorn ISB202 - Section 7 Drs. Ebert-May and Linton Carbonless Lab Notebook - 8½" x 11" - at the College Store and the Student Bookstore can be graph paper or lined Student Response Pad - $6 - only at Ned's Register your transmitter Cost $15 - you'll need a credit card You need this number to register: Class Key: S2349Q361 Stapler (mini or regular sized) - Bring to class each day

Research Question: Do students’ perceptions of the value of cooperative learning change throughout this course? Design Experiment: Course over time (15 weeks) Pre-post: in-class inquiries/exam questions Data type: Indirect: self-report from “clickers” Direct: individual /group assessment

How often did you work in cooperative groups in high school?

How often have you worked in cooperative groups in college?

How much do you think cooperative learning will improve your understanding?

How much do you think cooperative learning has improved your understanding thus far?

How much do you think cooperative learning improved your understanding about the frog deformity problem?

Hierarchy has Structure has Concept Maps Visual Diagrams Or Models are represent Knowledge or Understanding Concepts display connected with Linking Words Used for AssessmentOrganization Reflection & Learning promotes Context is constructed with New Information Prior Knowledge

Tool: Concept Maps Reliability test-retest reliability not a concern inter-rater reliability (do people scoring maps agree) Validity face validity - represent meaningful learning: progressive differentiation, integration construct validity - do maps correlate with other measures of learning?

C-TOOLS

Tool: Bloom’s Rankings Use to classify assessment questions (1) Knowledge (2) Comprehension (3) Application (4) Analysis (5) Evaluation (6) Synthesis

Concept: Carbon Cycle Goals: Students will be able to … Trace the pathway of a carbon atom through living and nonliving components of an ecosystem, explaining both the processes and types of organisms involved.

Pre-test Which of the following substances has mass? Answer yes or no in the blank to the left. H 2 0 vapor _______ CO 2 _______ Glucose (C 6 H 12 O 6 ) _______ O 2 _______

Pre-test What are the different pathways that carbon can take once it is inside a plant? Select as many as apply a. it can exit the plant as CO 2 b. it can become part of the plant’s cell walls, protein, fat, DNA c. it can be consumed by an insect feeding on the plant and become part of the insect’s body d. as a plant decomposes in the soil, carbon can exit the soil as CO 2

Pre-test The majority of the actual weight (dry biomass) gained by plants as they progress from seed to adult plant comes from which of the following substances? a. molecules in the air that enter through the leaves b. particle substances in the soil taken up by the roots c. substances dissolved in water that are taken up by roots d. energy from the sun

Radish Experiment Experimental Setup: Weighed out 3 batches of radish seeds each weighing 1.5 g. Experimental treatments: 1. Seeds placed on moistened paper towels in LIGHT –2. Seeds placed on moistened paper towels in DARK –3. Seeds not moistened (left DRY) placed in light After 2 weeks, all plant material was dried in an oven overnight (no water left) and plant biomass was measured in grams. Predict the biomass of each treatment.

Results 1.46 g 1.63 g 1.20 g Write an explanation of the results.

Grandma Johnson Problem Hypothetical Scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon,Utah where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. She loved the idea that she'd become part of the wonderful wilderness and live on through the wildlife that lived there. Think to yourself and begin to trace the path of a carbon atom from Grandma Johnson's (GJ) remains to where it could become part of a coyote (NOTE: the coyote WILL NOT dig up Grandma and consume any of her remains). What fundamental pathways and processes of biology will be involved in the transit of GJ's carbon atoms to that of the wild coyote in Utah?

Select a question you want to answer in related to the CCLI innovations in your project....research design?...data collection tools?...analysis?...journal? In Groups of 3 or 4

Does an active, inquiry-based instructional design affect students’ understanding of evolution and natural selection? Method of analysis: Pre-post test Question:

■ Changes in a population occur through a gradual change in individual members of a population. ■ New traits in species are developed in response to need. ■ All members of a population are genetically equivalent, variation and fitness are not considered. ■ Traits acquired during an individual’s lifetime will be inherited by offspring. From literature and our research: Alternative Conceptions about Natural Selection

Instructional Design Cooperative groups in class: Guppy Problem: sexual vs. natural selection ry_activities/guppy_activity.htm - PBS film -Simulation -Analyze data -Written explanation

(AAAS 1999) Explain the changes that occurred in the tree and animal. Use your current understanding of evolution by natural selection.

Misconception: individuals evolve new traits % of Students n=80; p<.01

Misconception: evolution is driven by need % of Students n=80; p<.01

In guppy populations, what are the primary changes that occur gradually over time? a. The traits of each individual guppy within a population gradually change. b. The proportions of guppies having different traits within a population change. c. Successful behaviors learned by certain guppies are passed on to offspring. d. Mutations occur to meet the needs of the guppies as the environment changes. Anderson et al 2002

Posttest: Student responses to mc % of Students n=171 *

Animal/Tree Posttest: Gain in student understanding of fitness % of Students n=80; p<.01

Quantitative Data Qualitative Data Design Experiment Ebert-May et al Bioscience

How do assessment questions help us determine students’ prior understanding and progressive thinking about the carbon cycle. Question

Instructional Design Two class meetings on carbon cycle (160 minutes) Active, inquiry-based learning – Cooperative groups – Questions, group processing, large lecture sections, small discussion sections, multi-week laboratory investigation – Homework problems including web-based modules Different faculty for each course – One graduate/8-10 undergraduate TAs per course

Experimental Design Two introductory courses for majors: Bio 1 - organismal/population biology (faculty A) Bio 2 - cell and molecular biology (faculty B) Three cohorts: Cohort 1 Bio 1 (n=141) Cohort 2 Bio1/Bio2 (n=63) Cohort 3 Other/Bio2 (n=40)

Assessment Design Multiple iterations/versions of the carbon cycle problem Pretest, midterm, final with additional formative assessments during class Administered during instruction Semester 1 - pretest, midterm, final exam Semester 2 - final exam

Grandma Johnson Problem Hypothetical scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.

Analysis of Responses Used same scoring rubric (coding scheme) for all three problems - calibrated by adding additional criteria when necessary, rescoring: Examined two major concepts: Concept 1: Decomposers respire CO 2 Concept 2: Plants uptake of CO 2 Explanations categorized into two groups: Organisms (trophic levels) Processes (metabolic)

Coding Scheme

Correct Student Responses (%) Cellular Respiration by Decomposers Bio1/Bio2Other/Bio2 Friedmans, p<0.01

Pathway of Carbon in Photosynthesis Bio1/Bio2 Correct Student Responses (%) Other/Bio2 Friedmans, p<0.05

IRD Team at MSU Janet Batzli - Plant Biology [U of Wisconsin] Doug Luckie - Physiology Scott Harrison - Microbiology (grad student) Tammy Long - Plant Biology Jim Smith - Zoology Deb Linton - Plant Biology (postdoc) Heejun Lim - Chemistry Education Duncan Sibley - Geology *National Science Foundation

What is the question? What research and instructional designs? What data collection methods? How to analyze and interpret data? Are findings valid and generalizable? What are the next questions? WHO? What evidence will we accept?