1. Assessing the Impact of Integrating Scientific Research and Education
What evidence will we accept?
Diane Ebert-May
Michigan State University
Celia Evans
Paul Smith’s College

2. Anonymous (Change Magazine, 2001):
“I believe we would all agree that the absolute best teaching learning-assessment model is the one-on-one Socratic apprenticeship model with unlimited time with the student. But ever since Socrates took on two students rather than only one (to double his income), teachers have had to make compromises in teaching.”

3. Please respond on a scale if 0-100
Question 1
Please respond on a scale if 0-100
in increments of 10:
How important is it to use multiple kinds of data to assess student learning?

4. How important is it to use multiple forms of data to assess student learning?%
Relative Importance
n=127

5. Please respond on a scale of 0 - 100
Question 2
Please respond on a scale of
in increments of 10:
How often do you use data to make instructional decisions?

6. How often do you use data to make instructional decisions?%
Frequency
n=127

7. True or False?
Assessing student learning in science parallels what scientists do as researchers.

8. Parallel: ask questions
Description:
-What is happening?
Cause:
-Does ‘x’ (teaching strategy) affect ‘y’ (understanding)?
Process or mechanism:
-Why or how does ‘x’ cause ‘y’?

9. Parallel: collect data
We collect data to find out what our students know.
Data helps us understand student thinking about concepts and content.
We use data to guide decisions about course/curriculum/innovative instruction

10. Parallel: analyze data
Quantitative data - statistical analysis
Qualitative data
break into manageable units and define coding categories
search for patterns, quantify
interpret and synthesize
Valid and repeatable measures

11. Parallel: peer review
Ideas and results are peer reviewed - formally and/or informally.

12. Data collection with the purpose of answering questions about…
What is assessment?
Data collection with the purpose of answering questions about…
students’ understanding
students’ attitudes
students’ skills
instructional design and implementation
curricular reform (at multiple grainsizes)

13. Improve student learning and development.
Why do assessment?
Improve student learning and development.
Provides students substantive feedback about their understanding.
Challenge to use disciplinary research strategies to assess learning.

14. Research Methods

15. Data collection approaches

16. Assessment Gradient
low Potential for Assessment of Learning high
Multiple Choice … … Concept Maps … … Essay … … Interview
high Ease of Assessment low
Theoretical Framework
• Ausubel 1968; meaningful learning
• Novak 1998; visual representations
• King and Kitchner 1994; reflective judgment
• National Research Council 1999; theoretical frameworks
for assessment

17. Pre-Posttest Analysis
Does active, inquiry-based instructional design influence students’ understanding of evolution and natural selection?

18. Alternative Conceptions:
Natural Selection
■ Changes in a population occur through a gradual change in individual members of a population.
■ New traits in species are developed in response to need.
■ All members of a population are genetically equivalent, variation and fitness are not considered.
■ Traits acquired during an individual’s lifetime will be inherited by offspring.

19. Cooperative groups in class:
Instructional Design
Cooperative groups in class:
Guppy Problem:
sexual vs. natural selection
-PBS film
-Simulation
-Analyze data
-Written explanation

20. Explain the changes that occurred in the tree and animal
Explain the changes that occurred in the tree and animal. Use your current understanding of evolution by natural selection.
(AAAS 1999)

21. Misconception: individuals evolve new traits
% of Students

22. Misconception: evolution is driven by need
% of Students

23. In guppy populations, what are the primary changes that occur gradually over time?
The traits of each individual guppy within a population gradually change.
The proportions of guppies having different traits within a population change.
Successful behaviors learned by certain guppies are passed on to offspring.
Mutations occur to meet the needs of the guppies as the environment changes.
Anderson et al 2002

24. Posttest: Student responses to mc*
% of Students

25. Animal/Tree Posttest: Gain in student understanding of fitness
% of Students

26. Quantitative Data
Qualitative Data
Design Experiment
Bioscience 2003

27. Question
How do assessment questions help us determine students’ prior understanding and progressive thinking about the carbon cycle.

28. Instructional Design Two class meetings on carbon cycle (160 minutes)
Active, inquiry-based learning
Cooperative groups
Questions, group processing, large lecture sections, small discussion sections, multi-week laboratory investigation
Homework problems including web-based modules
Different faculty for each course
One graduate/8-10 undergraduate TAs per course

29. Experimental Design Two introductory courses for majors:
Bio 1 - organismal/population biology (faculty A)
Bio 2 - cell and molecular biology (faculty B)
Three cohorts:
Cohort 1 Bio 1 (n=141)
Cohort 2 Bio1/Bio2 (n=63)
Cohort 3 Other/Bio2 (n=40)

30. Assessment Design
Multiple iterations/versions of the carbon cycle problem
Pretest, midterm, final with additional formative assessments during class
Administered during instruction
Semester 1 - pretest, midterm, final exam
Semester 2 - final exam

31. Grandma Johnson Problem
Hypothetical scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.

32. Analysis of Responses
Used same scoring rubric (coding scheme) for all three problems - calibrated by adding additional criteria when necessary, rescoring:
Examined two major concepts:
Concept 1: Decomposers respire CO2
Concept 2: Plants uptake of CO2
Explanations categorized into two groups:
Organisms (trophic levels)
Processes (metabolic)

33. Coding Scheme

34. Cellular Respiration by Decomposers
Correct Student Responses (%)
Bio1/Bio2
Other/Bio2
Freidmans, p<0.01

35. Pathway of Carbon in Photosynthesis
Correct Student Responses (%)
Bio1/Bio2
Other/Bio2
Freidmans, p<0.05

36. Is Graduate Education Similar?
Often excellent at preparing individuals to design and carry out disciplinary research.
Often inadequate and haphazard in preparing future faculty/professionals to take on the increasingly complex demands of the professoriate.
Teaching is not mentored, peer reviewed, or based on accumulated knowledge.

37. Instructional Research and Development teams (IRDs)
Who: senior faculty, junior faculty, postdoctoral and graduate students - intergenerational teams.
What: scholarship of science teaching and learning is fully integrated into the professional culture along with discipline- based activities.
Assessment is critical to both practices.

38. IRD Team at MSU Janet Batzli - Plant Biology [U of Wisconsin]
Doug Luckie - Physiology
Scott Harrison - Microbiology (grad student)
Tammy Long - Plant Biology
Jim Smith - Zoology
Deb Linton - Plant Biology (postdoc)
Heejun Lim - Chemistry Education
Duncan Sibley - Geology
*National Science Foundation

39. What evidence will we accept?
What is the question?
What research and instructional designs?
What data collection methods?
How to analyze and interpret data?
Are findings valid and generalizable?
What are the next questions?
WHO?
What evidence will we accept?