1. Assessing Critical Thinking in the Science Classroom Paula Lemons, Duke University October 7, 2006

2. Why the skepticism?  Time  Nebulous definitions of critical thinking (CT).  Fear of wasting students’ time and content coverage.  Lack of know-how in educational theory and practice.

3. Key Insight #1  Choose a CT definition – from the myriad definitions - that works for you (i.e., your course and learning objectives).

4. Our choice: Bloom’s Taxonomy of Educational Objectives*  Basic knowledge - memorizing facts, figures and basic processes.  Secondary comprehension - understanding and illustrating the facts.  Application - generalizing the facts to other contexts and situations.  Analysis - understanding why the facts are the way they are, breaking problems down.  Synthesis - making connections between different elements on your own.  Evaluation - critically using your knowledge to ascertain the quality of information. *Taken from Bloom, B. S., ed. 1956. Taxonomy of Educational Objectives: The Classification of Educational Goals. McKay, New York.

5. Rationale for Using Bloom’s  Already used it informally as a guide for question-writing and course design.  Upon further examination, we found that it captured what we were trying to get students to do – learn and comprehend basic knowledge, and then use that knowledge to solve novel biological problems.

6. Key Insight #2  Questions can simultaneously require both Content knowledge/understanding and CT skill.  Thus, assessment of these questions can provide a measure of both Content knowledge/understanding and CT skill.

7. Writing CT Questions  Come up with a scenario that requires students to apply the vocabulary, facts, and/or concepts to a novel situation (i.e., one not covered in lecture, lab or written course materials).  Devise the question so that students have to do something with this scenario – e.g., analyze it, evaluate it (see Bloom’s Taxonomy).  Test the question on TAs, a colleague, etc. Judge timing and clarity, but, more importantly, get a second opinion about both the CT skill and content knowledge that are being assessed with the question.  Revise as needed.

8. Examples of Fully Developed CT Questions  Bissell and Lemons (2006). “A New Method for Assessing Critical Thinking in the Classroom” BioScience 56(1), 66-72.

9. Building a CT Question - Iteration #1 This figure indicates causal relationships between variables affecting the abundance of four size classes of a tree species in a particular region. Solid arrows indicate positive relationships, dashed arrows indicate a negative relationship. The numbers next to the arrows indicate the “coefficient of influence” between variables: the larger the number, the greater the influence. All paths shown are significant; insignificant paths were omitted from the figure. Identify the variable that has the greatest influence on abundance at each size class and, based on ecological concepts we have discussed in class, explain WHY this variable has the greatest effect.

10. Building a CT Question – Critiquing It  This question provides a novel scenario.  But it doesn’t ask students to think critically. Rather, asks only for interpretation of the figure.  Also, the content component can be strengthened.

11. Building a CT Question – Iteration #2  “Researchers interested in this system decided to try three different ‘removal experiments’ for some plots that were broadly similar to each other and to the diagram shown above. In one plot, all of the seedlings were removed. In the second plot, artificial lighting was used to increase the light exposure for all size-classes by 50%. In the third plot, all of the deer were removed. Which of these experiments is likely to have the greatest effect on the future abundances of the four size classes? Explain (and be sure to include mention of all three experiments).”

12. Building a CT Question – Critiquing Again  This iteration uses the novel scenario.  Now it asks students to think critically – to analyze and evaluate.  Problem: No content knowledge is needed.

13. Building a CT Question – Iteration #3  “Examine the scenario provided. For which of the four size classes of trees is competition the major factor affecting abundance? Explain your answer.”

14. Building a CT Question – Critiquing One More Time  This iteration uses the novel scenario.  It now requires specific content – knowing and understanding the definition of “competition.”  It still requires CT – applying the knowledge of competition to the scenario; analyzing each of the factors that affect abundance.

15. Building CT Questions - Can It Be Done?  Instructors’ question banks are rich with questions in the process of becoming CT questions. These questions can be revised to require specific content and CT skills.  We are working on the design of templates for CT question-writing that would speed the process.

16. Writing CT Rubrics  State the specific content knowledge required to answer the question.  State what critical-thinking skill is required. Use Bloom’s taxonomy as a guide.  Outline a complete answer.  Create a graphical map that shows the logical ways that students could approach the question, whether or not they get all of the content knowledge correct.  Get a second opinion on your rubric from a colleague.  Revise as needed.

17. Examples of Fully Developed CT Rubrics  Bissell and Lemons (2006). “A New Method for Assessing Critical Thinking in the Classroom” BioScience 56(1), 66-72.

18. Building CT Rubrics – Stating the Knowledge and Skills Required  Content required: Definition and understanding of competition.  CT skill required: Application of the definition of competition Analysis of each of the variables affecting each of the four size classes of trees in order to determine which size class is most affected by competition.

19. Building CT Rubrics – Outlining a Complete Answer I. State the definition of competition A. Competition occurs when two organisms utilize the same resource(s), thus limiting each other’s use of that resource.

20. Building CT Rubrics – Outlining a Complete Answer I. Analysis of the different variables by applying the definition to them. A. Only two size classes are affected by a variable that is a resource. The abundance of both seedlings and small saplings is affected by light – necessary for the plants to make food for themselves. B. Knowing that light is a limited resource, we can assume that individual seedlings and individual small saplings limit each others’ access to light. Thus, both seedlings and small saplings are impacted by competition. C. However, the numbers provided show that small saplings are more strongly impacted by seedlings than by light. D. Since seedlings are only impacted by light and since the magnitude of this effect is greater for seedlings than small saplings, we can infer that they are the class of trees whose abundance is most strongly affected by competiton.

21. Building CT Rubrics – Creating a Graphical Map

22. Question Analysis 1. Choose a question that you think asks students to think critically. 2. Ask yourself what content knowledge is needed to answer the question. 3. Compare the question to Bloom’s taxonomy – Does your question require higher-order thinking? If so, what type of skill is required? How so? 4. Map out the logical solutions to the problem – those using correct knowledge and skillful CT AND those using incorrect knowledge/misconceptions and skillful CT. 5. Write me (plemons@duke.edu) and tell me what problems you ran into, whether or not this is useful, etc.plemons@duke.edu

23. Can it be done?  Ways that we’ll know: Is inter-rater reliability on CT questions higher with our rubrics than without? Is our methodology generalizable to other faculty in different contexts? (Please help us learn  feedback to plemons@duke.edu) plemons@duke.edu Does assessing CT improve student CT skills within a discipline? across disciplines?

24. Special Thanks to...  Dr. Ahrash Bissell, Duke University  Dr. Jeremy Hyman, Western Carolina University  Dr. Julie Reynolds, Duke University  This work has been supported by Arts and Sciences Council on Faculty Research, Duke University. Trinity College of Arts and Sciences, Duke University.