1. Iglika Pavlova, University of Chicago, Chicago, IL Improving Understanding of Science by Teaching Explicit Principles of Good Reasoning in an Evolution and Intelligent Design Course Iglika Pavlova BSLC Rm. 208 924 E. 57th St. Chicago, IL, 60637 773-702-0683 iglikap@uchicago.edu To improve scientific understanding, a successful group of current approaches involves increased access to authentic science experiences. This study reports on a fundamentally different approach that emphasizes explicitly learning principles of good reasoning, in a net cast wider than specific disciplinary examples, and that is informed by philosophy of science, probability, and logic. The comparison between evolution and Intelligent Design (ID) serves as a springboard for inquiry into issues such as justification and reasoning with uncertainty. The hypothesis tested in Fall 2009 and Winter 2010 trials is that by the end of the course students will be able to: 1.Identify and explain features of good explanations 2.Correct misconceptions about science and justification Major conceptsCourse philosophy The guiding principles are that students should: Be exposed to multiple viewpoints Obtain tools for critical assessment by explicitly learning concepts in epistemology and philosophy of science. Apply concepts learned to real-life issues, both for the topic of the class and outside, and on multiple occasions Critically assess the issues and make up their own minds through continuous reflection Class combines original readings, lectures, discussions, and short essay responses to prompts (see Methods) Misconceptions are identified and explicitly addressed in the course. Major concepts that are commonly associated with misconceptions are marked with a * in the box to the left. Pre/Post Surveys Surveys were given on the second day of class (Pre) and after 10 weeks, during the In-class Final Exam (Post) in 3 parts: 1.Science attitudes and understanding (19 items) 2.Epistemic understanding (54 items) 3.Case scenario (on astrology) Items for part I (except one) and II are on an extended 7-point Likert scale (Strongly agree +3, +2, +1, 0 No opinion/Don’t know, -1, -2, -3 Strongly disagree) Each concept tested with 2 or more items, phrased in both affirmative and negative terms, in a randomized order Survey created over the course of several years, with extensive colleague and student feedback, to fill need for deeper testing for epistemic concepts that seems to be currently unavailable Short Essay Responses Short essays were included in several types of assignments: Worksheets - in class (typically small group work) and homework (typically individual work) Reflections at the start, middle, and end of the quarter (part of the Final Exam) Final Exam - In-class and Take-home Same concept tested 2 or more times throughout the quarter Same concept tested in 2 or more different ways: Reflection: What do you think/feel? Compared to before? Argument: Argue for a specific (given) point Response to actual or real life-like argument: How would you respond? What would you say? RESULTS Science Attitudes – Survey & Essay Responses Survey item% Agree% Disagree% No opinion PREPOSTPREPOSTPREPOST + Science helps us understand how the world actually works 93.198.20.01.83.40.0 - Science does not really tell us about how the world is 8.65.582.892.78.61.8 + Science gets to the truth more often than not 72.483.612.10.015.516.4 - What scientists think in their field of research is determined largely by the era that they live in, rather than truths that are independent of time and place 56.920.034.578.28.61.8 - Most of the time, when science contradicts your personal experience about medical issues it is best to trust your personal experience 36.212.755.287.18.60.0 + A scientific theory is more than just a guess 100.0 0.0 - A scientific theory is just a possible explanation of how things are 56.920.036.278.26.91.8 + Scientific method is more rigorous than everyday reasoning, and that is why it gives more reliable results with regard to issues of the natural world 75.994.513.83.610.31.8 + Scientific reasoning is some of the best reasoning we (humans) have 82.898.212.10.05.21.8 “ A scientific theory is just a possible explanation of how things are” "Most of the time, when science contradicts your personal experience about medical issues it is best to trust your personal experience" A. B. Scientific Understanding – Essay Responses 3 common problems with bad explanations 3/32/31/3 79%21%0% ExcellentGoodPoor/Not acceptable Explanations are clear, well-supported by specifics, no inaccuracies Explanations are vague and/or contain some inaccuracies No explanation or inaccurate explanation 48%47%5% Alternative explanation criterion 5 major criteria studied in class (excepting alternative explanations criterion) 5/54/53/52/51/5 63%61%13%18%5%3% ExcellentGoodPoor/Not acceptable Explanations are clear, well-supported by specifics, no inaccuracies Explanations are vague and/or contain some inaccuracies No explanation or inaccurate explanation 32%63%5% Table 4. What are common problems with bad explanations? Table 1 Table 3. What are features of good explanations? Table 3. Analysis of responses to the In-class Final Exam question “How can you tell if an explanation is correct?”, asking to identify and explain the features of good explanations that tend to increase their accuracy. Results presented from Winter 2010 quarter. Table 4. Analysis of responses to the In-class Final exam question “What are common “red flags” for a bad explanation?”. Results presented from Winter 2010 quarter. Question based on analysis of several case studies of poor explanations from class readings and discussions A. B. A. B. METHODSINTRODUCTION 1.Epistemological fundamentals. Issues in forming explanations.  The relationship between knowledge, truth, justification, and belief. Ethics of belief.  Issues in justification, including: Evidence and belief come in degrees * Withholding belief can be a justifiable option * Multiple explanations can account for the same observation * Better explanations can be identified in comparison with alternatives through their higher explanatory power, well- supported mechanism, lack of ad hoc hypotheses, etc. 2.Worldviews: Scientific, religious, skeptical, relativistic Scientific evidence is more reliable than personal experience * Social aspects of science can be a strength; aspects of justification can go beyond culture, historical period, etc.*

2. A different approach Learning explicitly principles of reasoning while applying to several real life cases, in a discussion with peers and with an opportunity for feedback, is an approach that has high potential to lead to more stable and deeper knowledge, and one that can be applied in real life and transferred to other fields Addressing explicitly underlying assumptions, misconceptions and other considerations (such as worldviews) that are not typically addressed in formal coursework may also be very important for a true change in beliefs and lifelong learning Reflective Essay Responses Selected quotes from reflective essay responses from the Take-home Final Exam (from Fall 2009 and Winter 2010 quarters)  I am now aware of how science is epistemologically distinguished from other pursuits. I can appreciate and value science as a strong mode of inquiry that emphasizes warranted evidence. Because I see that science is an epistemologically distinguished pursuit, I understand why scientific reasoning is an essential component in the search for truth. The correlation here is that epistemologically distinguished endeavors are more likely to yield the truth. Before this course I considered most of my interests to lie outside of science. For example, most of my academic interests are centralized around the humanities. I did not consider science or scientific reasoning, for that matter, to be particularly relevant to me or my pursuits. Prior to this course, if asked to describe scientific reasoning, I likely would have referenced the standardized method science so often employs. I could not have told you, however, what is particularly distinguished about science’s methods. I now feel confident that I can assess my own methods when reaching conclusions and forming beliefs. Following this course I am excited to apply epistemologically distinguished methods to my pursuits in other academic fields, such as the humanities and religious studies.  At the onset of this course, I would have readily accepted Intelligent Design models because they corresponded with my belief in God. Now, however, I can now distinguish a good model from a bad model (through applying Giere’s expanded framework.) I recognize that the models employed by the ID movement are weak and deficient (an occurrence caused largely by the ID movement’s failure to specify the properties of a designer.) Additionally, they provide no exact mechanisms to explain their hypothesis. My faith in God has never hinged on how the world was created, but I assumed that since I believe in God I should just accept models that advocate a designer (no matter how flawed they are.) Following this course, I do not believe that science and religion are mutually exclusive, or that you can only have one or the other. I feel perfectly confident in maintaining a belief in God along with an interest and acceptance of evolutionary biological principles.  I did spend a lot of time on scientific topics in school – physics, biology, etc. However, the idea of scientific inquiry was simply something I learned early on without focusing on its meaning. I knew the process of hypothesis, data collection. In my mind, I always viewed science as a group of specific areas of study. When someone asked me if I liked science, I equated the question with whether or not I liked biology, chemistry, etc. After this class, however, and especially after reading Giere’s chapter, I have come to appreciate more science as a theory… It [science] is a method of investigation that can be applied everywhere. One does not need science to be scientific. The scale of evidence/belief is a way to scientifically approach problems in life, and it really is useful. For arguments, I will now attempt to use this tool, so as not to believe anything without justification, or cite as evidence things that aren’t.  When I first started this class, I thought that you had to believe something 100% or not believe it at all. If there were equally strong reasons for and against an idea, then I thought you still had to choose a side. I believe in objective truth, and thought that I needed to have a strong opinion on every issue. Now I still believe in objective truth, but I understand… that doesn’t mean that I have to be 100% sure of what it is… If there is very little evidence for a claim, then I should have very little belief for the claim. Table 1. Summary for Likert scale responses for 9 of the survey items focusing on science as a truth-finding endevor. Results presented from Fall 2009 and Winter 2010 quarters. Data for all “Agree” (+3, +2, +1) and “Disagree” (-3, -2, -1) responses are presented as a sum, along with the 0 response for “No opinion/Don’t know” Survey items are labeled as + for those associated with a positive attitude toward science and/or science as truth-finding, and -- for those associated with a negative attitude and/or limiting the role of science as truth-finding The Post-Survey responses are color-coded red if there is an increase compared to the Pre-Survey, blue if there is a decrease, and black if there is no change Figure 1. Detailed results for the 7-point Likert scale responses for 2 of the survey items from Table 1 showing a significant change after the course. Results presented from Fall 2009 and Winter 2010 quarters. From the data presented here and additional data not presented, it seems that most students in this group come in with an overall positive attitude toward science (even though most plan to declare or have declared a non- science major and informally talk about themselves as “not science” people) However, many students display doubts about science in several ways or in certain situations that tend to diminish or discount science as truth-finding, such as: – Thinking of scientific results as primarily influenced by financial and political interests as well as cultural and individual biases – Considering personal evidence as more reliable than scientific evidence, even for complex cases such as efficacy of medicines In addition, while many students have a general idea about the “scientific method” and understand some of the specific considerations, they are missing important components and hold some misconceptions After the course, students increase their positive attitude toward science for all relevant Survey items The Survey (which is anonymous) also indicates that students are changing their minds about issues they had misconceptions about. In particular, students feel more confident in science as a truth-seeking and truth- finding process (Tables 1 and 2 and Figure 1). Essay responses identify that this confidence comes from a more complete and deeper understanding of the process of science, specifically components of good explanations that increase the likelihood of truth- finding (Tables 3, 4, and 5 and reflective essay responses). CONCLUSIONS Future Direction ExcellentGoodAcceptablePoor/Not acceptable Argument links models with truth- indicativeness and explains link well Argument links models with truth- indicativeness, does not explain link well Makes good argument but does not identify link No argument or argument does not address question 63%26%9%2% Table 5. “Science is just a lot of models – there is no truth in science” Table 5. Analysis of responses to the In-Class Final Exam Response prompt “Someone says to you: “Science is just a lot of models – there is no truth in science”. How do you respond?”. Results reflect % of students who provide excellent, good, acceptable, or not acceptable answers. Results presented from Fall 2009 and Winter 2010 quarters. A. % of students who describe that a good explanation must be compared to alternatives (first column) and who describe 1 to 5 of the 5 major criteria studied in class for choosing a good explanation that is also best among its alternatives (next columns): 1.More supporting evidence and less disconfirming evidence 2.More well supported supplemental hypotheses and less ad hoc hypotheses 3.Well supported mechanism 4.Genuine higher predictive strength 5.Genuine higher explanatory power B. % of students who provide excellent, good, or not acceptable explanations of the criteria they list. A clear and specific explanation of the criteria is linked to understanding how a criterion is linked with the accuracy of explanations Table 2. Results reflect % of students choosing their top 2 choices in answer to the question “What do you think are the most common reasons when scientists disagree on a scientific issue (.e.g., whether global warming is a significant threat or whether artificial sweeteners are safe to eat)?”. Results presented from Fall 2009 and Winter 2010 quarters. Possible reasonsPREPOST The evidence is not enough, and different scientists have different interpretations of the existing evidence 7895 Different scientists are thinking about different aspects of the issue 4060 Some scientists have financial or political interests in promoting a certain conclusion 2915 Scientists, like all people, have their own personal opinions 225 There is no one truth about the matter, each different viewpoint is relative to a particular way of understanding the issue 225 Different scientists were brought up in different ways 95 Table 2. Why do scientists disagree? Acknowledgments Kayla Lewis, co-developer and co-teacher of these courses Biological Sciences Collegiate Division, and especially Jose Quintans ASM Research Residency Scholars Program, especially our leaders! Incoming attitudes and knowledgeAfter the course Continue to improve learning in this course! Complete analysis for Fall 2009 and Winter 2010 quarters Continue with the studies (longitudinal study; research other questions, such as role of religious belief) Improve accuracy of the tools for this study by improving existing tools, in collaboration with specialists in the field Share teaching method with other faculty Expand teaching method to other populations (beyond 2 pilots, with high school students and grad students/postdocs)