Natural Sciences Assessment At Kennedy-King College Fall 2014 DRAFT - FOR DISCUSSION ONLY 1 Prepared by Robert Rollings, spring 2015

DRAFT - FOR DISCUSSION ONLY 2 In fall 2014, Kennedy-King College administered an assessment of natural sciences learning to 1,370 student respondents in a wide sample of classrooms in various subjects. The test design progressed from questions requiring basic reading comprehension of science passages to questions demanding critical thinking and application of concepts. The assessment designers hypothesized that students would perform well with basic comprehension and poorly with critical application. With these results, we test that hypothesis.

DRAFT - FOR DISCUSSION ONLY 3 The sample was demographically typical of KKC, except for a disproportionate representation of full-time students.

DRAFT - FOR DISCUSSION ONLY 4 The assessment was administered in a broad array of classes, but in a disproportionate number of biology classes. Thus we see the disproportionate number of respondents who report having taken a class in biology. We might expect higher overall performance on biology questions than if the survey were evenly distributed.

DRAFT - FOR DISCUSSION ONLY 5 Students broadly agree with the value of science but are more likely to agree strongly that it is important for their careers than that they find it personally interesting.

DRAFT - FOR DISCUSSION ONLY 6 About a quarter of respondents have not taken a science class at KKC, and only 20.8% of the sample had taken no college science courses at KKC or elsewhere. 37% of the sample had taken more than one course in the sciences. If we have been achieving our SLOs, an overwhelming majority of respondents should have mastery of scientific method and basic concepts.

DRAFT - FOR DISCUSSION ONLY 7

8 Questions 1-5 measured ability to describe the scientific method and hypothesis testing. Most students performed well on this section. People who believe that science leads to inarguable facts (or worse, is “just a theory”) may be unlikely to engage critical thinking when engaging scientific findings, and still less likely to perform science themselves. Oddly, having taken a KKC science course makes no significant difference in correct response rates to this question! However, a large minority of students show confusion about the basic nature of scientific inquiry.

DRAFT - FOR DISCUSSION ONLY 9 Questions 6-10 measured conceptual mastery of precision, accuracy, and significant digits, then on application of a basic formula (Density=Mass/Volume) to measurements. Most students could not read volume measurements from these figures correctly, nor could they distinguish between precision and accuracy. This is foundational language in any quantitative work in science. Respondents performed poorly on this section. KKC students, including those who have taken science classes, answered correctly concerning significant figures (Q7) at levels worse than if they had responded to the question randomly.

DRAFT - FOR DISCUSSION ONLY 10 Students had a choice between section III, concerning atoms, and section IV, concerning DNA. Section III began with two questions that required reading comprehension and proceeded to three questions that required application of concepts from the reading. Section IV measured reading comprehension throughout. The performance gap between questions that measured comprehension versus application is visible and profound. These results appear to confirm the hypothesis that students have learned how to read and recall basic findings of science but learn far less how to apply concepts and use critical problem-solving skills.

DRAFT - FOR DISCUSSION ONLY 11 Only a handful of students answered every question correctly, whereas here we see another way of expressing that most students got the first two or three questions per section correctly, and then the harder questions requiring critical application incorrectly. The overall average of correct responses for Sections I-II was 4.40 out of 10 possible. The standard deviation is 1.9.

DRAFT - FOR DISCUSSION ONLY 12 How well did various groups perform relative to each other? Gender & Race matter a little— but not much is the overall average of questions 1-10 answered correctly. These graphs will all be set within boundaries of 3.5 to 5.5 so that they can easily be compared against each other. Part-time and Full-time students did about equally well. Students who were careless or couldn’t be bothered to fill in that blank performed worse than others. Note that women performed slightly better, breaking a stereotype concerning the natural sciences.

DRAFT - FOR DISCUSSION ONLY 13 Strong agreement with the importance of science correlated with higher overall performance. Age mattered a surprising amount. Students over 41 fared far worse on the assessment.

DRAFT - FOR DISCUSSION ONLY 14 Performance could also be divided between students who had done a lot of science, or science at anywhere other than KKC, and everyone else. Possibly it should raise eyebrows that students who have taken 3 or more science classes still only averaged about half the questions correct. Critical application skills are not being learned. However, students appear to learn the skills relevant to this assessment more in chemistry classes than in other sciences at KKC.

DRAFT - FOR DISCUSSION ONLY 15 CONCLUSIONS Students understand how to report back scientific findings, but they have weaknesses in measurement, application, and critical thinking. How does it matter in your class if students can engage empirical critical thinking skills? Why do students not acquire these skills more frequently? How can we help them learn these skills? Strong agreement with the value of science correlates with better performance on the assessment. Are there ways to increase student perceptions of the value of science? Students aged above 40 performed significantly worse on the assessment. Are older students out of practice with these kinds of problems? Are different teaching methods depending on generation appropriate? Students who had taken any class in chemistry performed significantly better on the assessment. Are several fundamental skills being included more in the chemistry curriculum than in other core science courses? Or could there be a confounding variable involved?