1. Dennis Sunal, Cynthia Sunal, Cheryl Sundberg, Glenda Ogletree, & Erika Steele The University of Alabama Cheryl L. Mason & Corrine Lardy San Diego State University Dean Zollman & Mojgan Matloob-Haghanikar Kansas State University College Science Teachers (SCST) March 27–30, 2008, Boston, MA TPC 0554594 Impact of Undergraduate Science Course Reform on Student Outcomes

2. NSEUS Collaborative Partners: v University of Alabama v San Diego State University v Kansas State University National Study of Education in Undergraduate Science

3. Overview 1. Background Summary of previous research & problem 2. NSEUS Project Plan NSEUS goals What is a reformed course? Research questions Participants 3. NSEUS Results to date Data collection & analyses Pilot Study National Survey Timeline Implications

4. This study addresses the overall problem How do undergraduate entry-level science courses, differing in level of reform, affect student learning outcomes?

5. Study components described 1) Literature review of research on previous undergraduate science reform, 2) National survey of faculty and courses at institutions involved in a professional development program aimed at undergraduate reform program, 3) Pilot study determining the feasibility or proof of concept, of the procedures and instruments for gathering data, and 4) Description of a national study of a population of institutions involved in reforming undergraduate courses with the intention of improving student outcomes.

6. 1.Literature review of previous undergraduate science reform: Summary of Findings from the Literature v Reform courses use inquiry-based instruction to increase student achievement v Process of reform positively impacts use of reform pedagogy in undergraduate science v Graduates of reform courses more likely to use inquiry science teaching

7. Summary (cont.) v Patterns emerging from the studies indicate reform coursework is possible, has positive effects on participants, and must overcome barriers. v Patterns suggest strategies for planning and implementing reform coursework. v The research base is limited but starting points are suggested in it.

8. What is a “Reformed” Course? v Incorporates the following characteristics; Based on national science standards Student-centered, i.e. active student roles Inquiry-based pedagogy Builds on students’ prior knowledge Incorporates interdisciplinary learning and collaborative approaches

9. Problem Developed from Literature v Question What is the impact of engaging higher education faculty in implementing standards-based reform in undergraduate science courses on student learning outcomes? v Study will investigate Short- & long term effects on learning outcomes Comparisons between reform and non-reform courses Comparisons of courses demonstrating differing levels of reform

10. Study Design

11. NOVA Program NASA Opportunities for Visionary Academics v Created to develop and disseminate a national framework for enhancing education of pre- service teachers in science, mathematics, and technology. (103 institutions in NOVA Network) v Mission: to improve the STEM literacy of future teachers by implementing standards- and research-based change nationally in higher education using NASA’s unique content. v NOVA Program Web Site http://www.nasa.gov/audience/foreducators/ postsecondary/grants/NOVA.html

12. Nova Professional Development Model Professional Development Ongoing Research Collaboration among institutions Instructional Technology Inquiry-based Instruction Interdisciplinary Collaboration Standards-based Courses NASA-based Content

13. 2. NSEUS national survey to characterize study population Conducted a survey of faculty and courses at institutions involved in a national professional development program Surveyed faculty and courses at institutions involved in undergraduate course reform

14. NSEUS Project Timeline v Year 1 v Revise and complete literature review v Survey & characterize population v Plan data collection protocols & instruments, select study sample v Conduct training on instruments and in research site protocols for all data collectors, and v Begin pilot study,

15. NSEUS Study Population

16. NSEUS National Survey Description of Institutions (Study Population) v 103 institutions v 354 faculty in collaborative teams of 3+ v 185 courses v Diverse institutions

17. Institutions Offering Reform Courses Created under the NOVA Program InstitutionsNumber of Courses Institutions Offering NOVA-Like Courses Reform Courses still offered 7214641(104) Reform courses stopped 31398(14) Total 10318549(118)

18. Instructional Methods Used in Reform Courses Instructional MethodAverage % of Time per Week Lecture 14% Discussion/Interaction of student groups 15% Integrated inquiry lab 64% Integrated use of technology 07%

19. Selected Reform Course Characteristics CharacteristicAverage Years course offered at institution 6 Times offered in a year 2 Course enrollment 35 Minority enrollment 25% Credit hours 4 (range = 3 - 5) Sections per semester 1.5 (range = 1 - 6) Students enrolling in reform courses per year ~10,000

20. 3. NSEUS case study to determine the feasibility of the procedures and instruments Case (pilot) study to determine validity and reliability of Instrumentation Data collection protocols Site visit protocol Classroom science learning context (climate)

21. NSEUS Project Timeline v Year 2 v Select study sample v Conduct pilot (case) studies with on site data collection using 2 institutions, 54 undergraduate courses, and 12 elementary teachers v Complete pilot studies v Revise data collection protocols & instruments, v Conduct additional training in research protocols for all data collectors, and v Select 30 study sample institutions from population

22. Instrumentation (partial list) v Thinking About Science Survey Instrument (TSSI) (Cobern, 2000) v Classroom Learning Environment Survey (CLES) (Taylor & Fraser, 1991, 1997) v Draw-A-Scientist Test (DASTT-C) (Thomas, Pedersen, & Finson, 2001) v Science Teaching Efficacy and Beliefs Instrument (STEBI A & B) (Riggs & Enochs, 1990), v Reformed Teaching Observation Protocol (RTOP) (Sawada, Turley, Falconer, Benford & Bloom, 2002) v PCK Content Representation (CoRe) & Pedagogical and Professional experience Repertoires (PaP-ers) (Loughran, Mulhall & Berry, 2004).

23. DAST Results v Short-term misconceptions of science, pre-post with undergraduate courses (lower score represents higher level) v 2 institutions, 4 courses (2 reform/ 2 comparison) v Significant change & higher posttest DAST scores

24. CLES Results v Reform course students report more positive classroom learning environment (CLES) v Environment related to science learning outcomes

25. Undergraduate Science Classroom Observations Using the RTOP Classroom Observations Made in the Reform Undergraduate Science Course Classrooms Common Classroom Observations Made in Both Types of Undergraduate Science Classrooms Classroom Observations Made in the Comparison Undergraduate Science Course Classrooms  Extensive student- student interaction during the class  Extensive teacher- student interaction during the class  Lectures were short and provided in a “ just in time manner ” coordinated with students ’ inquiry activities  Lecture and laboratory were integrated  Teachers used technology: smart boards, PowerPoint etc.  Content presented in both courses was current, appropriate, and accurate.  Little requested, or planned, student- student interaction  Teacher lecture took up the majority of the time  Students appeared bored and unengaged with the teacher and the content  Lecture and laboratory were separated in time

26. Reform Undergraduate Course Pedagogy in Study Reform Courses – Weekly Time Use

27. Pedagogy in Study Comparison Courses – Weekly Time Use Comparison Undergraduate Course

28. 4. NSEUS National Study National Study of a population of institutions involved in a long term professional development program, reforming undergraduate courses with the intention of improving student outcomes. RESEARCH QUESTIONS 1. What are the essential characteristics of an entry level reformed undergraduate science course? 2. How do reform science course characteristics differ from traditional courses? 3. How do course characteristics relate to undergraduate students short-term learning outcomes? 4. How do characteristics differ between courses with varying degrees of reform? 5. How do varying degrees of reform relate to undergraduate students short- term learning outcomes? 6. How do reform and traditional courses differ in their long-term impacts on K-6 teachers in their own science classrooms?

29. NSEUS Project Timeline v Year 3 v Collect & begin analysis of data from national sample in institutions and elementary classrooms v Conduct data analyses v Year 4 v Collect data from national sample of institutions and elementary classrooms v Conduct analyses of data

30. Research Participants v 30 institutions nationally (stratified random sampling of NOVA institutions), 2 courses from each institution (reformed & comparison) v 3 Regions coordinated by 3 Co-PIs: East (UA), Central (KSU), West (SDSU) v Faculty (60), students (~4000), and in-service K- 6 teachers (180) of reformed & comparison undergraduate science courses at sample institutions

31. NSEUS National Study Sample Description of Institutions (Study Sample) v 30 institutions v 142 faculty in collaborative teams of 3+ v 60 courses v Diverse institutions

32. Data Collection v Faculty, students, and in-service teachers v Content analysis of course materials v Multiple site visits with classroom observations v Student perceptions of science, science achievement outcome product measures v Interviews and focus groups

33. NSEUS Project Timeline v Year 5 v Complete data collection v Complete data analysis v National dissemination of results and conclusion on undergraduate science reforms v Conduct National Conference on Undergraduate Science Reform and its impact on faculty, undergraduate students, and inservice teachers

34. Contact Information http://nseus.org

35. National Study of Education in Undergraduate Science TPC 0554594