NOVA Evaluation Report Presented by: Dr. Dennis Sunal

Evaluation Questions Data has been collected as part of a formative evaluation effort in the continuous enhancement of NOVA structure and functions to met its objectives. Ten questions and data follow.

1. What are the characteristics of the institutions, faculty, and students who matriculate in NOVA courses? - Principles from the National Science Education Standards - Principles from the National Council of Teacher of Mathematics - NOVA Professional Development Workshop Components

National Science Education Standards-Five principles 1. Change throughout the entire science education system. 2. What students learn is influenced by how they are taught. 3. The actions of teachers are influenced by their perceptions of science.

National Science Education Standards-Five principles, cont. 4. Student understanding is constructed through individual and social processes. 5. Teaches actions are influenced by the relationships with their students.

National Council of Teacher of Mathematics-Six principles 1. Math education requires equity. 2. The curriculum must be coherent, and well articulated across the grades. 3. Effective math requires the teacher to understand students prior knowledge, and continue to challenge and support the students to learn.

National Council of Teacher of Mathematics-Six principles, cont. 4. Students must learn math by actively building new knowledge from experiences and prior knowledge. 5. Assessment needs to support the learning of math. 6. Technology influences math that is taught, and enhances students’ learning.

NOVA Development Workshop Components-Twelve skills 1. NASA strategic enterprises 2. Inquiry-based learning 3. Innovative instructional strategies 4. Assessment strategies that facilitate learning 5. Teaching science for all Americans 6. New curriculum goals

NOVA Development Workshop Components-Twelve skills, cont. 7. Interdisciplinary approaches 8. Using technology to facilitate learning 9. Overcoming barriers 10.Action research for college faculty 11. Writing proposals for NOVA 12. Connections to NSES, NCTM, ISTE standards.

2. To what extent is the NOVA Model being disseminated and implemented by other faculty at NOVA institutions or at other institutions? 100 publications in journals, manuscripts, and presentations at national and regional conferences. Regional and local workshops, consultancies, and other professional activities.

3. How congruent is the NOVA program with recommendations found in major reports on the preparation of teachers? The NOVA Model for Professional development includes eight major elements aimed at changing how science and math are taught in K-12 classrooms by influencing “how” content is taught at the college level.

Eight major elements 1. Continuous professional development. 2. Sustained collaboration among interdisciplinary teams of faculty and administrators. 3. Courses based on national standards for science, math, and technology. 4. Courses utilize content based on NASA Strategic Enterprises.

Eight major elements, cont. 5. Program improvement at NOVA institutions through ongoing research. 6. Instructional strategies that are inquiry based and center on student interaction. 7. Extensive use of technology. 8. Collaborations among institutions in the NOVA network.

4. What are the participant reactions to the NOVA professional development model? (Two part question) Part A: Participant Reactions to the NOVA program Positive Useful workshops NOVA course online modules has been considered time well spent in professional development.

4. What are the participant reactions to the NOVA professional development model? (Two part question) Part B: The change process for NOVA faculty Commitment to the project. Belief in the model. The use of NASA resources. Monetary support for innovations. Introduce innovative teaching and/or interdisciplinary courses to faculty.

Part B, cont. Inspiration that came from changes in state education requirements and/or content standards. Team of teachers attend a Phase I NOVA workshop. Staff development.

5. How has the implementation of the NOVA Model impacted the collaborative work and organizational climate of science, mathematics, engineering, and education faculties? (Two part question) Part A: Collaborative work changes When a team member leaves the university. Teams no longer exist at the institution. Faculty turnover and retirement. New faculty has not been integrated into the team.

Part A, cont. Collaboration continues with institutionalization of courses. Team approach is vital to course development.

5. How has the implementation of the NOVA Model impacted the collaborative work and organizational climate of science, mathematics, engineering, and education faculties? (Two part question) Part B: Organizational climate changes The NOVA course has become the “favorite lab.” The collegiality and impact on faculty and students. Collaboration between the College of Education and the science disciplines in the College of Arts and Sciences.

Part B, cont. Time barriers. Institutional support barriers. Funding was a barrier. Conflicts between or within department conflicts and the lack of mutual respect. Students who had weak backgrounds, lacked motivation, or were fearful of math and science.

6. What new and modified courses are being offered as a result of NOVA? 135 courses were developed and offered at 84 NOVA institutions in the topic areas of : -Biology -Life science -Life in space -Environmental science

7. To what extent does the NOVA course content integrate data and information that are unique to NASA, its fundamental questions, and its missions? The NOVA courses integrate at least one of the five NASA Strategic Enterprises as a fundamental course component: ATE, BPRE, ESE, HEDS, and SSE.

8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question) Part A: What are the faculty perceptions of effective elements in the NOVA professional development process? Cognitive apprenticeship Facilitators of learning High self-efficacy Knowledge of pedagogy

8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question) Part B: Does action research conducted as a regular part of faculty teaching facilitate development of effective courses in science and mathematics? Multiple techniques concerning assessment: journals, on-line tasks, rubrics, observation, performance tasks, interviews, and discussion groups, etc.

Part B, cont. Faculty members may need additional support to engage in action research. Investigating effective ways to teach science for improved student learning. The need for instructors to create a “friendly learning environment” for students with poor attitudes.

8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question) Part C: What factors relate to sustaining change brought about through the NOVA professional development process? Continued interaction between departments or disciplines. Teams should have ‘people with similar ideas.’

9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question) Part A: Course disciplinary content NOVA course had significantly impacted student attitudes to science as inquiry and to the study of climate. Students taking the NOVA course had significantly higher teaching efficacy. Students in the NOVA course also displayed more positive attitudes to geology.

Part A, cont. Students in the NOVA course demonstrated significant gains in content knowledge. Students valued the hands-on, collaborative approach to learning science.

9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question) Part B: Classroom pedagogy More interactive methods to enhance student learning. Some classes were team-taught by two or more faculty. Good collaboration helped shift faculty from teacher-centered teaching to student-centered approaches.

Part B, cont. The team approach provided the necessary expertise, collaborative preparation, and sharing of the workload. 98% of the new courses were using technology of some sort in course delivery.

9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question) Part C: Specific disciplinary pedagogy Hands-on Less or little lecture Lots of group work Student input A diversity of teaching styles The content was “experienced–not told”

Part C, cont. NOVA instructor was motivational. Students content knowledge was improved. Technology enhanced learning. Course was clearly connected to understanding the content.

10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question) Part A: What are students’ perceptions of effective learning in science, mathematics or engineering classrooms resulting from the NOVA course? Hands-on approach The goal of teaching was student understanding, not just covering content.

Part A, cont. Awareness of students’ prior knowledge. Having the ability to explain things on the students level. Teaching in diverse ways. The teachers had the ability to “make science come to life.”

10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question) Part B: What changes in attitude toward science and mathematics are found in the students who experienced the NOVA course? Becoming comfortable with various technology aids. Positive student attitudes. An increase of minority students who participated in the NOVA classes.

10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question) Part C: What changes in teaching efficacy and learning outcomes resulted from students taking the NOVA course? An increase in students’ content knowledge. An increase in scientific processing skills. A greater understanding of the mathematical concepts. Increased self-efficacy in science.

Part C, cont. An increased ability and willingness to teach science and mathematics. An increase in the students’ ability to use technology. Enhanced critical thinking skills. A high level of interest in the activities and were able to relate the content to real life.

10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question) Part D: What changes in the nature of science and nature of mathematics, related to the literacy goals in the national standards, are found in students taking the innovative NOVA courses? Undergraduate students in NOVA reform- oriented classes demonstrated a significantly higher end of course growth and level of understanding of the nature of science when compared to students in other science, “non- reformed,” courses.

Part D, cont. An increased interest of females in science and mathematics literacy. The learning climate was significantly more positive in the NOVA courses as compared to traditional NOVA courses.

10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question) Part E: What are the indicators of change in students in NOVA course related to knowledge and performance of standards-driven reforms as a K-12 teacher of science or mathematics? Improvements in the achievement of science content assessments. Students had a high level of interest in the activities and were able to relate the content to real life.

Part E, cont. Students’ self efficacy has increased.