Student Engagement and Mission Effectiveness: Looking for Links between Outcomes and Institutional Goals Ellen M. Boylan, Ph.D. Association for Institutional Research 2005 Forum San Diego, California May 29 – June 1

2 Background  Sadlak (1986) asserts that the institutional mission statement influences student achievement…it describes the desired academic and social development skills to be possessed at end of the educational experience.  How does an institution measure the effect of its mission on students’ educational experience? Can it be measured? Do students’ perceptions of mission differ by institution?

3 Research Questions Research Questions 1. Do student ratings of engagement and learning environment differ by student perceptions of institutional mission? (a) Are the Research Mission Questions reliable and valid measures of student engagement and perception of learning environment? (b) What is the relationship between the Research Mission Questions and NSSE benchmark scores for Consortium schools? 2. Do students and administrators share the same perceptions of institutional mission?

4 Study Design Study Design  Select the National Survey of Student Engagement (NSSE) 2004 instrument (The College Student Report) to test first year and senior students  Assemble an NSSE 2004 Catholic College Consortium for mission research  Develop Research Mission Questions  Administer the NSSE 2004 with added Research Mission Questions to Consortium students  Administer a sub-set of Research Mission Questions to Consortium college administrators  Analyze results: factor analysis, correlation and regression, and t-test analysis of means Marywood University Institutional Review Board Approval

5 Sample Mission Statement Concepts common to the Consortium Colleges’ mission statements were drawn out, reduced, and formulated into 20 Research Mission Questions leaders in service professional disciplines spiritual, ethical, and religious values learning diverse and interdependent world. A Catholic university sponsored by the Congregation of the Sisters, Servants of the Immaculate Heart of Mary, Marywood University roots itself in the principle of justice and a belief that education empowers people. Enacting its ideals, Marywood offers students a welcoming and supportive community that encourages men and women of all backgrounds to shape their lives as leaders in service to others. Proud of its liberal arts tradition and host of professional disciplines, Marywood challenges students to broaden their understanding of global issues and to make decisions based on spiritual, ethical, and religious values. Marywood calls upon students to seek their full potential and invites all to engage in a lifelong process of learning. Witnessing the efficacy of teaching and scholarship, Marywood educates students to live responsibly in a diverse and interdependent world.

6

7 Administration The NSSE Survey with attached Research Mission Questions was administered to 3,827 first-year and senior students at 14 Catholic Colleges in the United States in Spring 2004.

8 Institution Demographics  Gender of students  2 single-sex institutions  Percent female – 61 to 85  Size of Institutions  Large - 1  Small - 2  Medium - 12  Location of Institutions  Urban – 5  Suburban - 9

9 Characteristics of Institutions Gender

10 Characteristics of Institutions Size and location

11 Research Question 1a. Are the Research Mission Questions reliable and valid measures of student engagement and perception of learning environment?

12 Reliability and factor analysis  Reliability analysis of Research Mission Questions (20 items). Three items removed.  Mission Perception Inventory (MPI) is produced (17 items).  Factor analysis of MPI conducted.  Four subscales produced: Sense of Mission Respect for Diversity Individual Actions Religious Practice

13 Mission Perception Inventory (MPI) (17 items) Sense of Mission (8 items) Respect for Diversity (5 items) Individual Actions (2 items) Religious Practice (2 items) Research Mission Questions (20 items) Derivation of Mission Perception Inventory (MPI) and four subscales from Research Mission Questions

14 Sense of Mission Scale (8 items  =.867)  The mission is widely understood by students  Social and personal development is important  Ethical and spiritual development is important  This institution’s religious heritage is evident  Professors here discuss the ethical implications of what is being studied  This institution offers opportunities for  volunteering and community service  developing leadership skills  students to strengthen their religious commitment

15 Respect for Diversity Scale (5 items  =.854)  The faculty, staff, and students here  respect different religions  respect different races, cultures  Students free to express their individual spirituality  Different sexual orientations are accepted  The environment encourages appreciation of diversity

16 Rotated Factor Matrix 1 Sense of Mission 2 Respect for Diversity 3 Individual Actions 4 Religious Practice Social and personal development of students is an important part of the mission at this institution Ethical and spiritual development of students is an important part of the mission at this institution This institution offers opportunities for: Volunteering and community service Developing leadership skills At this institution, there are opportunities for students to strengthen their religious commitment The heritage of the founding religious community… is evident here The mission of this institution is widely understood by students The professors at this institution discuss the ethical implications of what is being studied The faculty, staff, and students here are respectful of people of different religions The faculty, staff, and students here are respectful of people of different races and cultures Students feel free to express their individual spirituality here People of different sexual orientations are accepted socially here The environment here encourages students to develop an appreciation of diversity How important is it to you that you accomplish the following: Volunteering in community service Influencing social values Within the past week… participated in a religious service? Within the past week…spent time in private prayer or meditation?

17 Research Question 1b. What is the relationship between the Research Mission Questions and NSSE benchmark scores for Consortium schools?

18 NSSE Benchmark Scales  Level of Academic Challenge  Active and Collaborative Learning  Enriching Educational Experiences  Supportive Campus Environment  Student-Faculty Interactions

19 Reliability of Scales Consortium data

20 Significant Correlations by Consortium Scale ***All correlations are significant at p<.001 level, 2-tailed, df=3,825.

21 Institution Scale Correlations by Level MPI Scales with NSSE Scales: Correlation is significant at the * p<.05 level (2-tailed), df=12; **p <.02 level (2-tailed), df=12; ***p<.01 level, 2- tailed, df=12; ****p<.001 level, 2-tailed, df=12,

22 Research Question 1 Do student ratings of engagement and learning environment differ by student perceptions of institutional mission?

23 Procedure for regression analysis  Conduct regression analysis using Sense of Mission and Respect for Diversity as dependent variables, entering all NSSE scales as a group.  Check regression coefficients for significance.  Repeat analysis employing stepwise regression, entering scale(s) with significant beta as indicated.

24 Standardized Regression Coefficients: Sense of Mission and Student Class Level by NSSE Scales

25 Model Summary – Sense of Mission Regression analysis

26 Standardized Regression Coefficients: Respect for Diversity and Student Class Level by NSSE Scales

27 Regression analysis Model Summary – Respect for Diversity

28 Procedure for post hoc analysis  Institution scale correlations suggest differences between institutions may exist.  Explore this by comparing means on Sense of Mission and Respect for Diversity by level and institution.

29 Post hoc - Mean Sense of Mission scores by institution and student level

30 Post hoc - Mean Respect for Diversity scores by institution and student level

31 Research Question 2 Do students and administrators share the same perceptions of institutional mission?

32 Student and administrator means by scale and item FY N=1868 SR N=1705 Admin. N=60T*Sig.* SENSE OF MISSION RESPECT FOR DIVERSITY N.S. RELIGIOUS PRACTICE *Comparison of senior and administrator means only

33 Findings and Conclusions  Reliability analysis and factor analysis confirmed that the Mission Perception Inventory (MPI) is a reliable and valid measure of student perception of institutional mission (  = ).  Factor analysis of the MPI resulted in four reliable subscales, Sense of Mission, Respect for Diversity, Individual Actions, and Religious Practice.  The four subscales produced indicate the MPI reflects these concepts which are included in Consortium college mission statements.  The high reliability of subscales indicates that the MPI can be employed with confidence as measures of student mission perception in subsequent studies.

34 Findings and Conclusions  The MPI, Sense of Mission, and Respect for Diversity scales correlate with the NSSE benchmark scales, particularly Supportive Campus Environment (SCE).  Respect for Diversity was significantly related (for first year students only) to SCE, Enriching Educational Experiences (EEE) and Student-Faculty Interactions (SFI) scales.

35 Findings and Conclusions  Supportive Campus Environment (SCE), followed by Enriching Educational Experiences (EEE) and Level of Academic Challenge (LAC), are most closely related to Sense of Mission.  SCE contributed about 21% of the variance to the Respect for Diversity scale and was significant for both first year and senior students.

36 Findings and Conclusions Post hoc  The institutions showing higher senior than first year scores on the Sense of Mission scale are medium in size, selective.  Institutions showing higher senior means compared to first year, have low to moderate percentages of Roman Catholic students, and/or are about 75% female.  The two schools with the lowest senior mean scores for Respect for Diversity have the largest percentage of students self-identifying as Roman Catholic.

37 Findings and Conclusions Research Question 2  The mean score differences between administrators and seniors is significant for nine items.  Administrators perceive Sense of Mission concepts and behaviors as more widely expressed on campus than students perceive they are.  Not unexpectedly, mean scores for administrators and students differed significantly on the Religious Practice scale.

38 Conclusions F actor analysis and analysis of reliability indicate the Mission Perception Inventory (MPI) has extremely high reliability and four correlated, independent and strong factors. T here is a significant relationship between students’ Sense of Mission and Respect for Diversity and the Supportive Campus Environment they experience; and to a lesser extent, the academic challenges and enriching educational experiences provided them by higher education. S ense of Mission is rated significantly higher by college administrators than students; however, Respect for Diversity ratings are not significantly different.

39 For further information… Ellen M. Boylan, Ph.D. Planning and Institutional Research Marywood University