1. 1 MPI Mission Perception Inventory Ellen Boylan, Ph.D. and Sister Jane Wakahiu, LSOSF Marywood University Assessing Institutional Effectiveness with the Mission Perception Inventory (MPI): Linking Mission Goals and Learning Environment North Carolina State University Undergraduate Assessment Symposium Aligning Pedagogy, Curriculum & Assessment A pril 24-26, 2009

2. 2 What’s in a mission?

3. 3 scope of research Purpose Develop an instrument to measure student perception of institutional mission. Test instrument reliability. Uncover constructs (factors). Observe constructs longitudinally. GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

4. 4 scope of research Purpose Develop an instrument Test instrument reliability. Uncover constructs. Observe constructs longitudinally. R esearch Questions 1. Is the Mission Perception Inventory (MPI) a valid and reliable reliable measure of student perception of institutional mission? 2. What are the factors in the MPI? 3. Do the factors recur in repeated administrations of the revised MPI? 4. Are the factors equally reliable over time?

5. 5  leaders of public and private institutions alike are thinking about spirituality these days, as the data suggest that's what their students are thinking about, too (Inside Higher Ed, 2009).  There is strong connection between institutional programs and student learning environment (Pascarella, 2001).  …institutions influence levels of engagement on campus as a result of structural features, programs, policies, and organizational culture ( Kuh et al., 2005). background

6. 6 S elect the National Survey of Student Engagement (NSSE) as the vehicle for inserting research questions. A ssemble a NSSE consortium to jointly engage in research to explore student perception of mission. D evelop question items; administer them to the consortium as a NSSE attachment. T est the questions: reliability, factor analysis, and correlation analysis. Repeat annually. design GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

7. 7 developing the question items… 1. Conduct a qualitative analysis of mission statements 2. Distill common constructs 3. Draft question items based on mission statement constructs 4. Vet with participating institutions, experts, prospective subjects Concepts common to mission statement of consortium colleges, like this one, were drawn out, compared, and distilled into 20 questions. leaders in service professional disciplines …T he University roots itself in the principle of justice and a belief that education empowers people…offers students a welcoming and supportive community that encourages men and women of all backgrounds to shape their lives as leaders in service to others. Proud of its liberal arts tradition and host of professional disciplines,… challenging students to broaden their understanding of global issues and to make decisions based on spiritual, ethical, and religious values…

8. 8 Concepts become question items... GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

9. 9 A peek at the questionnaire See consortia questions by year: http://nsse.iub.edu/html/consortia-list_2009.cfmhttp://nsse.iub.edu/html/consortia-list_2009.cfm

10. 10 The NSSE survey with attached Mission Perception Inventory (MPI) questions has been administered 155 times to more than 44,198 first-year and senior students at 112 unique institutions across the United States every year since 2004. administration GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

11. 11 Reliability analysis of the mission questions (20 items) is performed. Low-loading items kicked out. A Mission Perception Inventory (MPI) is produced (17 to 19 items). Is the Mission Perception Inventory (MPI) valid and reliable?

12. 12 Factor analysis of MPI is conducted each year. By year, three or four subscales are produced: Sense of mission. Respect for diversity. Individual values. Religious practice/Spirituality. What are the factors (scales) in the Mission Perception Inventory?

13. 13 Mission Perception Inventory (MPI) (~19 items) Sense Of Mission Respect for Diversity Individual Values Spiritual Practice Administer Consortium mission questions (20 items) derivation of the MPI and subscales

14. 14 sense of mission sense of mission (10 items  =.90) The mission of this institution is widely understood by students. Social and personal development is an important part of the mission. Ethical and spiritual development of students is important. This institution offers opportunities for volunteering and community service. This institution offers opportunities for developing leadership skills. There are opportunities for students to strengthen their religious commitment. This institution’s religious heritage is evident. Professors here discuss the ethical implications of what is being studied. As a result of my experience here, I am more aware of my own personal values. The mission of this institution is reflected in course offerings.

15. 15 respect for diversity (5 items  =.878) The faculty, staff, and students here… respect different religions respect different races and cultures ……………………………………………………………………………… Students feel free to express individual spirituality. Different sexual orientations are accepted. The environment encourages appreciation of diversity.

16. 16 Consortia institutions and respondents by year Respondents InstitutionsFirst YearSenior 2004152,0001,827 2005161,2791,332 2006242,6842,854 2007364,5334,331 Catholic 20083545205063 Indepen’t 20082968056970 155*21,82122,377 *(112 unique institutions)

17. 17 Do the factors recur in repeated administrations of the revised Mission Perception Inventory (MPI)

18. 18 (MPI) Mission Perception Inventory Report GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

19. 19 Are the factors equally reliable over time? Subscales2004200520062007 2008 Catholic 2008 Indept cronbach  Sense of Mission.87.88.90.91.88 Respect for Diversity.85.84.87.86.88.87 Individual Actions.67n/a Religious Practice/Spirituality.62.55.54.62.64 MPI scale.88.89.90.91.90

20. 20 2009… Teagle Foundation Research Continues Develop the Mission Engagement Index. Obtain NSSE 2008 consortia data Test using reliability analysis to produce factors Distribute MPI reports Compare consortia results Conduct regression analysis to create the new index Produce Mission Engagement Index (MEI) Reports for consortia institutions. GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

21. 21 Does the Mission Engagement Index describe causal relationships among variables that affect mission perception? Dependent variable (Institution score) Mission Perception Inventory Sense of Mission scale Respect for Diversity scale Independent variables ( need 15 cases per ) * Selectivity 15 institutions Enrollment 30 institutions Urbanicity 45 institutions Resident % 60 institutions Female % 75 institutions Another? 90 institutions *…a recommended ratio of subjects to IVs of at least 15 to 1 will provide a reliable regression equation (Stevens, 1992).

22. 22 Selecting IVs for regression analysis

23. 23 participating institutions by region 2008 = 2 consortia, 54 institutions Is there sufficient variability?

24. 24 “urbanicity” of participating institutions 2008 Is there sufficient variability?

25. 25 considering elegance  Consistency of factors is affirmed.  Institution sizes reasonably varied.  Data sufficient to conduct analysis.  Variables, initially selected on an a priori basis, tested.  Variability investigated.  Useful. β eta juice

26. 26 MPI Scales: Comparison of means by institution type

27. 27 Correlation of regression variables with MPI mean

28. 28 Deriving coefficients for the Mission Engagement Index (MEI)

29. 29 Correlation of regression variables with Mission scale mean

30. 30 Predictive Equation* Institution Predicted MPI Score = (B 1 )*(Value of USN_setting)+ (-0.414)*(1, 2, or 3) (B 2 )*(Value of Religious Affiliation)+ (0.345)*(1 or 2) Constant (3.687) *Mortenson, T. (1997). Actual Vs Predicted institutional graduation rates for 1100 Colleges and universities. Opportunity, 58.

31. 31 Mock Mission Engagement Index Report  The MEI will show an institution’s actual versus predicted scores on mission constructs.  Progress on mission effectiveness can be assessed by comparing MEI outcomes to institutional goals.

32. 32 References Inside Higher Education (2009). Spiritual accountability. Retrieved from http://www.insidehighered.com/layout/set/pri...assessment/01/02/2007 /News http://www.insidehighered.com/layout/set/pri...assessment/01/02/2007 /News Kuh, D. G., Kinzie, J., Schuh, J. H., and Whitt, E. J. (2005). Never let it rest: lessons about student success from high-performing colleges and universities. Change, 37(4), 44-51. Mortenson, T. (1997). Actual Vs predicted institutional graduation rates for 1100 colleges and universities. Opportunity, 58. Pacarella, E. T. (2001). Identifying excellence in undergraduate education. Change, 33(3), 18-27. Stevens, J. P. (1992). Applied multivariate statistics for the social sciences (2 nd edition). Hillsdale, New Jersey: Erlbaum.

33. 33 Discussion

34. 34 Ellen Boylan, Ph.D. Director of Institutional Research and Assessment eboylan@marywood.edu Sister Jane Wakahiu, LSOSF, MA Graduate Assistant jwakahiu@marywood.edu Office of Planning and Institutional Research http://cwis.marywood.edu/instresearch/activity.stm Assessing Institutional Effectiveness with the Mission Perception Inventory (MPI): Linking Mission Goals and Learning Environment Supported by a grant from http://www.teaglefoundation.org/grantmaking/grantees/assessmentmethods.aspx Marywood University

35. 35

36. 36 MPI Mission Perception Inventory Ellen Boylan, Ph.D. and Sister Jane Wakahiu, LSOSF Marywood University Assessing Institutional Effectiveness with the Mission Perception Inventory (MPI): Linking Mission Goals and Learning Environment North Carolina State University Undergraduate Assessment Symposium Aligning Pedagogy, Curriculum & Assessment A pril 24-26, 2009