Assessing Institutional Effectiveness: The Mission Engagement Index (MEI) as a Measure of Progress on Mission Goals Ellen Boylan, Ph.D. ● Marywood University NEAIR 35 th Annual Conference ● Providence, Rhode Island ● Nov. 1 – 4, 2008

Scope of Research Purpose  Develop a measure (inventory) of student perception of institutional mission.  Test instrument reliability. and observe them longitudinally.  Uncover constructs (factors) and observe them longitudinally.  Exploratory multiple regression: Develop a measure (index) that allows institutions to compare their predicted with actual outcomes on mission constructs. Research Questions 1. Is the Mission Perception Inventory (MPI) a valid, reliable measure of student perception of institutional mission? 2. What are the factors in the Mission Perception Inventory? 3. Do factors recur with equal reliability in repeat administrations of the MPI? 4. Does the Mission Engagement Index (MEI) describe causal relationships among variables that affect mission perception?

Study Design Study Design  Select the National Survey of Student Engagement (NSSE) as the vehicle for delivery of the research questions.  Assemble a NSSE consortium to jointly engage in research to explore student perception of mission.  Develop consortium questions that are then given as an attachment to NSSE; analyze results via reliability, factor analysis, and correlation.  Repeat annually. Once a sufficient volume of institutional data is gathered, conduct regression analysis to develop the index (MEI). GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS

4 Developing the question items  Conduct a qualitative analysis of mission statements.  Distill common constructs.  Draft question items based on those constructs.  Vet with participating institutions, experts, and prospective subjects. Concepts common to consortium colleges’ mission statements, like this one, were drawn out, reduced, and formulated into 20 question items. leaders in service professional disciplines …T he University roots itself in the principle of justice and a belief that education empowers people…offers students a welcoming and supportive community that encourages men and women of all backgrounds to shape their lives as leaders in service to others. Proud of its liberal arts tradition and host of professional disciplines,… challenging students to broaden their understanding of global issues and to make decisions based on spiritual, ethical, and religious values…

Concepts became question items...

A peek at the questionnaire See consortia questions by year:

7 Administration The NSSE survey with attached Mission Perception Inventory (MPI) questions has been administered 145 times to more than 30,000 first-year and senior students at 83 colleges across the United States every year since 2004.

Reliability analysis of the mission questions (20 items) is performed. Low-loading items kicked out. A Mission Perception Inventory (MPI) is produced (17 or 18 items). Is the Mission Perception Inventory (MPI) valid and reliable?

9 What are the factors in the Mission Perception Inventory? Factor analysis of MPI is conducted each year. By year, three or four subscales are produced: Sense of mission. Respect for diversity. Individual values. Religious practice.

10 Mission Perception Inventory (MPI) (~17 items) Sense Of Mission Respect for Diversity Individual Values Religious Practice Administer Consortium mission questions (20 items) Derivation of Mission Perception Inventory (MPI) and subscales

11 Sense of Mission Scale Sense of Mission Scale (8 items  =.867) The mission of this institution is widely understood by students. Social and personal development is an important part of the mission. Ethical and spiritual development of students is important. This institution offers opportunities for volunteering and community service. This institution offers opportunities for developing leadership skills. There are opportunities for students to strengthen their religious commitment. This institution’s religious heritage is evident. Professors here discuss the ethical implications of what is being studied.

Respect for Diversity Scale Respect for Diversity Scale (5 items  =.854) The faculty, staff, and students here… respect different religions respect different races and cultures ……………………………………………………………………………… Students free to express individual spirituality. Different sexual orientations are accepted. The environment encourages appreciation of diversity.

13 Count of institutions and respondents by year respondents

14 Do the factors recur in repeated administrations of the revised Mission Perception Inventory (MPI)

15 Are the factors equally reliable over time?

… Teagle Foundation Research Continues Develop the Mission Engagement Index. Obtain NSSE 2008 consortia data Test using reliability analysis to produce factors Distribute MPI reports Compare consortia results Conduct regression analysis to create the new index Produce Mission Engagement Index (MEI) Reports for consortia institutions.

Does the Mission Engagement Index describe causal relationships among variables that affect mission perception? Dependent variable (Institution score) Mission Perception Inventory Sense of Mission scale Respect for Diversity scale Independent variables ( need 15 cases per ) * Selectivity 15 institutions Enrollment 30 institutions Urbanicity 45 institutions Resident % 60 institutions Female % 75 institutions Another? 90 institutions *…a recommended ratio of subjects to IVs of at least 15 to 1 will provide a reliable regression equation (Stevens, 1992).

18 Selecting IVs for regression analysis

19 Participating institutions by region 2008 = 2 consortia, 54 institutions Is there sufficient variability?

20 “Urbanicity” of participating institutions 2008 Is there sufficient variability?

21 Participants by gender and class rank by year Is there sufficient variability?

22 Considering elegance  Consistency of factors is affirmed.  Institution sizes reasonably varied.  Data sufficient to conduct analysis.  Variables, initially selected on an a priori basis, tested.  Variability investigated.  Useful. β eta juice

23 Mock Mission Engagement Index Report  The MEI will show an institution’s actual versus predicted scores on mission constructs.  Progress on mission effectiveness can be assessed by comparing MEI outcomes to institutional goals.

Assessing Institutional Effectiveness: The Mission Engagement Index (MEI) as a Measure of Progress on Mission Goals Ellen Boylan, Ph.D. Marywood University Scranton PA available Supported by a grant from