Presentation on theme: "LESSONS LEARNED from a MULTI-CAMPUS DIVERSITY INITIATIVE THE USE OF EVALUATION & ORGANIZATIONAL LEARNING FOR EFFECTIVE DIVERSITY INITIATIVES:"— Presentation transcript:
LESSONS LEARNED from a MULTI-CAMPUS DIVERSITY INITIATIVE THE USE OF EVALUATION & ORGANIZATIONAL LEARNING FOR EFFECTIVE DIVERSITY INITIATIVES:
The American college or university is a prototypic organized anarchy. It does not know what it is doing. Its goals are either vague or in dispute. Its technology is familiar but not understood. Its major participants wander in and out of the organization. These factors do not make a university a bad organization or a disorganized one; but they do make it a problem to describe, understand, and lead [and as a result evaluate]. -- Michael D. Cohen & James G. March in Leadership and Ambiguity (1974)
Assessment: Systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student [and campus] learning and development. Marchese, T. in Assessment Essentials, Palomba, C.A. & Banta, T.W. (1999)
Empowerment Evaluation: The use of evaluation concepts, techniques, and findings to foster improvement and self-determination…it is designed to help people help themselves and improve their programs using a form of self-evaluation and reflection. David Fetterman in Empowerment Evaluation (1996)
Purposes of Assessment Test Assumptions Our bridge program is highly successful We are / are not diverse because… Diversity is central to our mission Initiate & Guide Dialogue Identify Gaps, Disconnects & Potential Solutions Develop a sense of ownership and empowerment Develop and sustain capacity for organizational learning
Steps in effective assessment of diversity initiatives 1.Define the Purpose 2.Primary Audience 3.Evaluation Team 4.Identify Context 5.Target Topic 6.Formulate Questions 7.Obtain Data 8.Assess Data 9.Analyze Data 10. Report Findings Source: Assessing Campus Diversity Initiatives: A Guide for Campus Practitioners, Garcia, M. et al. (2001)
Utilize Existing Data Institutional Research Office Vice-President for Planning / Research Varied Campus Offices Past Strategic Plans Self-Studies for Accreditation Agencies Past Diversity Initiatives Integrated Postsecondary Education Data System (IPEDS)
Benefits of Varied Types of Data Quantitative Data –Helps us explore the what and the who (e.g. campus-based surveys) Qualitative Data –Helps us understand the why and the how (e.g. focus groups; interviews; observations; document analyses) Process Data –Helps us understand and explore where pockets of resistance, inertia, or hope are located –And, whether IT is resistance…inertia…or hope
Assess and Analyze the Data Response Rates Recognize and Qualify Data Limitations Background Variables (e.g. race/ethnicity, gender, socioeconomic status, major etc.) Disaggregation Guided by mission / purpose of program or initiative
Report the Findings Review target audience Format for dissemination Written reports; oral presentations; internal/external dissemination Guide the discourse Involve critics Balanced Analysis that emphasizes the good, the bad, and the ugly Connect findings with overall institutional mission
Research must function as a tool for appropriating the codes and cultural symbols of institutional power in an effort to transform institutional environments in the interest of cultural democracy…In this way, researchers who carry out their work in the interest of cultural democracy can function as social advocates, facilitating a production of knowledge that is committed to the creation of institutional conditions where people find their voices and their rightful places as full and equal participants. -- Antonia Darder Institutional Research as a Tool for Cultural Democracy
References Assessing Campus Diversity Initiatives: A Guide for Campus Practitioners by Garcia, M.; Hudgins, C.A.; Musil, C.M.; Nettles, M.T.; Sedlacek, W.E.; & Smith, D.G. Leadership and Ambiguity by Cohen, M.D. & March, J.G. Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education by Palomba, C.A. & Banta, T.W. Empowerment Evaluation: Knowledge and Tools for Self-Assessment & Accountability edited by Fetterman, D.M.; Kaftarian, S.J.; & Wandersman, A. Institutional Research as a Tool for Institutional Democracy by Darder A. in Studying Diversity in Higher Education edited by Smith, D.G.; Wolf, L.E.; & Levitan, T.
iALTO! BREAK !!
Nuts and Bolts of Evaluation: How to Use the Process to Advance Institutional Learning about Diversity
IRVINE FOUNDATION GOALS FOR THE CAMPUS DIVERSITY INITIATIVE Increase the institutionalization of diversity in the context of educating all students for participation and leadership in a diverse society. Increase the educational success of disadvantaged and ethnic minority populations. GOALS AND PURPOSE
GOALS AND PURPOSE PURPOSE OF AN EVALUATION EFFORT: ORGANIZATIONAL LEARNING Provide ongoing information about the implementation so that mid-course corrections can be instituted. Build the capacity of campuses to assess and learn from their own progress. Identity and provide opportunities for campuses to share problems and solutions.
ORGANIZATIONAL LEARNING Using institutional data to inform progress Interrupting the usual Engage myths that become self-fulfilling prophecies Maintaining the link between campus diversity efforts and institutional goals for effectiveness and excellence Taking time for reflection and self-learning Keeping all campus constituencies informed Building synergy among many efforts on campus
Approaches evaluation from an organizational learning point of view Manageable for campus and capable of being maintained Monitors key goals and elements of proposal Focuses on institutional issues/change, not simply project-specific issues Reveals success and problems along the way in both results and processes Guides interim reports to senior leadership, campus, board PRINCIPLES
PRINCIPLES Takes into account: –Institutional differences and stages with respect to diversity –That institutions vary in mission, needs, goals and culture –That strategies, goals and emphasis differ –The possibility of taking some risks and learning from them –Differences within institutions (Disaggregation of information) Encourages institutional sharing Uses outside resources as appropriate
INSTITUTIONAL AUDIT Education and Scholarship Access and Success Institutional Viability and Vitality Climate and Intergroup Relations CONTEXT Global Local
ACCESS AND SUCCESS Transfer among fields (esp. SMET) Success of students -- graduation, persistence, honors, performance Pursuit of advanced degrees Undergraduate/ graduate population by fields and levels ACCESS and SUCCESS Disaggregated
Four & Six-year Graduation Rates by Race/Ethnicity (%)
CLIMATE AND INTERGROUP RELATIONS Type and quality of interaction among groups Quality of experience/ engagement on campus Perceptions of institution (climate, commitment, engagement) CLIMATE AND INTERGROUP RELATIONS Disaggregated
% Satisfied with Climate for Minorities by Attendance in Racial Cultural Awareness Session:
% Satisfied with Climate for Minorities by Attendance in Racial Cultural Awareness Session and Race/Ethnicity:
EDUCATION AND SCHOLARSHIP Experience Presence of diversity-related courses, requirements Degree to which courses include diversity issues and the placement of such courses Quantity and substance of student learning about diversity Course taking patterns of students Level of faculty expertise on diversity-related matters Level & diversity of faculty participating in diversity efforts Availability Faculty Capacity Learning EDUCATION AND SCHOLARSHIP
INSTITUTIONAL VIABILITY AND VITALITY Constituency perceptions of institutional commitment to diversity Perceptions of access, equity and inclusion among all constituents Public perceptions of the institution Centrality of diversity in the planning process, mission statements Institutional Strategies and dedicated resources Institutional history on diversity issues and incidents Diversity of faculty and staff by level INSTITUTIONAL VIABILITY AND VITALITY
Faculty Turnover Quotient RESULTS TQ = 0%: No Turnover TQ = 100%: 100% of URM new hires replaced URM faculty who left the institution
FACULTY Overall --DISAGGREGATED--over time New hires Retention/turnover--TQ Location --departments/fields
EVALUATION PROCESS Institutional Goals Process Strategies (Objectives) REPORTING Interpretations/Perceptions Lessons Learned What has been the impact on for example, curriculum, student success, etc.? What has been done? What has happened? What is working? What is not working?
iALTO! BREAK !!
Principles of Evaluation & Institutional Learning Framework and Monitoring Progress Centrality to Mission Leadership & Communication Organizational Learning Inclusive and Differentiated Approach Alignment