Assessing and Evaluating Impact Abigail J. Stewart and Janet E.Malley University of Michigan

What are we trying to assess?  Impact of project  Takes place over five years  Addresses different aspects of the problem  Recruitment  Retention  Climate  Leadership  Includes interventions at the  University level  College level  Departmental level  Individual level

What data will we need?  Will need to address  Change over time  Different elements of the problem  Recruitment  Retention  Climate  Leadership  Different levels of intervention

Model of data needed  Pre-post data  Identify change across five years (curve)  Identify level at “pre” and then “post”  Addressing particular problem  Recruitment (how many women scientists)  University  College (large vs. small)  Department (is this meaningful? Problem of aggregation, variable resources)  Individual (really not meaningful)

Thinking about the model  Are numbers/outcomes the goal?  What about practices and policies that produce the numbers?  What about sustainability?

Revising the model  Identify what changes (recruited women)  Identify what can produce change  Changed practices  Changed policies  Identify what can sustain change  Institutionalization of the practices  Institutionalization of the policies

Engage in this process for all elements  Retention  Outcomes  More women stay; fewer leave  More women succeed  Processes that produce change in the outcomes  May be different processes that produce staying vs. succeeding  Policies that sustain change in the outcomes  Processes that promote staying (partner policies? Family friendly policies?)  Processes that promote succeeding (flexible clock, fairer reviews, less invisibility and marginalization)

Engage in this process for all elements  Climate  Outcomes (changes in reported climate)  Processes that produce change in the outcomes (more difficult to pinpoint; may need to collect data on “best practices” in high morale places)  Policies that sustain change in the outcomes (once have best practices, easier to identify policies)

Engage in this process for all elements  Leadership  Outcomes  More women chairs  More women in other kinds of leadership positions  Fewer women who feel marginalized  Processes that produce change in the outcomes  Selection processes  Overall impact on climate for women’s voices  Policies that sustain change in the outcomes  Accountability (not same as selection)  New procedures institutionalized

What timeline do we need for assessing change?  Five year program:  Can anything change in five years?  At what level?  What indicators?  recruitment? Yes  attrition? Unlikely  Climate?  Pre-post?  Annual change? (Growth curve?)  Significant change from baseline to outcome?  Some interventions/key transformation points within program  Appointment of woman chair  Introduction of crucial policy  Attainment of critical mass  First successful tenure case

How can we use data collection as part of the change process?  Obvious with climate surveys  Target points of intervention  Mentoring  Chairs  Service burden  Contract renegotiation (counter-offers)  Self-studies in departments  Graduate student attrition  Competitive climate in department

How can we use data analysis as part of the change process?  Obvious with salary analyses  Identify inequities  Intervene  Institutionalize analysis and intervention

How can we use data analysis as part of the change process?  Obvious with space analyses  Identify inequities in square footage  Intervene  Institutionalize analysis and intervention  Identify process of assigning space based on fair criteria  Need  Funding  Real issue may be inequities in quality of space not square footage

How can we use data reporting as part of the change process?  NSF data on women and named chairs  Identifies a problem (too few women get them; discover too few nominated)  Requires a nomination mechanism for solution  Propose alternative nomination strategies

How can we use data reporting as part of the change process?  NSF data on women and key committees  Identifies a problem (too few women on them)  Requires a mechanism for solution (selection, election, etc.)  Propose alternative selection or election strategies

Making absence of data visible can help promote changes  No exit interviews  Poor data on attrition/retention  Need better data  Need better process

Must use data to support the change process  Institutional change is most persuasively based on data  Collecting, analyzing and reporting the data costly in time and resources  Evidence of change provides crucial support for hope, faith in the change process