1. Choosing & Working With An Evaluator Welcome!! We will begin approximately 2 minutes past the half-hour. Please MUTE your telephone. If you have no mute button, you may press *6 on your phone, or press the Mute Me icon on your screen.

2. Welcome COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Nan Maxwell Senior Researcher Mathematica Policy Research

3. Welcome COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Shanna Jaggars Senior Research Associate CC Research Center Teachers College, Columbia

4. Agenda Overview Welcome – Nan Maxwell, MPR – Shanna Jaggars, CCRC Writing an Effective RFP – Ann Person, MPR – Dawn Busick, Missouri CC Association Negotiating with your Evaluator – Shanna Jaggars, CCRC – Chris McRoberts, NW Arkansas CC (Arkansas) Creating an Effective Partnership – Debra Bragg, Office of CC Research & Leadership (U. Illinois) – DeRay Cole, Robeson CC (North Carolina) COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

5. Handling Q&A Use the chat function to type in relevant questions at any point. If you have answers / input on someone else’s question, feel free to type a response. We will monitor the written questions and choose 1 or 2 to ask the presenters after each topic. If time allows, we will also answer questions at the end more generally. Key / common questions & answers will be summarized and sent out to all participants. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

6. Guidelines for Preparing a Request for Proposals Ann Person Mathematica Policy Research TAACCCT Evaluation TA Webinar June 28, 2012 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

7.  Who? Describe your consortium/program  Why? Describe what you want to get from the evaluation  What? Describe the scope of work for the evaluation  How? Describe terms of contract, proposal requirements, and vendor selection criteria  When? List proposal process dates and period of performance The RFP: Answer basic questions 7 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

8.  Describe TAACCCT grant program and its goals  Describe your consortium  Describe your program/theory of change –Inputs –Activities –Participation –Outputs –Outcomes Who: Overview and Context 8 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

9.  What do you want to learn? –Implementation study –Outcomes study  Who are the primary audiences? –Institutions, systems –DOL –The broader field  How will results be used (by the different audiences)? Why: Statement of Purpose 9 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

10.  Design: –How will they define the comparison cohort? –How will they collect, aggregate, and analyze data for implementation/progress/outcome measures? –Will they need IRB clearance? How will they get it?  Activities –Key evaluation tasks –Key engagement tasks  Deliverables  If you’re unsure about specifics, describe what you want the end result to look like What: Scope of Work 10 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

11.  Roles and responsibilities –To whom will the evaluator answer? –How will they engage with different stakeholders?  Budget –Know what you can get for the funding you have –Don’t be coy about how much you have available  Your procurement procedures How: Terms of Contract 11 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

12.  Proposal Requirements –Statement of work –Vendor qualifications –Budget  Selection criteria –How much weight will you give to each section above? –Consider trade-offs (e.g., timing/cost/quality) How: Proposal Requirements & Selection Criteria 12 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

13.  RFP process timeline  Period of performance  Deliverable dates (align with DOL?)  Activity/engagement dates  Gantt charts map activities and deliverables over time – you can develop one or include as a requirement of the proposal When: Timeline 13 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

14.  How will you distribute your RFP and to whom?  How will you ensure a good range of applicants?  Salient procurement policies and legal requirements? RFP Process: A few questions 14 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

15.  Be clear  Avoid jargon  Describe end result  Include supporting materials as appendices  Don’t make the vendor guess what you want – but rely on their expertise to figure out how to deliver on it Overarching Principles 15 COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH

16. Dawn Busick Project Director MoHealthWINs Missouri CC Association Writing an Effective RFP

17. Writing an Effective RFP – Q & A COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Michelle Hodara Postdoctoral Research Associate CC Research Center Teachers College, Columbia

18. Negotiating with your Evaluator COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Shanna Jaggars Senior Research Associate CC Research Center Teachers College, Columbia

19. Clarify your learning goals COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH “Just tell us if it works” Your role: provide data, clear description of program Their role: Decide on quantitative evaluation plan; clean & analyze data; provide results Least expensive, but also least informative

20. Clarify your learning goals COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH “Just tell us if it works” Your role: provide data, clear description of program Their role: Decide on quantitative evaluation plan; clean & analyze data; provide results “Help us understand why/how it works” Your role: collaborate on evaluation plan; provide access to staff, faculty, students; review ongoing qualitative & quantitative findings together Their role: understand implementation variation across sites or subject areas; implementation successes & challenges; faculty and student reactions More expensive, but much more informative

21. Clarify your learning goals COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH “Just tell us if it works” Your role: provide data, clear description of program Their role: Decide on quantitative evaluation plan; clean & analyze data; provide results “Help us understand why/how it works” Your role: collaborate on evaluation plan; provide access to staff, faculty, students; review ongoing qualitative & quantitative findings together Their role: understand implementation variation across sites or areas of study; implementation successes & challenges; faculty and student reactions “Help us learn how to do this on our own in the future” Your role: Active participation in all aspects of planning, qualitative data collection, and qual/quant data analysis Their role: A teacher; performing evaluation for its own sake, but also for the larger goal of involving you actively in the process Requires more staff time, but most useful in long run Different evaluators will have different levels of comfort with / skill in handling these different types of goals

22. Explicit conversation about their needs/wants COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Common Evaluator NEEDS Commitment from top people Access to key people involved Project manager returns my calls/emails Access to necessary data Enough time to do the job right Enough money to do the job right I will not be asked to evaluate people Common Evaluator WANTS Top people make support & commitment clear to those involved Project manager works closely with me Access to helpful data Very clearly-specified mutually-agreed timeline Task-specific assistance from local personnel Adapted from Peter Block, “Flawless Consulting” Clarify differences between their needs and wants Make clear which needs/wants you can meet If you’re not able to meet needs, the evaluator cannot effectively do the job If you’re not able to meet wants, evaluation may require more time, more money, or result in less useful evaluation.

23. Example: Quantitative data needs vs. wants COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH CategoryData ElementPriority Student identifier Unique student record identifier (NOT student’s actual SSN or college id number)1 Cohort of entry (term and year)1 Demographics Date of birth, or age at entry1 Gender1 Race/ethnicity1 Non-Native English speaker3 Geo-code for student home address3 Educational background High school diploma or GED, with date of award2 College credits transferred from other institutions2 Prior postsecondary credentials2 Prior enrollment in any adult basic education, GED or ESL courses at college2 Tested abilityPlacement test scores (English, reading, math) and dates2 ACT or SAT scores by subject area and date3 Priority 1: Needed as soon as possible Priority 2: Necessary, but can wait a while before receiving Priority 3: Not absolutely necessary, but would be very helpful

24. Negotiating with your Evaluator COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Chris McRoberts Executive Director, Path to Accelerated Completion & Employment (PACE) Northwest Arkansas CC

25. Negotiating with your Evaluator – Q & A COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Michelle Hodara Postdoctoral Research Associate CC Research Center Teachers College, Columbia

26. Creating an Effective Partnership: An Interview with Debra Bragg COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Debra Bragg Professor, Education Policy, Organization, and Leadership; Director, Office of CC Research & Leadership University of Illinois at Urbana-Champaign

27. A good experience COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Consider evaluation from the start Openness Curiosity Communication Trust

28. A less-good experience COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Mismatch of expectations Not allowing process to evolve

29. Two key things COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Allow evaluator to get to know leadership, colleges Don’t panic – tap your evaluator’s knowledge

30. TAA-specific advice COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Identification / documentation / mapping of Program of Study Understand data capacity of each college – don’t duplicate work

31. Creating an Effective Partnership COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH DeRay Cole Project Manager North Carolina Advanced Manufacturing Alliance Robeson Community College

32. Creating an Effective Partnership – Q & A COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Michelle Hodara Postdoctoral Research Associate CC Research Center Teachers College, Columbia

33. General Q & A COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH “Raise your hand” if you would like to contribute to a question or answer If Michelle calls on you, un-mute your phone to allow you to reply When you are done speaking, re-mute your phone

34. Upcoming Events COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Evaluator convening, August 7-8, probably in Chicago, Ill. Follow-up webinar, probably September 12