1. ies.ed.gov Connecting Research, Policy and Practice Evaluation of State and Local Education Programs & Policies (84.305H) Allen Ruby, Ph.D. National Center for Education Research Jacquelyn Buckley, Ph.D. National Center for Special Education Research

2. ies.ed.gov Overview Overview of IES and its mission Requirements Specifics – Purpose – The project narrative Significance Partnership Research Plan Personnel Resources Other important sections of the application Preparing and submitting an application 2

3. ies.ed.gov Legislative Mission of IES Describe the condition and progress of education in the United States Identify education practices that improve academic achievement and access to education opportunities Evaluate the effectiveness of Federal and other education programs 3

4. ies.ed.gov Organizational Structure of IES 4 National Board for Education Sciences Standards & Review Office Office of the Director National Center for Education Evaluation National Center for Education Statistics National Center for Education Research National Center for Special Education Research

5. ies.ed.gov IES Grant Programs: Research Objectives Develop or identify education interventions (i.e., practices, programs, policies, and approaches) – that enhance academic achievement – that can be widely deployed Identify what does not work and thereby encourage innovation and further research Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness 5

6. ies.ed.gov Partnerships & IES Priorities IES seeks to... Encourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day- to-day work of education practitioners and policymakers Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies (See http://ies.ed.gov/director/board/priorities.asp)http://ies.ed.gov/director/board/priorities.asp 6

7. ies.ed.gov Short Description Evaluation of State and Local Programs & Policies (State/Local Evaluation) – New or established partnerships – Carry out rigorous evaluations of education programs or policies (from pre-k to adult education) that are implemented by state or local education agencies

8. ies.ed.gov General Requirements Focus on student education outcomes Partnership between research institutions and state or local education agencies (SEAs/LEAs) Evaluate SEA/LEA education programs and policies

9. ies.ed.gov Focus on Student Outcomes IES funds research to improve the quality of education for all students through advancing the understanding of and practices for teaching, learning, and organizing education systems Research must address education outcomes of students – Academic outcomes – Social and behavioral competencies that support student success in school 9

10. ies.ed.gov Student Population Students from prekindergarten through postsecondary and adult education – Typically developing students – Students with disabilities or at risk for disabilities see http://ies.ed.gov/ncser/definition.asp for specific requirements for identifying students at risk for disabilities statushttp://ies.ed.gov/ncser/definition.asp 10

11. ies.ed.gov Ultimate Outcomes of Interest: Student Outcomes GradeOutcome Prekindergarten (center based) School readiness (e.g., pre-reading, language, vocabulary, early math and science knowledge, social and behavioral competencies) Kindergarten – Grade 12 Learning, achievement, and higher-order thinking in reading, writing, mathematics, and science; progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout); social and behavioral competencies important to academic and post-academic success. 11

12. ies.ed.gov Ultimate Outcomes of Interest: Student Outcomes GradeOutcome Postsecondary (Grades 13 – 16) (baccalaureate and sub-baccalaureate) Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics; success in gateway math and science courses, introductory English composition Adult Education (Adult Basic Education, Adult Secondary Education, Adult ESL, and HS equivalency preparation) Student achievement in reading, writing, English language proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs 12

13. ies.ed.gov Additional Outcomes for Students With or At-Risk for Disabilities Applicants are encouraged to also include outcomes accepted under the NCSER grant programs Development outcomes for younger students – cognitive, communicative, linguistic, social, emotional, adaptive, functional or physical development Functional outcomes for older students – that improve educational results and transitions to employment, independent living, and postsecondary education

14. ies.ed.gov Applications must be from a Partnership Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from a U.S. state or local education agency – PI from research institution: Must have the ability and capacity to conduct scientifically valid research and expertise in the education issue to be addressed – PI from state or local education agency: Must have decision-making authority for the issue within his or her agency 14

15. ies.ed.gov Partnership Partnership may be new or existing Research institution has a broad definition – Ability and capacity to conduct scientifically valid research 15

16. ies.ed.gov Partnership: SEAs State education agencies – Examples: education agencies, departments, boards, commissions – Oversee early learning, elementary, secondary, postsecondary and/or adult education – Also includes education agencies in District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying areas

17. ies.ed.gov Partnership: LEAs Local education agencies which are primarily public school districts Community college districts Tribal education agencies State and city postsecondary systems – If there is a state or city higher education agency that oversees the postsecondary system, include them as an agency partner – If there is no state or city education agency that oversees the postsecondary system, the system can apply as the sole agency partner – A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project Adult education providers (defined under WIOA) can serve as the partner when there is no state or local education agency for adult education 17

18. ies.ed.gov Additional Partners Partnerships may include more than one state or local education agency if they share similarities and interests Intermediary or service districts that provide services to LEAs Non-education state and local agencies may be partners as long as an education agency is a partner Partnerships may include more than one research institution if they have shared interests and will make unique contributions Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups) that will contribute to the partnership and its work 18

19. ies.ed.gov Evaluate SEA/LEA Education Programs and Policies Program or policy of high importance to the SEA or LEA Substantial modification of existing practice Implemented by the SEA or LEA Implemented under routine conditions

20. ies.ed.gov Check the Fit of Your Research and the State Evaluation Grant Program If you are not looking at student outcomes, then IES is not the appropriate funding agency for your research If you need time and effort to build a partnership and carry out initial research in preparation for an evaluation, consider: – Researcher-Practitioner Partnerships under 84.305H If the program/policy you want to evaluate is not implemented by or of high importance to the SEA/LEA, consider: – Education Research Grants Program (84.305A) or – Special Education Research Grants Program (84.324A)

21. ies.ed.gov State/Local Evaluation: Purpose Promote joint evaluation research by research institutions and SEAs/LEAs – On an education program/policy of key importance to SEA/LEAs – That will directly contribute to SEA/LEA program and policy decisions – Provide opportunities to develop the partnership through the evaluation Foster longer-term research partnerships – Provide and support the use of rigorous research-based evidence in decision making – Continue practitioner input into research agenda 21

22. ies.ed.gov What should the partnerships do during the grant? Identify an education program or policy Implemented by an SEA or LEA Of high priority to that agency Intended to improve student education outcomes Carry out an evaluation of that program/policy – Overall impacts – Subgroup impacts – Fidelity of implementation – Cost analysis 22

23. ies.ed.gov Expected Products of the Grant Causal evidence of the impact of a clearly specified program/policy implemented by an SEA – Overall impacts – Impacts under a variety of conditions and/or by subgroup Conclusions on and revisions to the theory of change that guides the program/policy – Contributions to our theoretical understanding of education processes and procedures 23

24. ies.ed.gov Expected Products of the Grant If a beneficial impact is found… – The organizational supports, tools, and procedures needed for sufficient implementation of the core components of the program/policy under routine practice should be identified If a beneficial impact is not found… – A determination should be made whether and what type of further research would be useful to revise the program/policy and/or its implementation The financial costs of the program/policy 24

25. ies.ed.gov The Project Narrative Significance Partnership Research Plan Personnel Resources 25

26. ies.ed.gov Significance In the Significance section, clearly describe… The education program or policy to be evaluated – Components – Processes and materials to support implementation – Evidence it is ready to be or already implemented – How it differs from existing practice in the same location or different locations

27. ies.ed.gov Significance Its implementation – By an SEA or LEA – The target population and sites – End users of the program or policy and how they are to carry it out – Under routine conditions

28. ies.ed.gov Significance The theory of change – How the program or policy is to effect changes that ultimately lead to beneficial impacts on student outcomes – Intermediate outcomes in this process Rationale for testing its impact on student education outcomes – In widespread use but not well-evaluated – A component of the program/policy – Explain why the program/policy is likely to produce improved student outcomes relative to other practice Theoretical justification Empirical work

29. ies.ed.gov The Project Narrative Significance Partnership Research Plan Personnel Resources 29

30. ies.ed.gov Describe the Partnership Describe the partners – The research institution and the education agency – Any other members of the partnership – Common interest in and benefits from this evaluation – The process through which they decided to propose a State/Local Evaluation project – Past or ongoing collaborations and results from them – Management structure and procedures to keep the project on track and ensure quality of the research – Data sharing and housing agreement

31. ies.ed.gov Partnership Development Plan Partnership’s decision-making process Improving the education agency’s capacity to participate in and use education research – Identify the agency’s interests in capacity building – The agency’s specific understanding of the proposed research design and the validity and generalization of the evidence provided from it – The agency’s general capacity to understand and use research

32. ies.ed.gov The Project Narrative Significance Partnership Research Plan Personnel Resources 32

33. ies.ed.gov Research Plan State research questions and hypotheses Describe sample and setting – Define population and how your sample and sampling procedures will allow inferences to the population – Exclusion and inclusion rules and their justification – Strategies used to increase participation and reduce attrition – Describe the setting and its implications for the generalizability of your study 33

34. ies.ed.gov Research Plan: Design Rationale for the Selected Research Design Causal inference Threats to internal validity Describing and justifying the counterfactual Degree of equivalence at baseline Bias from overall and differential attrition Meet WWC evidence standards (with or without reservations) 34

35. ies.ed.gov Research Plan: Design Preferred Design: Randomized Controlled Trial (RCT) Note unit of randomization and justify choice Describe process for random assignment and maintaining its integrity Different Approaches to RCTs - Potential Issues – Entire population-mandatory: Treatment fidelity – Volunteers: Comparison group status – Lotteries: Attrition of non-accepted parties – Staggered roll out: Little time for true comparison – Variations of program/policy: Issue of overall significance 35

36. ies.ed.gov Research Plan: Design Alternatives to the RCT Design If RCT is not possible, justify why Alternatives to minimize or model selection bias – Regression discontinuity designs – Well-designed quasi-experimental designs Comparative interrupted time series 36

37. ies.ed.gov Research Plan: Statistical Power Detailed description of power analysis and justification for method used to calculate power – Including assumptions Power for main analyses and important subgroup analyses Practical meaning of minimum detectable effect sizes Reviewers should be able to check power calculations 37

38. ies.ed.gov Research Plan: Outcome Measures Student education outcome measures relevant to states, districts, and schools – Often found in administrative data – Can include researcher-developed measures but not as the primary outcome measures Provide reliability, validity, and appropriateness Include measures of intermediate outcomes – For example, if program/policy is to change instruction, describe measures of instruction Link measures to theory of change 38

39. ies.ed.gov Research Plan: Moderators & Mediators May explain differential impacts of intervention Identified in theory of change Describe how they will be measured in both treatment and control Discuss if doing exploratory or confirmatory analysis of each one examined Describe analysis plan 39

40. ies.ed.gov Value of Mixed Methods (Quantitative & Qualitative) To better understand any causal relationship – To avoid relying on non-empirical based explanations To understand the fidelity of implementation and comparison group practice To examine intermediate outcomes and mediators Have a team that can do both

41. ies.ed.gov Research Plan: Fidelity of Implementation Describe your plan for determining the fidelity of implementation of the program or policy Describe how your fidelity measures capture core components of the program or policy – Note their psychometric properties Discuss how data will be analyzed and will contribute to overall evaluation Initial study of fidelity can be used to provide input to SEA/LEA to improve implementation 41

42. ies.ed.gov Research Plan: Comparison Group Practice Describe who makes up comparison group Detail how you will measure whether they are similar/different from treatment group Detail how you will measure what they receive in place of the treatment Determine if control group receives components similar to intervention and how much 42

43. ies.ed.gov Research Plan: Analysis Detailed description of data analysis procedures Make clear how analyses directly answer your research questions and can be done based on the design Quantitative: Statistical procedures, model, and software Qualitative: Methods to index, summarize, and interpret data Will quantitative and qualitative data be used for separate or combined analyses? Address clustering of students in classrooms in schools Address missing data Include plans for analyses of subgroups, mediators, moderators, and fidelity of implementation Examine attrition (overall and differential) 43

44. ies.ed.gov Research Plan: Cost Analysis Document financial costs of program implementation – Detailed enough for another SEA/LEA to use Can include a cost-effectiveness or cost-benefit analysis but not required 44

45. ies.ed.gov The Project Narrative Significance Partnership Research Plan Personnel Resources 45

46. ies.ed.gov Personnel Identify all key personnel on the project team – Roles and responsibilities on the project – Qualifications (i.e., expertise and experience) to carry out the roles and responsibilities – % FTE on the project (one key person should have enough time to maintain progress of project) – Past success at working in similar partnerships PI qualifications for managing a grant of this size and type Ensure objectivity of evaluation 46

47. ies.ed.gov Resources Describe the institutional resources of all the institutions involved in the partnership and how these resources will contribute to building the partnership and to the research – Institutional capacity to manage the grant – Resources available at the partner institutions that will be used – Plans to acquire any major resources not yet in hand – Joint Letter of Agreement by partners (Appendix D) 47

48. ies.ed.gov Resources If districts or schools are taking part… – Districts and schools should document their involvement – E.g., Letters of Agreement in Appendix D If secondary data is being analyzed… – The organization holding those data should document their willingness to provide the data – E.g., Letters of Agreement in Appendix D If district/school staff are taking part… – E.g., through surveys, observations, logs – Discuss how their cooperation will be obtained (e.g., use of incentives) and their current knowledge of the project 48

49. ies.ed.gov Resources: Dissemination of Results Results are expected to be useful to the SEA/LEA partner and other SEAs/LEAs – Both findings of beneficial impacts or no impacts Describe your capacity to disseminate findings Identify all your audiences and how you will disseminate the results to them – The SEA/LEA (through an ongoing process) – Other education agencies, policymakers, and practitioners – The research community – The public 49

50. ies.ed.gov Other Important Sections of the Application Appendix A Appendix B Appendix C Appendix D Budget & Budget Narrative 50

51. ies.ed.gov Appendix A Page Limit: 3 If you are resubmitting an application, use up to 3 pages to discuss how you responded to reviewer comments 51

52. ies.ed.gov Appendix B Page Limit: 15 Figures, charts, or tables that supplement the project narrative Timelines for the project (very useful) Examples of measures to be used – E.g., tests, surveys, observation, and interview protocols Do not include narrative text

53. ies.ed.gov Appendix C Page Limit: 10 Examples of materials used in the program or policy: – curriculum material – computer screen shots – training documents – assessment items – other materials 53

54. ies.ed.gov Appendix D No Page Limit Letters of Agreement from all the research partners – Joint Letter from key partners – Separate Letters from other organizations involved – Letters should clearly state the organization’s expected role in the partnership and their commitments to the project – Similar letters from any consultants, districts, and schools taking part – Letters from holders of data should make clear that the data described in the application will be provided for the proposed use by the project 54

55. ies.ed.gov Budget & Budget Narrative Maximum project length is 5 years Maximum award is $5 million – Funds must be used for evaluation only (e.g., cannot be used for implementation of the program/policy) – Award size depends on project scope Include a detailed budget form (SF 424) and a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested 55

56. ies.ed.gov Application Deadline Letter of Intent Due Date Application Package Posted Start Dates August 4, 2016 4:30:00 PM DC Time May 19, 2016 July 1 to September 1, 2017 Important Dates & Deadlines 56

57. ies.ed.gov Information Sources Request for Applications – http://ies.ed.gov/funding/ http://ies.ed.gov/funding/ Abstracts of Projects – http://ies.ed.gov/funding/grantsearch/index.asp http://ies.ed.gov/funding/grantsearch/index.asp Application Package – www.grants.gov www.grants.gov Program Officers – Allen.Ruby@ed.gov Allen.Ruby@ed.gov – Jacquelyn.Buckley@ed.gov Jacquelyn.Buckley@ed.gov 57

58. ies.ed.gov Peer Review (Standards & Review Office) Compliance screening for format requirements Responsiveness screening for program requirements Assignment to review panel – 2 to 3 reviewers (substantive and methodological) – The most competitive proposals are reviewed by full panel Many panelists will be generalists to your topic Panels contain experts in relevant methodologies – Panel provides an overall score plus specific scores on Significance, Partnership, Research Plan, Personnel, and Resources 58

59. ies.ed.gov Notification All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS): Status of award Reviewer summary statement If you are not granted an award the first time, consider resubmitting and talking with your Program Officer 59

60. ies.ed.gov For More Information http://ies.ed.gov/funding Jacquelyn Buckley National Center for Special Education Research Jacquelyn.Buckley@ed.gov Allen Ruby National Center for Education Research Allen.Ruby@ed.gov 60