1. June 11, 2010 Development Panel Monitor Orientation Investing in Innovation (i3) Orientation Note: These slides are intended for internal use only and as guidance to help you complete the Validation peer review process.

2. 2 What is the role of a PANEL MONITOR? 2 Contact reviewers to schedule panel calls Assist reviewers with any questions Identify any conflicts of interest Facilitate panel calls for all 20 applications on each panel Make sure reviewers enter comments/scores in e-Reader promptly Promptly send feedback on written comments/scores to reviewers and notify reviewers when scores are to be SUBMITTED in final form Panel Monitor

3. 3 Investing in Innovation (i3) Fund Summary 3 PurposeFundingApplicants To provide competitive grants to applicants with a record of improving student achievement and attainment in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on: Improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates $650 million to be obligated by September 30, 2010 Eligible applicants are: (1)Local educational agencies (LEAs) (2)Nonprofit organizations in partnership with (a) one or more LEAs or (b) a consortium of schools

4. 4 Types of Awards Available Under i3 4 i3 Estimated Funding Available Up to $5MM/awardUp to $30MM/awardUp to $50MM/award Evidence Required Reasonable – research findings or hypotheses, including related research or theories in education and other sectors Moderate – either high internal validity and medium external validity, or vice versa Strong – both high internal validity and high external validity Scaling Required Able to further develop and scale Able to be scaled to the regional or state level Able to be scaled to the national, regional, or state level

5. 5 5 MUST Eligible Applicant: LEA Eligible Applicant: LEA Demonstrate that it: (a) significantly closed achievement gaps between groups of students or demonstrated success in significantly increasing academic achievement for all groups of students, and (b) made significant improvement in other areas Establish partnerships with private sector Secure commitment for required 20% private sector match Meet the evidence requirement for the type of grant for which it has applied Demonstrate that it: (a) significantly closed achievement gaps between groups of students or demonstrated success in significantly increasing academic achievement for all groups of students, and (b) made significant improvement in other areas Establish partnerships with private sector Secure commitment for required 20% private sector match Meet the evidence requirement for the type of grant for which it has applied TO RECEIVE A GRANT, MUST

6. 6 6 MUST Eligible Applicant: Non-profits, in partnership with LEA(s) or a consortium of schools Eligible Applicant: Non-profits, in partnership with LEA(s) or a consortium of schools Demonstrate that the non-profit organization has a record of significantly improving student achievement, attainment, or retention through its record of work with an LEA or schools Secure commitment for required 20% private sector match Meet the evidence requirement for the type of grant for which they have applied Demonstrate that the non-profit organization has a record of significantly improving student achievement, attainment, or retention through its record of work with an LEA or schools Secure commitment for required 20% private sector match Meet the evidence requirement for the type of grant for which they have applied TO RECEIVE A GRANT, MUST

7. 7 All Eligible Applicants Must Implement Practices, Strategies, or Programs for High-Need Students 7 MUST High-need student means a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high- minority schools, who are far below grade level, who are over- age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient.

8. 8 8 1. Teacher and Principal Effectiveness 2. Improved Use of Data Systems 3. College- and Career- ready Standards and High Quality Assessments 4. Improving Achievement in Persistently Low-performing Schools 5. Early Learning (0 or 1 point) 6. College Access and Success (0 or 1 point) 7. Serving Students with Disabilities and Limited English Proficient Students (0 or 1 point) 8. Serving Students in Rural LEAs (0, 1, or 2 points) i3 Priorities Must select one (Absolute Priority) May select one or more (Competitive Preference Priorities)

9. 9 9 Notes on Absolute Priority 1 Innovations that Support Effective Teachers and Principals …increase the number or percentages of teachers or principals who are highly effective teachers or principals or reduce the number or percentages of teachers or principals who are ineffective, especially for teachers of high-need students… …by identifying, recruiting, developing, placing, rewarding, and retaining highly effective teachers or principals (or removing ineffective teachers or principals). …teacher or principal effectiveness should be determined through an evaluation system that is rigorous, transparent, and fair; performance should be differentiated using multiple rating categories of effectiveness; multiple measures of effectiveness should be taken into account, with data on student growth as a significant factor, and the measures should be designed and developed with teacher and principal involvement. Two Possible Routes Multiple Measures of Effectiveness Multiple Methods

10. 10 Notes on Absolute Priority 2 Innovations that Improve the Use of Data …(a) encourage and facilitate the evaluation, analysis, and use of student achievement or student growth data by educators, families, and other stakeholders in order to inform decision- making and improve student achievement, student growth, or teacher, principal, school, or LEA performance and productivity; or (b) enable data aggregation, analysis, and research …data must be disaggregated using the student subgroups described in section 1111(b)(3)(C)(xiii) of the ESEA… Two Possible Areas of Focus Data Disaggregation

11. 11 Notes on Absolute Priority 3 Innovations that Complement the Implementation of High Standards and High-Quality Assessments …standards and assessments that measure students progress toward college and career-readiness… …may include, but are not limited to, … increase the success of underrepresented student populations in academically rigorous courses and programs…; increase the development and use of formative assessments or interim assessments, or other performance-based tools and metrics that are aligned with high student content and academic achievement standards; or translate the standards and information from assessments into classroom practices that meet the needs of all students, including high-need students. …eligible applicant must propose a project that is based on standards that are at least as rigorous as its States standards… Focus on College & Career Readiness Range of Allowable Projects Rigorous Standards

12. 12 Notes on Absolute Priority 4 Innovations that Turn Around Persistently Low-Performing Schools Whole-school reform, including, but not limited to, comprehensive interventions to assist, augment, or replace Investing in Innovation Fund Absolute Priority 4 schools, including the school turnaround, restart, closure, and transformation models of intervention … OR … Targeted approaches to reform, including, but not limited to: Providing more time for students to learn core academic content by expanding or augmenting the school day, school week, or school year, or by increasing instructional time for core academic subjects integrating student supports into the school model to address non-academic barriers to student achievement creating multiple pathways for students to earn regular high school diplomas Projects May Choose Either Approach

13. 13 i3 Priority 4 Schools Under Absolute Priority 4, the Department provides funding to support strategies, practices, or programs that are designed to turn around schools that are in any of the following categories: (a)persistently lowest-achieving schools (as defined in the final requirements for the School Improvement Grants program, see http://www2.ed.gov/programs/sif/faq.html); http://www2.ed.gov/programs/sif/faq.html (b)Title I schools that are in corrective action or restructuring under section 1116 of the Elementary and Secondary Education Act (ESEA); or (c)secondary schools (both middle and high schools) eligible for but not receiving Title I funds that, if receiving Title I funds, would be in corrective action or restructuring under section 1116 of the ESEA.

14. 14 i3 Absolute Priorities Applicants must address ONE Absolute Priority No perks for addressing more than one Absolute Priority Absolute Priority is not scored 14

15. 15 Notes on Competitive Preference Priority 5 Innovations for Improving Early Learning Outcomes …improve educational outcomes for high-need students who are young children (birth through 3rd grade) by enhancing the quality of early learning programs …(a) improving young childrens school readiness (including social, emotional, and cognitive readiness) so that children are prepared for success in core academic subjects (as defined in section 9101(11) of the ESEA); (b) improving developmental milestones and standards and aligning them with appropriate outcome measures; and (c) improving alignment, collaboration, and transitions between early learning programs that serve children from birth to age three, in preschools, and in kindergarten through third grade. Focus on High- need Children Projects Must Address All 3

16. 16 Notes on Competitive Preference Priority 6 Innovations that Support College Access and Success … enable kindergarten through grade 12 (K–12) students, particularly high school students, to successfully prepare for, enter, and graduate from a two- or four-year college… …(a) address students preparedness and expectations related to college; (b) help students understand issues of college affordability and the financial aid and college application processes; and (c) provide support to students from peers and knowledgeable adults. Focus on College Graduation Projects Must Address All 3

17. 17 Notes on Competitive Preference Priority 7 Innovations to Address the Unique Learning Needs of Students with Disabilities and Limited English Proficient Students …address the unique learning needs of students with disabilities, including those who are assessed based on alternate academic achievement standards, or the linguistic and academic needs of limited English proficient students. …must provide for the implementation of particular practices, strategies, or programs that are designed to improve academic outcomes, close achievement gaps, and increase college- and career-readiness, including increasing high school graduation rates (as defined in this notice), for students with disabilities or limited English proficient students. Focus on Either Student Population Projects That Improve Specific Outcomes

18. 18 Notes on Competitive Preference Priority 8 Innovations that Serve Schools in Rural LEAs …focus on the unique challenges of high- need students in schools within a rural LEA… …must include practices, strategies, or programs that are designed to improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or improve teacher and principal effectiveness in one or more rural LEAs. Focus on Specific Locations Projects May Address Range of Outcomes

19. 19 Scoring the Competitive Preference Priorities (CPP) Applicants may address ONE or MORE CPPs CPP 5, 6, and 7 are scored as all or nothing (1 or 0) CPP 8 is scored with 0,1, or 2 A CPP may have a score of 0 or 1 and still have strength and weakness comments Only the subject matter reviewers will score the CPPs. 19

20. 20 i3 Selection Criteria and Points 20 * Development grants will be judged in two tiers: all eligible applications will be scored on Criteria A, C, E, F, and G and the competitive preference priorities; then high-scoring applications will be scored on Criteria B and D by a different panel of reviewers. Selection CriteriaDevelopmentValidationScale-Up Need for the Project and Quality of the Project Design 252015 Strength of Research, Significance of Effect and Magnitude of Effect 10*1520 Experience of the Eligible Applicant 252015 Quality of the Project Evaluation 15*15 Strategy and Capacity to Bring to Scale or to Further Develop and Bring to Scale 51015 Sustainability 10 Quality of the Management Plan and Personnel 10 Total Points 100

21. Subject Matter Reviewers will Score Criteria A, C, E, F, and G (NOT the Evidence and Evaluation Criteria) Subject Matter Review

22. 22 Development Grants A. Need for the Project and Quality of the Project Design (up to 25 points) 22 The Secretary considers the need for the project and quality of the design of the proposed project. In determining the need for the project and quality of the design of the proposed project, the Secretary considers the following factors: 1)The extent to which the proposed project represents an exceptional approach to the priorities the eligible applicant is seeking to meet (i.e., addresses a largely unmet need, particularly for high-need students, and is a practice, strategy, or program that has not already been widely adopted). 2)The extent to which the proposed project has a clear set of goals and an explicit strategy, with the goals, objectives, and outcomes to be achieved by the proposed project clearly specified and measurable and linked to the priorities the eligible applicant is seeking to meet.

23. 23 Development Grants C. Experience of the Eligible Applicant (up to 25 points) 23 The Secretary considers the experience of the eligible applicant in implementing the proposed project. In determining the experience of the eligible applicant, the Secretary considers the following factors: The past performance of the eligible applicant in implementing projects of the size and scope proposed by the eligible applicant. The extent to which an eligible applicant provides information and data demonstrating that (a) In the case of an eligible applicant that is an LEA, the LEA has (i) Significantly closed the achievement gaps between groups of students described in section 1111(b)(2) of the ESEA, or significantly increased student achievement for all groups of students described in such section; and (ii) Made significant improvements in other areas, such as graduation rates or increased recruitment and placement of high- quality teachers and principals, as demonstrated with meaningful data; or (b) In the case of an eligible applicant that includes a nonprofit organization, the nonprofit organization has significantly improved student achievement, attainment, or retention through its record of work with an LEA or schools.

24. 24 Development Grants E. Strategy and Capacity to Further Develop and Bring to Scale (up to 5 points) 24 The Secretary considers the quality of the eligible applicants strategy and capacity to further develop and bring to scale the proposed project. In determining the quality of the strategy and capacity to further develop and bring to scale the proposed project, the Secretary considers: The number of students proposed to be reached by the proposed project and the capacity of the eligible applicant and any other partners to reach the proposed number of students during the course of the grant period. 2)The eligible applicants capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to further develop and bring to scale the proposed practice, strategy, or program, or to work with others (including other partners) to ensure that the proposed practice, strategy, or program can be further developed and brought to scale, based on the findings of the proposed project. The feasibility of the proposed project to be replicated successfully, if positive results are obtained, in a variety of settings and with a variety of student populations. Evidence of this ability includes the availability of resources and expertise required for implementing the project with fidelity, and the proposed projects evidence of relative ease of use or user satisfaction. The eligible applicants estimate of the cost of the proposed project, which includes the start-up and operating costs per student per year (including indirect costs) for reaching the total number of students proposed to be served by the project. The eligible applicant must include an estimate of the costs for the eligible applicant or others (including other partners) to reach 100,000, 250,000, and 500,000 students. The mechanisms the eligible applicant will use to broadly disseminate information on its project so as to support further development or replication.

25. 25 Development Grants F. Sustainability (up to 10 points) 25 The Secretary considers the adequacy of resources to continue the proposed project after the grant period ends. In determining the adequacy of resources for the proposed project, the Secretary considers the following factors: The extent to which the eligible applicant demonstrates that it has the resources, as well as the support of stakeholders (e.g., State educational agencies, teachers' unions), to operate the project beyond the Development grant. The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the eligible applicant and any other partners at the end of the Development grant.

26. 26 Development Grants G. Quality of the Management Plan and Personnel (up to 10 points) 26 The Secretary considers the quality of the management plan and personnel for the proposed project. In determining the quality of the management plan and personnel for the proposed project, the Secretary considers: The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks. The qualifications, including relevant training and experience, of the project director and key project personnel, especially in managing projects of the size and scope of the proposed project.

27. DEVELOPMENT REVIEW Review Process & Reviewer Expectations

28. 28 What is the basic i3 review process? 33 For all three grant types… The Department will use independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest Applications will be assigned to panels by absolute priority where possible Evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation in the Tier 2 review Peer reviewers will determine whether any competitive preference priority points should be added Development Only… Development applications will be reviewed in a two tier process –In Tier 1, all eligible applicants will be reviewed and scored against Selection Criteria A, C, E, F and G by three peer reviewers. Competitive Preference Points will also be added as appropriate by peer reviewers. –Only those highest rated in Tier 1 will advance to Tier 2, where Selection Criteria B and D will be scored by two peer reviewers who are evidence and evaluation experts.

29. 29 What is expected of a Peer Reviewer? 34 ProcessBehavior Review the entire i3 application package and FAQs Review and become thoroughly familiar with the selection criteria, factors, and notes Familiarize yourself with the e-Reader system – used for inputting scores and comments Participate in all scheduled conference calls Provide appropriate comments that justify the score awarded and are helpful to the applicant Revise comments as suggested by your panel monitor Return all forms as required to ensure payment and completion of review process Be available the entire review process Draw upon your expertise Maintain confidentiality throughout the review process Preparation

30. 30 What must a Peer Reviewer really do? Receive applications and panel assignments from Synergy Identify any Conflicts of Interest, and notify Synergy or panel monitor immediately Participate in Orientation webinar Register in and become acquainted with e-Reader Read applications (It may be helpful to you to take notes!) Write draft comments, assign scores, and submit drafts in e- Reader before panel discussion (We recommend cutting and pasting from Word) Receive feedback on comments from Panel Monitor Revise draft comments based on feedback as appropriate When comments/scores are acceptable to Panel Monitor, submit final comments/scores Sign and return Technical Review Signature Forms to Synergy Remember to submit All Necessary Forms to Synergy 35 Prior to Panel Discussion After Panel Discussion

31. 31 Registering in e-Reader http://e-grants.ed.gov Call the e-Grants help desk at 1-888-336-3930 to request a PIN# for the registration process Click the Continue button and the Register button Indicate that you are an ED Employee and select the e- Reader module Complete the User Registration page and Submit A system generated password will be e-mailed to you Use the username and password to login Existing e-Reader users should use their existing username and password. Help Desk Hours of Operation Tuesday, Friday and Saturday: Available 24 hours Monday and Thursday: 6am – midnight; Sunday: midnight – 8pm and Wednesday: midnight – 7pm

32. 32 Panel Call 101 Panel calls are held at 8:30 AM, 11:30 AM, and 2:30 PM daily Reviewers/Panel Monitors have selected their preferred time slot Panel calls are held by phone and each panel has its own conference call number 4-6 Applications should be covered on each panel call until all 20 applications are completed Panel calls will likely last the entire 2 ½ hour period June 14-18, all panel calls should be complete Review follow up should be completed by June 25, 2010 Adjustments to panel call times may be arranged between the reviewers and panel monitor Be FLEXIBLE and expect EMERGENCIES

33. 33 Conducting a Panel Call Contact reviewers no later June 11 to schedule panel calls Use the first call to go over the review process, introductions, and answer initial questions Go over the selection criteria and CPPs for each application Have reviewers discuss strengths and weaknesses for each criterion and CPPs, if applicable Use prompt questions & reviewers scores to generate panel discussions Use the score comparison sheet to keep track of reviewers scores and scoring changes Make sure scores received are justified by the written comments If needed, conduct a second panel call to further discuss applications that have a 20 percent variance Comments should be continuously reviewed until you give reviewers final approval to SUBMIT comments in e-Reader 43

34. 34 Reviewing Application Scores Numeric scores indicate how well the applicant responded to the selection criteria Use the entire range of points for each criterion. There is NO scoring rubric, so all factors should be evaluated equally For each of the competitive preference priorities, applicants do not earn a point simply for addressing the priority. Reviewers must award points based on how well applicants address the requirements of the priority. Make sure that your comments are consistent with your numeric scores If full points are awarded to a criterion, all comments should be in strengths section and no weaknesses found should be entered in the weaknesses section. If partial points are awarded to a criterion, there should be points and comments under both the strengths and weaknesses sections. ZERO means that something is missing from a criterion and all reviewers MUST agree in order to award ZERO points to a criterion For CPPs, ZERO means that the applicant did not provide a quality response to the priority. All reviewers DO NOT have to agree to award ZERO points on a CPP 37

35. 35 Reviewing Application Comments 38 DosDonts Application should be read and scored against the selection criteria Comments should support the numeric score and should be based on the strengths and weaknesses of an applications responses to the selection criteria. Comments should be clear, concise, and objective. Comments should be written in complete sentences Remind reviewers to use the spelling and grammar check on all comments before they submit Remind reviewers to include application page numbers as they are helpful in the comments Dont compare applications. The should be evaluated independently Dont Summarize, paraphrase or quote information presented in the application without adding comments that explain your judgment about that information Dont Write one- or two-word comments to justify your scores Dont Use inflammatory or derogatory language when writing comments Dont ask questions of the applicant in comments Dont copy and paste comments from applications and use them in other applications

36. 36 Reviewing Application Comments (contd) Provide feedback/edits to reviewers on their comments promptly (next slide) Have reviewers keep you posted on the status of their comments Develop a system to track what you have reviewed, need to review, need from reviewers, and what TRFs are complete Keep in mind that applicants will be receiving the reviewer comments, so review them thoroughly Keep in mind that comments may be made public, so review them thoroughly 39

37. 37 How to Provide Feedback to Reviewers Copy and paste reviewers comments from e- Reader into a Word doc, then use Track Changes to edit comments. Email this document to the reviewer Use Outlook or e-Reader to email edits/feedback to reviewers. You could list edits for each application by criterion Either way is fine, use what works best for you

38. 38 Sample Comment – #1 The application proposes to create a professional development program for teachers that will instruct them how to use performance data from class assignments and to use it to improve their teaching. Its an excellent idea. Problems: The comment simply paraphrases the application, and it does not explain why the reviewer likes the idea. Revision: One of the strengths of the proposed professional development program is that it gives teachers direct practice in using the newly created online repository of student assignments that can be used in the moment when classroom assessments identify a learning deficiency. As the application demonstrates on p. 17, this technology-based approach aligns with research showing that learning deficits can be remedied if addressed immediately. Why is it Better?: It explains more clearly why the reviewer believes this is a quality program, and it includes specific references to the proposal. 40

39. 39 Sample Comments – #2 This management plan seems thorough and thoughtful, with appropriate LEA representation on the advisory council. The allocation of leadership and responsibilities seems appropriate to the tasks, and the schedule laid out in the chart seems reasonable. Problems: At first glance, these comments seem fine, but they are not as helpful as they could be. More detail is necessary to explain what about the management plan is thorough and thoughtful. What characteristics of the schedule or the allocation of responsibilities is the reviewer referencing? Partial Revision: …It makes sense in this case to split administrative leadership of the project from the coordination of the each of the school-based implementation sites and to assign these duties to different individuals… Why is it Better?: The revision makes more clear what specific aspects of the leadership plan are appropriate and reasonable. 41

40. 40 Sample Comments – #3 (Weakness) The administrative employee is on contract and is not a full- time employee. Is that advisable? Problem: It may appear that points have been deducted based on an issue of uncertainty, and the applicant cannot respond. Its generally best to avoid questions. No weakness has been specifically identified. Revision: Because the administrative employee is contracted, rather than serving as a full-time employee, it is unclear whether the applicant would be able to absorb this individuals critical functions after the grant period is over. Why is it Better?: The revision refrains from asking a question and more clearly explains why there might be a weakness in the way this employee has been budgeted. 42

41. 41 Accessing the Technical Review Form (TRF) Log into e-Reader, select Reports, select 84.396C (2010) Score Comparison gives you overall scores; selecting the application gives you a breakdown of scores Reviews brings you to the TRF for each reviewer for each application Select a TRF for a reviewer, then select Print/View Form to view or print the TRF

42. 42 Disable Smart Quotes in MS Word Peer Reviewers are encouraged to write their comments is MS Word, then copy and past them into e-Reader. Before you start typing in MS Word, you should disable Smart Quotes to avoid formatting problems in e-Reader. The instructions are below: oIn MS Word on the Tools menu, click AutoCorrect. oGo to the AutoFormat As you Type tab. oUncheck the option Straight quotes with smart quotes. oUncheck the option Symbol characters (- -) with symbols (--). oGo to the AutoFormat tab. oUncheck the option Straight quotes with smart quotes. oUncheck the option Symbol characters (- -) with symbols (--). oClick the OK button. 45

43. 43 Other Important Resources 46 Investing in Innovation Fund Website: (http://www.ed.gov/programs/innovation/index.html)http://www.ed.gov/programs/innovation/index.html Notice of Final Priorities, Requirements, Definitions, and Selection Criteria Application Package (includes the Notice Inviting Applications) Frequently Asked Questions Evidence Summary Table Selection Criteria Summary Table i3 Overview (PowerPoint) i3 At-A-Glance (Quick Reference) Archived recordings of the i3 Webinars i3 Glossary Orientation Slides Panel Monitor Orientation Handouts

44. 44 Questions Please direct all questions to: Thelma Leenhouts (i3 Competition Manager) Thelma.Leenhouts@ed.gov (202) 260-0223 47

45. 45 Compensation for a job well done!! We will offer Comp time or Overtime. Each person must select only one You may receive up to 40 hours of comp time or overtime over the two week review period Comp time expires after 26 pays (one year) Overtime is time and a ½ if you are a grade 7 or 9 Grades 11 and higher will only receive you regular hourly pay http://www.opm.gov/oca/10tables/html/dcb_h.asp http://www.opm.gov/oca/10tables/html/dcb_h.asp Part time employees may receive only comp time or they may work up to 40 hours 48

46. The i3 Core Team would like to thank you for all your support in helping us to complete this i3 grant competition. THANK YOU