1. Mathematics and Science Partnership (MSP) Programs U.S. Department of Education: Administration Meeting

2. The Annual Progress Report (APR) SEC. 2202.(f): Each eligible partnership receiving a grant or sub-grant shall report to the Secretary… Partnerships/projects must send their APRs to their State Educational Agency (SEA), the SEA must review and approve the APR, and submit it to the U.S. Department of Education (USED) within 60 days from the end of the performance period.

3. APR (continued…) Typical Example: –Project Achieve is awarded a 3 year MSP sub- grant that has a start date of 3/1/2009 and an end date of 2/28/2012 (performance cycle); with “performance period 1” starting on 3/1/2009 to 2/28/2010. –The project has 12 months to complete their activities for that first year (as indicated in their application to the state). –During the first 12 months the project would have completed their first years activities, collected student data, collected teacher data and would be ready to start the report by 2/28/2009.

4. APR (continued…) –The project finishes the APR and sends it to their SEA (at a time determined by the SEA; before 60 days). –The SEA will review the APR and determine if The APR is complete and accurate; or The APR needs revision. –If the APR is complete and accurate the SEA will submit the report to USED by 4/30/2009 (60 days after the performance period). –If the SEA has determined that the APR needs revisions, the SEA will comment on the report, the project will make the needed changes, resubmit to the SEA, the SEA will review again, and submit to USED by 4/30/2009 (60 days after the performance period).

5. APR (continued…) –This process will continue until… The project has completed a cycle (in this case 3 years) and submits their final APR (projects are encouraged to also submit their Final Evaluation Report that summarizes all 3 years of activities); or if The project is discontinued (a decision made by the SEA). –At the end of the performance cycle projects must re-compete to receive an additional cycle of funding. (Note: not all states have 3 year cycles)

6. Overview of the APR Submission Process (Norm) SEA creates an APR for the project to complete The project completes the APR and submits it to the SEA by the indicated date. The SEA reviews the APR and determines if the APR is acceptable OR if revisions are needed. SEA sends the APR to the USED. The USED reports MSP results to Congress.

7. OR…Partial Submission (extraordinary) SEA creates an APR for the project to complete The project completes the APR and performs a partial submission and informs the SEA of when the APR will be completed… by the indicated date. (Only with SEA approval). The SEA reviews the partially submitted APR and determines if the APR is acceptable OR if revisions are needed & accepts or negotiates the final submission date with USED input. SEA sends the APR to the USED. The USED reports MSP results to Congress.

8. Ed-MSP.NET

9. APR Due Dates

10. Accessing your APR Post-award, your SEA coordinator should create a new APR for newly funded projects and continuing projects. You should receive an email from Ed- MSP.net (check your spam box). If you have forgotten your login or password please access the forgotten password option on the login page located on www.apr.ed-msp.net.

11. Logging In After accessing this link, it will ask you for your email (if you are not sure which email is listed contact your SEA). The system will send you an email to re-establish your new login and password. If that does not work contact msp@wasoftware.com or your states contact at USED msp@wasoftware.com

12. Levels of Access There are three levels of access for projects 1.Project Directors can read, write, and submit reports to their state coordinators. (2 Max) 2.Other Contacts can read and edit reports. (5 Max) 3.Evaluators have read-only access. (5 Max) * It is the projects responsibility to tell the SEA all individuals that require access and their levels.

13. Completing the APR Most of your answers can be found in the User's Guide…please refer to this document.

14. User’s Guide A copy of the User’s Guide can be found as a part of your handouts!

15. User’s Guide Overview Step-by-step instruction on how to complete the report. Includes definitions that describe what each section should include. Includes FAQ’s Includes technical information on the Mathematics and Science Partnerships Teacher Content Knowledge (MSPTCK) User’s Guide for the MSPTCK

16. Completing the APR (continued…) Lets the project see current APR’s they must… -complete, or -older APR’s they have completed. Browse Report…allows the project to enter data into the APR Edit…allows the PD (only) to submit the report to the SEA coordinator.

17. Completing the APR (New Changes)! In the Near Future, the Browse Report Link will be changed to “Edit”. The Edit link will be changed to “Submit”.

18. Completing the APR (continued…) The Status column indicates where the project is on the APR submission process. -Incomplete…not submitted to SEA -Approval Pending…sent to SEA; SEA has yet to review. -Revisions Needed…reviewed by the SEA; project must re-submit to the SEA after changes have been made. -Partial Approval…project is missing core data (teacher, student, and/or evaluation); project must re-submit when data is complete by the approved date. -Approval Granted…SEA has approved the APR and submitted to USED.

19. APR Table of Contents

20. APR Section I- MSP Project Information This section provides a summary of basic project information: a)Project: Name, funding amount, etc. b)Lead Organization: the fiscal agent. c)Partner Organizations: includes the fiscal agent, the High-Need Local Education Agency (LEA), the Institute of Higher Education (IHE), partner school (public and private, other LEAs (charters, etc.), business, non-profits, or other.

21. Section II- Abstract This section asks the project to briefly describe their project. The abstract should include a description of: -the participants (teachers, student, etc); -the professional development models and/or interventions; -any other information the project would like to share. The abstract will be viewed as a public document on the ed-msp web-site and must be cut & pasted into this section. It may not be saved as an attachment. * The abstract will be viewed as a public document on the ed-msp web-site and must be cut & pasted into this section. It may not be saved as an attachment.

22. Section II- Abstract Do not attach the abstract…please cut and paste or type into the text box below.

23. Section III- Responsibilities This section is designed to identify the various partners roles and responsibilities. a)Administer the Overall Partnership: (logistical management, budgeting, recruitment, etc). b)Designing Professional Development: (Creating PD curriculum, designing learning tools, assessments, etc). c)Deliver Professional Development: (Leading the PD sessions, technical assistance, etc). d)Evaluate MSP: (Collecting and/or analyzing data, designing the evaluation plan, etc).

24. Section III- Responsibilities Projects should make their best estimate in this matter; however it should be noted that the percentages should add up to 100%

25. Section III- Responsibilities (continued…) e)Number of Higher-Ed faculty. f)Indicate the Primary Target for intervention 1.Individual Teacher: (individual teacher model) intervention targets the individual teacher. 2.Schools: (teacher leader model) intervention targets teacher leaders (coaches); teacher leader impacts their schools’ student achievement level. 3.District: (teacher leader model) intervention targets teacher leaders (coaches); teacher leaders impacts all student achievement within a district.

26. Section III- Responsibilities

27. Section IV- PD Participants This section identifies the number of participants in an MSP project (especially the number of teachers served). a)Elementary School Teachers b)Middle School Teachers c)High School Teachers d)Administrators e)Participant Students *Be sure to count each teacher only once.

28. Section IV- PD Participants If a teacher teaches multiple groups of student (e.g. general education and AP) count the teacher only once – under the teachers’ majority/dominant role.

29. Section V- PD Models This section asks the project to identify the number of contact hours, the structural model, and a detailed account of activities in the professional development models. a)Contact Hours: the average number of PD hours provided to each teacher (e.g. summer institute hours, on-line follow-up, etc…not independent study, etc.) b)Type of PD Activities: this section asks the project to identify the structural model: 1.Summer institute: PD that is conducted for at least two weeks (80 hrs or what a normal two week time period). 2.Summer institute with follow-up: PD that is conducted for at least two weeks with follow-up in the school year. 3.Other: Any structure that is not listed above. c)Description of Professional Development Model -Provide a description about the person that provided the PD and a more detailed account of the activities.

30. Section V- PD Models Choosing a PD type will open a specific series of windows that allow the project to provide further detail.

31. Section VI- Professional Development Content This section identifies the Professional Development Content. Mathematics Content Science Content Both

32. Section VI- Professional Development Content

33. Section VII- Program Evaluation A)Evaluation Design: A brief description of the evaluation design. Did your project administer pre/post test? When did the tests take place (immediately before the PD, immediately after, 3 weeks after, etc) & how often? Did you access student content knowledge? Did you use comparison groups? Where they match comparisons, equivalent at baseline, etc (please describe). Did the project use other measures of success, if so please describe? Include any relevant information about sampling, data collection procedures, and analysis.

34. Section VII- Program Evaluation B) Design Type: Who evaluated your project? C) Evaluation Method: Random Assignment Matched Comparison Non-Matched Comparison

35. Section VII- Program Evaluation D) Assessment Measures: each eachProvide information on eachassessment used to evaluate teachers & information on each assessment used to evaluate students. Indicate whether each assessment has been tested for validity and reliability. * If it was developed for the MSP project; please indicate the sources for each of the questions and the number of questions for each source.

36. Section VII- Program Evaluation If the project uses multiple assessments (as many projects do)…the project must access this page and option multiple times to report on each assessment or evaluation tool.

37. Section VII- Program Evaluation E) Classroom Assessment: How did your project measure success in the classroom. Identify the protocol or instrument used to measure success in the classroom. F) Phase of Implementation Identify the projects level of implementation (this helps USED place context to the available data).

38. Section VII- Program Evaluation Choose the Stage of Implementation

39. Section VII- Program Evaluation G) Impact on Teachers H) Impact on Students I) Other Impacts J) Impact on Partners *In these narrative sections, please describe anecdotes or describe evidence of the programs effectiveness.

40. Section VII- Program Evaluation K) Upload Report: Evaluation report In the Evaluation Report, we are looking for specific information on how the evaluation was designed and implemented. How did the project determine if teachers experienced statistically significant gains in content knowledge? Were there any threats of contamination? If so, how were they handled? Provide a time-line and description of each of the evaluation activities. The Final reports should address the Data Quality Initiative (DQI) rubric criteria (if projects are reporting experimental or quasi-experimental designs.

41. Section VII- Program Evaluation If projects do not have the evaluation report ready by the date of submission; projects may (with consultation of the SEA and USED) partially submit the APR until the evaluation report is completed.

42. Section VII- Program Evaluation Please note…if this is your projects final year (and will have to re-compete to acquire additional funds); please check this box & attach an evaluation that evaluated the project across the projects performance cycle. This should include (among other things) the aggregate student and teacher information and evaluation across the life of the project.

43. Section VIII- Government Performance & Results Act A)Teachers: Use the MSPTCK spreadsheet to indicate the number of teachers have made statistically significant gains. *Remember to save the spreadsheet to your personnel computer as it will not save on the site. B)Students: Include student data using only the most important assessment for mathematics or science. *For Both Sections: -Do not: combine information pertaining to different assessments for the same person; only use the most important assessment. -If the results are unavailable projects may (with consultation of SEA and the USED) perform a partial submission.

44. Section VIII- Government Performance & Results Act

45. Click the spreadsheet… Note: if the project has administers more than one content knowledge assessment to different groups of teachers, run the MSPTCK for each group and aggregate the results. If you have difficulty please access the users guide.

46. Section VIII- Government Performance & Results Act Select Enable Macros

47. Section VIII- Government Performance & Results Act

48. If you need detailed instructions please access this link for step by step directions.

49. Section VIII- Government Performance & Results Act States may have changed their assessment structure; but the concept of application remains the same

50. Section IX- Lessons Learned In this section we are asking the project to provide an overview of the challenges and successes encountered in the project over the past 12 months. A) MSP Implementation B) MSP Evaluation

51. Section X- State Review This section is to be filled out by the state coordinator. A)Awards B)Funding

52. Section X- Supplementary Docs.

53. Questions? Patricia Johnson at Patricia.Johnson@ed.gov Patricia.Johnson@ed.gov Miriam Lund at Miriam.Lund@ed.govMiriam.Lund@ed.gov Michelle Meier at Michelle.Meier@ed.gov Michelle.Meier@ed.gov Jimmy Yun at Jimmy.Yun@ed.govJimmy.Yun@ed.gov