1. In Depth Panel Review Training

2. Activity: Mock Panel Review To evaluate the Need for Assistance, reviewers will consider the extent to which the application includes: 1.A concise problem statement that identifies the current condition(s) to be addressed by the project 2.Supporting information or data detailing the scope and nature of the problem 3.Current challenges standing in the way of addressing the problem 4.A clear description of the community to be served and who the intended beneficiaries are What score, out of 10 possible points, did you give the application narrative? Why?

3. Panel Review Roles Reviewer: A member of the panel charged with evaluating the application and documenting their evaluation by scoring and commenting on the application strengths and weaknesses Facilitator: A member of the panel charged with managing the panel discussions and writing the PSR Subject Area Manager (SAM): An ANA representative charged with assisting the panel on the process and the editing of the PSR Program Area Manager (PAM): An ANA representative with the responsibility to approve the PSR Review Director: oversee the entire panel review process Logistics Contractor: handles payment and ARM support

4. Panel Review Director PAMSAMFacilitator 13 ReviewersFacilitator 23 ReviewersSAMFacilitator 13 ReviewersFacilitator 23 ReviewersPAMSAMFacilitator 13 ReviewersFacilitator 23 ReviewersSAMFacilitator 13 ReviewersFacilitator 23 Reviewers Panel Review Structure

5. Approval Process of PSRs

6. Calendar of Events Pre-Panel (April 1 st – April 27 th ) Reviewers and facilitators are identified and trained Session 1: P&M, EMI, and ERE (April 25 th – May 9 th ) Reviewers and facilitators evaluate and panel applications Session 2: SEDS and SEEDS (May 9 th – May 23 rd ) Reviewers and facilitators evaluate and panel applications Post Panel Payment is sent to reviewers and facilitators

7. The Application Review Module (ARM) Do Enter the Score First Use the ARM text box to write your comments Spellcheck Save every 20-30 minutes Do Not ×Use Google Chrome ×Click the ARM support link on the ARM login page Email ANAReviewer@acf.hhs.gov with all ARM questions! ANAReviewer@acf.hhs.gov

8. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by facilitator & submitted to SAM/PAM Three Principles for Reviewing Applications Evaluate the extent to which the application responds to the FOA Evaluation Criteria The FOA Evaluation Criteria is the only measure used to evaluate the application Do not compare applications against each other

9. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Narrative DescriptorScore Excellent93-100 Very Good86-92 Good78-85 Fair70-77 Needs Significant Improvement0-69

10. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM How Do You Determine the Significance of a Weakness and a Strength?

11. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach) The applicant did not describe anywhere in the application how community input was used in the development of the project, making it unclear if the project proposal is truly community-driven. (page 25, Approach) The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be evaluated, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected) Examples of Good Weakness Comments

12. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM What is a panel discussion like? Not everyone agrees at the beginning Initial scores vary. Sometimes by 20 points or more Structured and focused We spent about an hour talking about each application Hearing the other reviewers point of view made me reconsider my score

13. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM The FACILITATOR sets group norms, focuses the meeting, asks probing questions, shares information, and documents the discussion Panel Group Norms Attend all meetings Keep Focused Listen and show respect Focus the meeting by creating a meeting plan Probing Questions What criteria did you use? How did you determine this score? What documentation did the application provide that substantiates the approach? Share initial reviewer comments Document the discussion to help write the Panel Summary Report

14. Requirements of a Panel Summary Report (PSR) 1.Factually Accurate 2.Adheres to the FOA Evaluation Criteria 3.Consistent 4.All comments justify the score 5.Polished and Well-Written Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM

15. Example of an Inconsistency in a PSR Strength Comment The applicant clearly showed how the current staff will manage and complete the project on time. (page 75, Objective Work Plan) Weakness Comment The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach) Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM

16. Writing a PSR that Represents the View of the Panel Create an initial PSR Take notes during the panel discussion Draft the final PSR Submit the final PSR to the PAM/SAM for comments and approval Share the final PSR with the reviewers Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM

17. Tips on Writing a PSR that will be Approved Reference Application Page Numbers Use the wording of the relevant FOA evaluation criteria Make it grammatically correct Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM

18. Example of a Good PSR Comment The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be measured, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected) FOA Text in Outcomes Expected Evaluation Criteria To evaluate the projects intended impact, reviewers will consider the extent to which: 1.The condition(s) identified in the problem statement will be addressed 2.The lives of community members and beneficiaries will improve To evaluate the impact indicator(s), reviewers will consider the extent to which: How impact will be measured with at least one impact indicator using the same measure at three points in time, baseline (beginning of project), end of project, and three years post- project. Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM

19. PSR Approval Reviewers read application & evaluate using FOA criteria Reviewers write comments & determine preliminary score Panel members discuss applications Reviewers revise comments & scores; PSR compiled by Facilitator & submitted to SAM/PAM Yes No Is the PSR factually accurate? Does the PSR adhere to the FOA Evaluation Criteria? Is the PSR Consistent? Do the comments justify the score? Is the PSR polished and well-written?

20. Roles and Responsibilities Recap Reviewer 1.Evaluate the Application 2.Document evaluation by scoring and justifying score with comments 3.Submit score and comments to facilitator through ARM Facilitator 4.Create initial PSR by compiling all reviewer comments 5.Holds panel meeting with reviewers 6.Revises PSR based on reviewer comments, discussion or SAM/PAM recommendations 7.Submit PSR to SAM through ARM SAM/PAM Responsibility 8. Determine whether the PSR can be approved. If not, back to step 6. Facilitator Responsibility 9. Share final PSR with reviewers to ensure it reflects panel opinion. SAM Responsibility 10. Submit PSR to PAM for approval through ARM PAM Responsibility 11. Approve the PSR