1. Program Review Guide-Through An interactive, step by step guide to preparing for RSA’s program review Rik Opstelten- NATTAP Begin This is an inter-active self-evaluation tool, created using PowerPoint. To use it as it is intended, please start a Slide Show, either from the “View” menu, or by pressing F5.

2. About the Guide The purpose of this guide is to allow program mangers to work their way through process of preparing for RSA’s Program Review. You will be guided through the process of: Understanding the basis for the program review Preparing program staff and procedures to participate in the review process Gathering and checking documents to be reviewed by RSA Preparing for the Webinar-based virtual site visit component of the review Understanding the questions asked by RSA and the grading criterion

3. Navigating Through the Guide In the next page, you will be shown a Table of Contents. You can use this to skip to a particular part of the guide, or you can use the forward and backward buttons to move through the process in sequence. Other navigation buttons you will see: Home: Returns you to the Table of Contents Info: Brings you to further detail on a particular point

4. Table of Contents Requirement for Program Review Role of Lead Agencies and CBOs How are Program Selected for Review? What Happens When a Program is Selected? How Does a Grantee Prepare for the Review? Program Management Fiscal Management, Contract Oversight,Personnel Management Fiscal ManagementContract Oversight Personnel Management Program Performance Consumer Choice and Control, Timely and Equitable Access to Financial Assistance, Capacity to Continue on a Permanent Basis Consumer Choice and ControlTimely and Equitable Access to Financial AssistanceCapacity to Continue on a Permanent Basis The Virtual On-Site Visit. Preparing the Webinar Appendix 1- RSA’s rubric on Program Management Appendix 2- RSA’s rubric on Program Performance

5. ED Monitoring Policy It is ED’s policy to monitor active discretionary grants with a focus on technical assistance, continuous improvement, and attaining promised results. Monitoring a grantee shall continue for as long as ED retains a residual financial interest in the project, whether or not ED is providing active grant support.

6. Requirements for Program Review Section 5.3.2 Monitoring Policy ED staff is to monitor each grantee to the extent appropriate so as to achieve expected results under approved performance measures, while assuring compliance with grant requirements. Existing requirements in EDGAR extend equally to all grantees and their partners.

7. Requirements for Program Review Section 5.3.3 Monitoring Purpose Systematic and regular monitoring by ED staff of a grantee’s activities measures the project quality and progress, including strengths and weaknesses.

8. Requirements for Program Review 1. A grantee’s project must: a) Conform to the grantee’s approved application and any approved revisions, as well as the effectiveness and quality of the project (Program Management); b) Progress against previously established performance measures (Performance Measurement); c) Adhere to laws, regulations, conditions of the grant, certifications and assurances (Compliance); and d) Manage Federal funds according to Federal cash management requirements, including expenditures of funds for authorized purposes (Fiscal Accountability).

9. Requirements for Program Review Regular monitoring enables ED staff to provide customized technical assistance, appropriate feedback, and follow-up to help grantees to: Improve areas of need Identify project strengths Recognize significant achievements

10. Role of Lead Agencies and Community Based Organizations in Program Reviews The LEAD AGENCY is being reviewed. Participation from BOTH the Lead Agency and CBO is expected. Participation of all personnel being paid using grant funds is required (see exception in the next bullet). Participation of the Certifying Representative is encouraged but not required.

11. How are AFP/Telework Programs selected for review? Random selection of an equal number of grantees in two categories: (1) Title III AFP/Telework grantees claimed by a Statewide AT Program as a state financing activity; OR (2) Title III AFP/Telework grantees not claimed by a Statewide AT Program as a state financing activity.

12. How are AFP/Telework Programs selected for review? Exceptions to random selection: Recent major change in the AFP/Telework Program or State Plan for AT Recent major disaster RSA is concerned

13. How are AFP/Telework Programs selected for review? 6 grantees per cycle (3 x 3) A “cycle” is a fiscal year (Oct. 1 to Sept. 30) 9 weeks per grantee (some program reviews may overlap) First year of program reviews: Jan. 1 to Sept. 30 Only 3-4 reviews

14. What happens when an AFP/Telework Program is selected? RSA notifies grantee in spring prior to next cycle. RSA and grantee negotiate 9-week window within next cycle. RSA recruits third-party (peer) reviewers. Grantee begins preparing for review.

15. How does a grantee prepare for the review? A review assesses two “Core Components” of grant implementation, each in a slightly different way. Core Component 1 – Program Management Core Component 2 – Program Performance

16. Core Component Selection Now that you have learned about the reasoning behind the Program Review, and how the process is structured, we will begin going through each of the “Core Components” of the Program Review. Each Core Component has several sub-points that comprises it.

17. Choose a Core Component Program Management Fiscal Management Contract Oversight Personnel Management Program Performance Emphasis and Expansion of Consumer Choice and Control Emphasis and Expansion of Consumer Choice and Control Timely and Equitable Access to Financial Assistance Timely and Equitable Access to Financial Assistance Capacity to Continue on a Permanent Basis

18. Program Management Fiscal Management Contract Oversight Personnel Management

19. Fiscal Management Documents Required Expenditure Report Proof of Title III or NFP fiscal management assurances Indirect Cost Agreement Copy of most recent audit report or Questionnaire

20. RSA Detail: Fiscal Management EDGAR requirements: Fiscal control and accounting procedures to insure proper disbursement of and accounting for Federal funds; controls and procedures adequate to establish that such funds have not been used in violation of applicable statutes. Records that fully show the amount of funds under the grant, how the grantee uses the funds, the total cost of the project, the share of the cost provided by other sources, and other records to facilitate an effective audit. Audits in accordance with OMB A-133. Lead Agencies must be able to show how they ensure appropriateness of indirect costs used by Implementing Agencies or CBOs.

21. RSA Detail: Fiscal Management Title III or NFP requirements: Assurance that the grantee will provide the non- federal share of the cost of the program in cash, from state, local, or private sources. Assurance that the grant funds will be used to supplement and not supplant other federal, state, and local public funds. Assurance that the grant funds will be placed in a permanent separate account and identified and accounted for separately from any other fund. Assurance that the funds are administered, expended, and invested in accordance with title III or the NFP.

22. Help: Choosing a Core Component To proceed through the guide, you must choose the Core Component of the Program Review you would like to learn about. Remember, you can always use the Table of Contents page to navigate in greater detail

23. Contract Oversight Documents Required Dates copy of the Grantee’s most recent contract with the CBO.

24. Contract Oversight Detail EDGAR requires grantees to: Directly administer or supervise the administration of each project. Have procedures for providing TA, evaluation and for performing other administrative responsibilities necessary to ensure compliance. Monitor activities to assure compliance with applicable Federal requirements and that performance goals are being achieved.

25. Contract Oversight Detail  Grantees are required to enter into a contract with a CBO or group of CBOs that has individuals with disabilities involved in the organizational decision-making to administer the AFP/Telework Program. The contract must: Include a provision requiring funds be administered consistent with title III or the NFP. Include any provision the Secretary requires concerning oversight and evaluation of federal financial interests. Require the CBO(s) enter into a contract with commercial lending institutions or organizations or state financing agencies.

26. Contract Oversight Detail What RSA is looking for: Appropriate oversight to protect Federal interests and ensure implementation Relevant to title III or the NFP

27. Personnel Management Documents Required: (A) Position descriptions of record for those being paid with grant funds and personnel activity reports for them; AND (B) Synopsis for key personnel describing his or her role and responsibilities associated with the grant.

28. Personnel Management Detail What RSA is looking for: Appropriate number of personnel Appropriate time commitments of personnel Appropriate duties for personnel

29. Personnel Management Detail OMB Circulars: Compensation to personnel must be reasonable. Charges to Federal awards for salaries and wages are based on documented payrolls. Charges for salaries and wages of employees who work solely on a single Federal award are supported by periodic certifications. Charges for salaries and wages of employees who work on multiple activities are supported by personnel activity reports.

30. Program Performance Consumer Choice and Control Timely and Equitable Access to Financial Assistance Timely and Equitable Access to Financial Assistance Capacity to Continue on a Permanent Basis

31. Consumer Choice and Control Documents Required: A narrative describing how individuals with disabilities are involved in organizational decision- making at all organizational levels to administer the loan program.

32. Consumer Choice and Control Detail Title III or NFP requirements: Assurance that, and information describing the manner in which, the grantee will expand and emphasize consumer choice and control.

33. Consumer Choice and Control Detail What RSA is looking for: Appropriate emphasis and expansion of involvement in organizational decision-making by individuals with disabilities The grantee should also be prepared to discuss consumer choice and control during webinars/teleconferences

34. Timely and Equitable Access to Financial Assistance Documents Required A dated copy of its most recent revised and adopted AFP/Telework policies and procedures manual. A dated copy of its most recent revised and adopted lender agreement(s).

35. Timely and Equitable Access Detail Title III or NFP requirements: Assurance that the grantee enter into a contract with a CBO that has policies and procedures to: Review and process requests for financial assistance in a timely manner. Ensure equitable access to all consumers for which financing is requested through the program. Ensure consumer-controlled oversight of the program.

36. Timely and Equitable Access Detail What RSA is looking for: Appropriate access to financial assistance. The grantee should also be prepared to discuss how the program assures the review and process of financial loans in a timely manner and describe the flow of an application for an AFP/Telework loan during teleconferences or webinars.

37. Capacity to Continue on a Permanent Basis Documents Required A dated copy of investment plan, if applicable, or narrative describing how loan program funds are invested. A copy of investment portfolio. A dated copy of the plan for sustainability, if one has been developed, or narrative explaining how the program will continue on a permanent basis.

38. Capacity to Continue Detail Title III or NFP requirements: Assurance that the AFP/Telework Program will: Continue on a permanent basis. Invest the funds within the permanent separate account in low-risk securities. Administer the funds with prudence, discretion, and intelligence.

39. Capacity to Continue Detail- Investment Plan What RSA is looking for: Appropriate investment of AFP/Telework Program funds in the permanent separate account. The grantee should also be prepared to discuss the implementation and regular revision of the AFP/Telework Program’s investment plan.

40. Capacity to Continue Detail- Sustainability Plan What RSA is looking for: Appropriate implementation of strategies for sustaining the AFP/Telework Program. The grantee should also be prepared to discuss how the program is implementing a plan for sustainability during a webinar/teleconference.

41. Virtual On-Site Visit/Webinar Develop teleconference/webinar presentations that serve as a virtual on-site visit to your program. Length, format, and content of presentation are up to the program. The “presenter” during a teleconference or webinar is up to the program. Need to be accessible.

42. RSA’s rubric on Program Management The following slides provide detailed information about the way in which RSA evaluates program management. Outlined are: How to prepare for evaluation Questions used for evaluation The rating scale of evaluation How ratings are assigned

43. How to prepare for a review of Program Management Submit all the previously described documents. RSA reviews the documents. RSA discusses the documents via conference call (the grantee is responsible for bringing together the appropriate parties). RSA uses this information to answer questions about program management. Yes/no with explanation.

44. Questions about Program Management 1. Has the grantee obtained its full matching funds from appropriate sources? If not, does the grantee have a plan for obtaining the remaining match funds? 2. Does the grantee account for grant funds accurately and completely in accordance with EDGAR? 3. Does the grantee use grant funds only for purposes under, and within the limitations of, title III or the NFP, as applicable? 4. Does the grantee comply with indirect cost requirements? 5. Does the grantee comply with audit requirements?

45. Questions about Program Management 6. Does the grantee exercise appropriate fiscal and performance oversight of the AFP/Telework Program through its contract with the CBO? If not, does the Lead Agency have a plan for improving its oversight? 7. Is the contract structured to meet the requirements and intent of title III of the NFP, as applicable? If note does the Lead Agency have a plan for changing its contract to make it more consistent with title III or the NFP, as applicable?

46. Questions about Program Management 8. Does the AFP/Telework Program have an appropriate number of staff to implement the grant? If not, does the Lead Agency have a plan for changing its contract to make it more consistent with title III or the NFP, as applicable? 9. Do the Lead Agency and CBO distribute the time of their personnel appropriately to implement the grant based on their responsibilities? If not, does the grantee have a plan to distribute time more appropriately? 10. Do the Lead Agency and CBO use grant funds to support only personnel with responsibilities germane to the grant?

47. Assessing Compliance of Program Management There is no overall rating of compliance. For each element of Program Management, a grantee will receive 1 of 3 ratings: Compliant In Need of Improvement Fails to Substantially Comply

48. Assessing Compliance of Program Management The rating is based on how RSA answers the questions: Initial question: Does the AFP/Telework Program have an appropriate number of staff to implement the grant? Subquestion: If not, does the grantee have a plan for adjusting the number of staff?

49. Assessing Compliance of Program Management If the answer to the initial question is “Yes” – the grantee is rated as compliant. If the answer to the initial question is “No” – the rating depends on the answer to the subquestion. If the answer to the subquestion is “Yes” – the grantee is rated as in need of improvement. If the answer to the subquestion is “No” – the grantee is rated as failing to substantially comply.

50. Assessing Compliance of Program Management Questions for which “ needs improvement ” doesn ’ t apply: Does the grantee account for grant funds accurately and completely in accordance with EDGAR? Does the grantee use grant funds only for purposes under, and within the limitations of, title III or the NFP, as applicable? Does the grantee comply with indirect cost requirements? Does the grantee comply with audit requirements? Does the grantee use grant funds to support only personnel with responsibilities germane to the grant?

51. Assessing Compliance of Program Management The onus is on the grantee to present evidence/plan sufficient to get to a “yes” answer. RSA will ask questions or provide examples of sufficient evidence, if needed. Grantee likely will know their status before the conversation is over.

52. RSA’s Rubric on Program Performance The following slides provide detailed information about the way in which RSA evaluates program performance. Outlined are: How to prepare Questions asked Grading Scale Grading Rubric Activities of Program Reviewers The Role of Data

53. How to prepare for a review of Program Performance Based on documents, data, and discussion, third-party reviewers respond to a series of questions: Yes/no with explanation. Evidence can come from any source of the review. Getting a “yes” answer does not depend on an AFP/Telework Program having any particular structure or type. A “no” answer does not automatically mean a grantee is out of compliance.

54. Questions about Program Performance 1. Is there evidence that individuals with disabilities are involved in the administrative decision-making of the AFP/Telework Program? a) If not, does the grantee provide justification for the current level of involvement of individuals with disabilities in the administrative decision-making of the loan program? b) If not, does the grantee have a reasonable plan for improving the involvement of individuals with disabilities in the administrative decision-making of the loan program?

55. Questions about Program Performance 2. Is there evidence that the AFP/Telework Program emphasizes and expands consumer choice and control? a) If not, does the grantee provide justification for why the current level of consumer choice and control is appropriate? b) If not, does the grantee have a reasonable plan for improving its emphasis and expansion of consumer choice and control?

56. Questions about Program Performance 3. Is there evidence that the policies and procedures of the AFP/Telework Program make it accessible to consumers regardless of type of disability, age, income level, type of AT device, equipment, or service for which financing is requested through the program? a) If not, does the grantee provide justification for why the program is not accessible to all individuals with disabilities in the State? b) If not, does the grantee have a reasonable plan for improving the accessibility of the program?

57. Questions about Program Performance 4. Is there evidence that the structure and practices of the AFP/Telework Program make it possible to meet the needs of individuals in most areas of the State? a) If not, does the grantee provide justification for why the areas reached in the State are sufficient? b) If not, does the grantee have a reasonable plan for improving their ability to reach consumers in other areas of the State?

58. Questions about Program Performance 5. Is the lender agreement structured to promote the goals of the AFP/Telework Program? a) If not, does the grantee provide justification for the structure of the current lender agreement? b) If not, does the grantee have a reasonable plan for improving the lender agreement?

59. Questions about Program Performance 6. Does the AFP/Telework Program administer its funds with prudence, discretion, and intelligence? a) If not, does the grantee provide justification for why the program administers its funds in the current manner? b) If not, does the grantee have a reasonable plan for improving the administration of program funds?

60. Questions about Program Performance 7. Does the AFP/Telework Program have an investment plan that is implemented and revised regularly? a) If not, does the grantee provide justification for why the program has not implemented an investment plan that is reviewed regularly? b) If not, does the grantee have a reasonable plan for the implementation and regular revision of an investment plan?

61. Questions about Program Performance 8. Is the AFP/Telework Program implementing appropriate strategies for sustainability? a) If not, does the grantee provide justification for why the program is not implementing appropriate strategies for sustainability? b) If not, does the grantee have a plan for sustainability that could be implemented?

62. Assessing Compliance of Program Performance There is no overall rating of compliance. For each element of Program Performance, a grantee will receive 1 of 3 ratings: Compliant In Need of Improvement Fails to Substantially Comply

63. Assessing Compliance of Program Performance The rating is based on how peer reviewers answer the questions: Initial question: Is there evidence that the structure and practices of the AFP/Telework Program makes it possible to meet the needs of individuals with disabilities in most areas of the State? Subquestion 1: If not, does the grantee provide justification for why the areas reached by the program in the State currently are sufficient? Subquestion 2: If not, does the grantee have a reasonable plan for improving the capacity of the program to reach individuals in more areas of the State?

64. Assessing Compliance of Program Performance If the answer to the initial question is “Yes” – the grantee is rated as compliant. If the answer to the initial question is “No” – the rating depends on the answer to the subquestions. If the answer to subquestion 1 is “Yes” – the grantee is rated as compliant. If the answer to subquestion 1 is “No” – the the rating depends on the answer to subquestion 2. If the answer to subquestion 2 is “Yes” – the grantee is rated as in need of improvement. If the answer to subquestion 2 is “No” – the grantee is rated as failing to substantially comply.

65. Assessing Compliance of Program Performance Activities of peer reviewers: Read documents. Participate in teleconferences/webinars. Complete review forms (answer questions). Meet as a group with RSA to discuss results. Finalize review forms and submit to RSA.

66. Program Data AFP/Telework Program data will not determine compliance. Data will be discussed during a teleconference or webinar and reflected in the report: (1) Grantees strengths and areas in need of improvement when it comes to data; and (2) How the grantee is learning from and using its data.

67. Program Data Data collection infrastructure: How the data is collected. How the data is checked. Strengths and areas in need of improvement.

68. Program Data Understanding your data: How the nature of your state and structure of your program influence the data. What your data tells you about the program and how it influences implementation. Strengths and areas in need of improvement.

69. Conclusion We hope this guide has been useful in understanding the steps that programs can take to prepare for the RSA’s Program Review Process. If you have any further questions, please feel free to contact NATTAP.