2. Session Outline Overview of session (5 minutes)
Receive abbreviated version of the training given to typical grant reviewers (40 minutes)
Break! (15 minutes)
Read mock evaluation criteria and applications, score, and write comments (30 minutes)
Hold mock panel reviews (30 minutes)
Report out on panel scores and experiences (15 minutes)
Discuss lessons learned and tactics for sustainability (45 minutes)

3. Overview
The goal of this session is for you to decide what tactics would be the most efficient and effective to sustain the innovative and critical services that you’ve identified.
As you listen to the training, read the mock application, write comments, and think about what evidence you would provide to justify sustaining each part of your program.

4. Mock Reviewer Training Overview
The following is a very abbreviated version of typical training that reviewers receive prior to reading, scoring, or paneling an application.
There are numerous protections in place for confidentiality, conflict of interest, and quality control, which we won’t be covering today for the sake of brevity.
Application review processes can vary dramatically between funding organizations (government, business, foundations, etc.).
The lessons learned from this process can help you articulate a clear plan to many potential stakeholders for how and why your program should be sustained.

5. Review Process
Independently read and score the mock application according to the evaluation criteria.
Write evaluative comments that support and justify your scores.
Do not discuss the application, scores, or your comments with anyone until steps 1 and 2 are complete.
Compare scores and comments in panels. Come to a consensus. Revise scores and comments as needed. Scores and comments do not need to be the same, but they cannot conflict with each other.
Comments reviewed by Panel Managers.
Prepare to report out on your experiences and comments.

6. Roles and Responsibilities
Reviewer: Reads, scores, writes comments, and participates in panels.
Chairperson: Reads, writes and compiles comments, facilitates panels, and creates summary reports. Chairperson does not score.
Panel Manager: Checks summary reports of comments, scores, and answers questions related to process or evaluation criteria. Panel Manager does not read applications, does not score, and is not part of panels.

7. How to Read an Application
Read the evaluation criteria and the mock application.
Score the application according to the evaluation criteria and nothing else.
Never contact the applicant or any other outside source. Do not discuss the application with anyone. Do not use the internet, personal bias for or against the applicant, or any external source of information.
Every score must be fully supported and justified by detailed written evaluative comments.
Evaluation criteria will be a handout.

8. What is an Evaluative Comment?
Evaluative comments assess the value, worth, or quality of the information in the application.
An evaluative comment describes the strengths and weaknesses of an application and justifies the scores given to an application.
Think about what evaluative comments would be written about your program. What rationale and evidence do you have to support your program?

9. Why are Evaluative Comments Important?
They determine scores. If a score isn’t supported by evaluative comments, it is sent back to the panel.
They assist in determining funding and explain and validate scores to decision makers.
They can be used for technical assistance to applicants, both funded and unfunded.
These comments are very important for several reasons. The comments for the unsuccessful applicants can be used for future reference if the applicant wishes to submit another application(s).

10. Evaluative Comments A comment must be either a strength or a weakness.
An individual comment can never be both.
Avoid conjunctions such as but, although, however, and yet.
Strength comments and weakness comments can never conflict with each other.
If a particular element of an application has both strengths and weaknesses, you must write a separate comment for each, and they must be carefully written so as not to conflict with each other.
When thinking of how you would write about your own program, if a reasonable person would have to parse what you’ve written into separate strength and weakness comments, you need to provide more evidence or change your approach.

11. Evaluative Comments (continued)
A high scoring application should have many strength comments and few or no weakness comments.
A perfect score should have no weakness comments. Anything less than a perfect score must have at least one weakness, and any application with a weakness comment must have points deducted.
Don't simply restate what the applicant has written; evaluate what it says.
Every comment should include a page and paragraph number citation.

12. Evaluative Comments (continued)
Cite the rationale and evidence that the applicant provides to support your evaluative comments. If these don’t exist or are unconvincing, explain why.
Assume that comments are being read by someone who has never read the application. A person reading your comments should understand the full story of what the applicant is proposing to do and why it should or should not be funded.
Use complete sentences, proper grammar, and correct spelling.

13. Evaluative Comments (continued)
Make comments tactful and constructive.
Ask yourself, “Will my comments help the applicant to prepare a better application next time?”
This is particularly important for weakness comments. Don’t just say that something isn’t there or isn’t good enough – explain why.
Think about how you would explain it for your own program.

14. Evaluative Comments (continued)
“The applicant uses appropriate outcome measures” is not an evaluative statement. You need to answer why the outcome measures are appropriate or inappropriate.
Avoid personal pronouns and comparisons.
Don’t write, “I think that this is bad. My program does it better.”
Instead write, “The application states… This is ineffective because… A superior practice would be…”

15. Sample Strength Comments
Poorly written: The applicant demonstrates an understanding of the needs of low-income adults. (Page 3, paragraph 1)
Well written: The applicant clearly demonstrates evidence and understanding of the changing needs of low-income adults and their families by referencing the increasing numbers of families struggling with educational and emotional issues and developing relevant goals, objectives, and activities. For example… (Page 3, paragraph 1)

16. Sample Strength Comments (continued)
Poorly written: The applicant’s needs assessments and focus groups demonstrate support for their proposal. (Pages 5, paragraph 2)
Well written: The applicant carried out a series of assessments and focus groups with community partners. The results demonstrate strong community support for their proposal for a career pathway program to train nurses and the need to provide those students with support services, such as child care and transportation assistance. (Pages 5, paragraph 2)

17. Sample Weakness Comments
Poorly written: The objectives and the work are too ambitious to accomplish. (Page 1, paragraphs 2-3)
Well written: The applicant proposes to train 1,000 nurses in the first twelve months of the project. Their institution currently trains 100 nurses per year, and the labor market information provided shows that there are only 500 new job openings for nurses each year. Therefore, the objectives for the project are too ambitious given the program’s capacity to serve students and the community’s capacity to employ them. (Page 1, paragraph 2-3)

18. Sample Weakness Comments (continued)
Poorly written: The applicant provides students with too many supportive services. (Page 2, paragraphs 1-2)
Well written: The applicant does not provide a rationale or evidence to justify the number or type of supportive services they propose to provide, or how they will improve the program’s completion and employment outcomes. (Page 2, paragraphs 1-2)

19. Avoid Subjective and Personally Biased Comments
“My agency would…”
“They didn’t answer the criterion completely, but I know how this program works…”
“I gave them a score of 15 out of 20. I thought it was a good application but not great.”
“I don’t like what they are proposing…”
“This is the worst application I’ve ever seen. What were they thinking?”
Avoid statements that are personally biased.

20. Next Steps Break! (15 minutes)
Read mock evaluation criteria and applications, score, and write comments (30 minutes)
Hold mock panel reviews (30 minutes)
Report out on panel scores and experiences (15 minutes)
Discuss lessons learned and tactics for sustainability (45 minutes)

21. Break (15 minutes)

22. Read Mock Applications, Score, and Write Comments (30 minutes)

23. Mock Panel Reviews (30 minutes)

24. Report Out on Panel Experiences
What was your panel’s average score? Did you have difficulty coming to consensus, and why?
What have you learned?
What implications does this have for your program and your future funding strategy and marketing?

25. Key Resources University of Kansas Toolkits:
A Financial Sustainability Resource Guide
Now we are going to shift gears, and apply what you’ve learned in your panel experiences in a discussion about specific strategies you may wish to consider.
These are two key resources that you may wish to draw upon. You should already be very familiar with the University of Kansas Toolkit. But I’d like to draw your attention to it again. It really is a wealth of different free resources and additional useful information regarding many different facets of sustainability that we’ve addressed in this roundtable.
We’re also handing out a new publication. The next part of my presentation will be walking us through parts of this resource, and at the end, we would like your feedback on if there is anything we should change or add to it before posting it to HPOG Connect.

26. Tactics for Sustainability
Develop a funding strategy.
Set realistic funding goals.
Create an approach.
Leverage your network.

27. Develop a Funding Strategy
Applying for publically available funding is extremely time-consuming and highly competitive.
Typically only very high scoring applications win an award.
It’s better to write a smaller number of very strong applications instead of a large number of poorly written ones.

28. Develop a Funding Strategy (continued)
As part of your preparation, we asked you to create a list of your program’s partners, employers, foundations, and institutional funders.
Create a financial sustainability committee composed of key partners and stakeholders.
Who would be on your sustainability committee and why?
Have your committee hold mock panel reviews for your applications before submitting them, and involve them in all aspects of sustainability planning.

29. Set Realistic Funding Goals
Come up with a realistic business plan to accomplish attainable goals.
Know the cost to providing each of your services per person so it can easily be adjusted to meet different funding opportunities.
Have evidence to support why funding each service accomplishes your goals so you can clearly explain the cost/benefit calculation for investing in your program.

30. Create an Approach
Have a short, easy-to-understand document that outlines your program to include in outreach to funders and to build upon for potential applications.
Think about your purpose, audience, message, delivery, and relevance.
Make sure your program has the capacity to deliver the services you are promising before you apply for funding.

31. Leverage Your Network
Different organizations can apply for different types of funding and support a variety of different missions, while still sharing common goals.
Reach out to your community, build relationships with individuals, and build partnerships with organizations.
Build a mutually beneficial relationship.

32. Identify Potential Funders
Government Funding
Federal Opportunities
State Opportunities
Local Opportunities
Private Businesses and Corporations
Volunteer Programs
Matching Initiatives
Impact Investments
Foundations
Fees for Goods or Services

33. Other Tactics
Avoid charts, graphs, and graphics whenever reasonably possible, unless the evaluation criteria specifically calls for it. They can be difficult for a reviewer to translate into a written comment, illegible when photocopied, and difficult to read for reviewers with poor eyesight.
Assume that the applicant cannot or will not click through hyperlinks to websites or other online materials.

34. Other Tactics (continued)
Use internal citations or footnotes. Avoid endnotes or a bibliography unless it is specifically required.
Internal citations and footnotes occur on the same page as the citation and allow the reader to easily see and substantiate the applicant’s claim.
Endnotes and bibliographies obscure citations and are often skimmed or skipped by reviewers.
Make sure there is real substance to any claim you make so that an expert reviewer can’t take issue with it.

35. Other Tactics (continued)
Applications should lay things out in the same order as the evaluation criteria. Make it as easy as possible for reviewers to follow so they can easily map different parts of your application to comments.
Use plain language so novice reviewers understand everything you’ve written.
Don’t assume your reader understands the subject matter. Explain it in clear terms.
Avoid jargon, acronyms, abbreviations, etc. (unless they are required).

36. Feedback Tomorrow we’ll be addressing Business and Marketing Plans.
Any feedback on the Resource Guide?
Do you have any insights from your mock panel experiences that can be applied to your sustainability tactics?

37. Discussion and Lessons Learned