Performance Stories Evaluation - A Monitoring Method to Enhance Evaluation Influence Riad Naji, Catriona King, Richard Habgood

Outline Background – Better Service to Farmers Strategy (BSTF) Performance Stories Evaluation (PSE) PSE Methodology Limitations of PSE PSE Findings Opportunities for improvement –Project level Opportunities for improvement –PSE methodology & implementation PSE as an influential tool

Background Better Services to Farmers strategy (BSTF) The Success Statement for Better Services to Farmers is: “Farmers and rural communities will see public and private service providers working cooperatively to deliver better targeted, more accessible and relevant products and services. This will enable farm businesses to improve decision making, increase their adaptability and enhance their ability to capture opportunities and manage risks”. BSTF Principles:  Target services to achieve greatest benefit for Victoria  Focus on public benefit, with industry funding supporting industry benefits  Not compete with private providers or community groups  Grow the capability of DPI staff and the Service Provider sector as a whole  Manage risks to ensure the strategy is successful.

BSTF Evaluation & Monitoring Project (BSTF E&M) The purpose of the BSTF evaluation is to: Determine whether or not BSTF is effective and in what situation Assess progress being made against the ‘success’ statement Determine whether the new service delivery model is an improvement on the previous approach (efficiency) Identify and understand what internal DPI mechanisms are enhancing or constraining the new strategy and therefore suggest actions for improvement

Performance Stories Evaluation (PSE) Overview What is it?  A key component of the BSTF Evaluation For what purpose? To improve project effectiveness and efficiency through: To improve project effectiveness and efficiency through: 1.providing timely feedback. 2.assessing the role of the BSTF implementation in influencing project change 3.identifying opportunities to improve the activities and performance of the BSTF implementation How?

PSE Methodology A questionnaire was developed based on the key evaluation questions in the BSTF E&M Plan. For the quantitative assessment, all respondents were asked to rate fifteen statements using a scale from 1 to 6, with an option for ‘not relevant’. The first round of interviews was conducted in May 2010 with the project leader and two nominated senior project staff from each of the 8 case study projects. The qualitative data was analysed using NVivo software. It is planned to periodically revisit each project over the life of the BSTF evaluation The case study projects will be monitored to provide regular reports to primary DPI evaluation audiences. Interviewing & reporting follow specific procedures & principles to secure quality & reliable data Designated case study project leaders have been provided with feedback and recommendations.

Limitations of PSE There are limitations in the use of longitudinal case studies: Positive selection bias & generalising Providing feedback overtime may well influence projects under analysis Difficult to attributing change to BSTF for a number of projects

Findings A. Precursors (or motivations) for change Almost all staff (95%) interviewed believed they understood the BSTF principles & 82% stated that BSTF is a relevant approach Main driver of accepting BSTF was the perceived ‘ its relevance to me and my project'. Respondents generally focused on the short term outcomes rather than the higher level outcomes Limited formal assessment of farmers’ and Service Providers’ reactions to the BSTF service delivery.

Findings (Continued) B. Implementation of Change There has been a diverse range of responses to the implementation of the BSTF described in this evaluation.  Business as usual (no need for change!). For a number of projects, the project leaders and staff felt they were already using the BSTF principles of delivering services in collaboration with Service Providers.  Need relatively modest changes to more traditional approaches of service delivery by including additional Service Providers in the project.  Three projects have taken quite innovative approaches to service provision. For other projects there was not a strong sense of urgency for implementing BSTF.

Findings (Continued) Service Providers and service provision The findings highlighted the complexity of the contemporary service delivery environment in which DPI operates. For some projects, the SPs readiness to engage is often an issue and needs to be considered. The assessment of the support provided by DPI to implement the BSTF varied. While only 50% of respondents felt the support was adequate, for project leaders this figure raised to almost 90%.

Findings (Continued) ‘Relevant’, ‘Targeted’ and ‘Accessible’ services to farm businesses Service Providers have played an important role for a number of the projects to improve the relevance of services delivered to farm businesses and increase their accessibility. Feedback from Service Providers is having an important role in developing and refining service delivery to farm businesses. Working through Service Providers has encouraged a number of the projects to target priority issues or areas or focus on the larger industry sectors. A common theme for greater service accessibility was that collaboration with Service Providers mobilised broader networks and more resources for service delivery resulting in greater coverage of farm businesses.

Other Findings Strengths of the BSTF model BSTF has provided the authority and support for projects to make innovative/significant changes to service delivery BSTF facilitates access to a diverse range of expertise outside DPI BSTF has the potential to substantially broaden DPI’s access to farmers.

Other Findings (Continued) B. Key success factors for BSTF projects A number of key factors have emerged from this evaluation, which appear to support the successful implementation of BSTF at the project level: Leadership committed to the BSTF Strategy. Strategic analysis, research and planning in the project development phase. Perceived relevance of BSTF to the area of work/project. A specific and clear focus for BSTF change plus a strategy with a plan at the project level. Active and ongoing engagement and communication with potential Service Providers and farmer stakeholder groups.

Other Findings (Continued) Concerns For some staff, the ‘wholesaling’ approach and reducing their contact with farmers is seen as a disadvantage. Wholesaling imposes an additional administrative burden on projects and constrain flexibility in using external Service Providers. Not all Service Providers have the expertise or desire to extend DPI research or support the delivery of all DPI services. BSTF may impose additional costs on some farmers and be perceived as cost shifting. Some farmers may not understand or agree with DPI’s role in supporting other providers to deliver services.

Opportunities for improvement – Project Level The complexity of the DPI service delivery and its challenges highlight: The need for a comprehensive strategic analysis of the operating environment and needs of the farmers and Service Providers as a part of the project development process. The importance for DPI to consider project support and resourcing beyond the immediate boundaries of the project. Improved project documentation (particularly monitoring & evaluation related) to make the alignment with BSTF more explicit would facilitate informed decision making. To engage farmers and SPs in the proposed changes. To provide project staff with support for the ‘how to’ of implementing BSTF.

Opportunities for Improvement - PSE Methodology & Implementation Reviewing the first round Performance Stories Evaluation has highlighted some areas for improvement in the evaluation methodology and implementation including: 1.Using five different interviewers seems to have affected data consistency. 2.Information on actual and potential outcomes of the projects involved in the PSE was derived mainly from the project team for the first round. 3.While the PSE provided valuable evaluation information for the 8 projects involved, the findings could have been disseminated and discussed more widely across the organisation and used to inform other projects delivering against the BSTF strategy.

PSE as an Influential Tool The findings of this evaluation provide the BSTF strategy and project leaders with evidence and feedback for timely responses. The following findings should have a high priority: The need for a comprehensive ‘strategic analysis’ of the operating environment and needs of both farmers and service providers as part of project development process. It is critical at the project development phase to ensure there is an alignment between the DPI goals as stated in the BSTF success statement and the goals of potential Service Provider(s) to provide mutual benefits for both groups The need for an adequate capability building program to equip project staff with relevant knowledge and skills to handle BSTF requirements including dealing with Service Providers. The need to consider the adequacy of support and resourcing beyond the immediate boundaries and time frames of funded projects. A need for a stronger sense of urgency for implementing BSTF strategy.

Does it worth doing?