1. PAS Project 1 Setting Up Performance Assessment System CEPT UNIVERSITY

2.  This session describes experience in India of setting up a performance assessment system  Participants have to realise that it is possible to replicate such a system in other context as well.  The focus of this presentation is on describing the data input modules and generating various reports and graphics.  Participants can be involved in the discussion by showing the live demo of the website. (www.pas.org.in)  More information is available on www.pas.org.in PAS Project Learning objectives 2

3. FrameworkDescription IWA system Consists of broadly four parts, namely, ‘data elements’ that feed into variables used for the performance indicators. PIs are analysed further with respect to ‘explanatory factors’, and with reference to ‘context information’. No. of PIs: 170 (for water) No. of variables– 182 (for wastewater) AWWA Consists of dataset in five areas of water and wastewater utility operation, including organisational development, customer relations, business operations, and water and wastewater operations. No. of PIs: 22 (for water and wastewater) IBNET system Consists of a ‘dataset’ broadly covering aspects of utility information, service area, water and sewerage service, financial and customer information. The indicators are drawn from the above dataset, along with additions of utility specific indicators. No. of data items: 148; No. of Performance Indicators (PI): 27 (including water and wastewater) ISO series Consists of indicators related to water and wastewater services like provision of service, contract management and billing, and consumer relations. Wastewater systems consider both sanitary as well as industrial wastewater drained into sewers. Frameworks and Indicators: IBNET, IWA and ADB Utility Data Books 3 Sources: AWWA: Cabrera (2011), IBNET: www.ib-net.org, IWA: Alegre et. Al (2006), ISO: ISO 24510: 2007, ISO 24511: 2007, ISO 24512: 2007www.ib-net.org

4. Utility associationsGovernmentRegulator Coverage National & regional level National & state (province) level National & regional level Examples Africa, South-East Asia, Australia, Netherlands, South Africa, Canada, Vietnam & Indonesia Performance monitoring: Brazil, Australia, Tanzania, South Africa and India Performance-based funding: Ecuador, Uganda, Tanzania & India Regulators: UK, Zambia and Philippines Performance-based contracts: Senegal, Uganda, Burkina Faso, Malaysia & Bangkok Objectives a. Sharing information across utilities; b. Promote process benchmarking a. Support decision making and improvement plans; b. promote process benchmarking; c. Funding as incentive for improved performance  a. Comparative regulation;  b. Review against agreed performance targets in contract Major themes Service levels, finance, consumer services, environment Service levels, consumer services, finance, environment, health and asset management Service levels, consumer services, finance FrequencyAnnual (Netherlands: once in three years) Annual Use of Performance Information around the World 4

5. Performance Benchmarking by Selected National Utility Associations 5 Data collection Canadian experience looked at all components excluding web based reporting Online web based reporting system Carry out Action Plan Denmark (DANVA) benchmarking comprised of all components, excluding onsite verification Onsite verification Monitoring key business trends Netherlands apart from covering all components (site visits excluded) also has made benchmarking mandatory Setting of goals/targets Performance based grants Vietnam experience looked at all components but process benchmarking Process bench- marking

6. Utility Management Model approach of Canadian benchmarking 6

7. Government Initiatives in Performance benchmarking: case of Brazil SNIS: 7 Setting up of the National Information System on Water, Sanitation and Solid Waste (SNIS) in 1996; over 16 yrs historical data Coverage of over 70% of municipalities in Brazil at Utility/ Municipalitiy level and Regional utility level SNIS database consists of over 80 indicators assessing technical performance, financial viability and customer satisfaction Data collection through software and online through SNISWEB Data quality control and data transfer to SNIS database Report generation and dissemination Outsourcing of services: Data quality handled by private firm, Help desk services to utilities, Verification of data received from utilities, Follow ups on returning data forms to SNIS Water Law of Brazil provides for establishment of SNIS. Performance criterion of utility operators is criteria in allocation of federal resources Major investment programs of government are result based (Growth Acceleration Program with fund allocation of US$ 23 billion)

8. Lessons from benchmarking efforts 8

9. Setting up a government led benchmarking system in India 9

10. State agencies, data and analysis Web based measurement and monitoring Data entry: at city level Data storage: at state level Shared data available at various levels Cities – data, results, analysis Citizens Access to analysis, results

11. Stages in Performance Measurement and Monitoring Online data entry Validation based on comparative analysis Updating data City and state profiles Review by city and state Final approval of data Performance measurement reports Access controlled (password for each user) START

12. We have come a long way from this

13. Online data entry camps in state capitals PAS Project 13

14. Time taken for annual assessment Annual Assessment in MAHARASHTRA over five year assessment period Data collection through city visits in Year I (2008-09), supported by Partners 2008-09 2009-10 2010-11 2011-122012-13 Data collection through online modules in Year V (2012-13). Self assessment by select ULBs capacity building Training in use of tools Tools translated in local language Constant hand holding by PAS team Support for resolving data issues Support by State Government for Annual Assessment Annual Assessment in GUJARAT over five year assessment period 2008-09 2009-10 2010-11 2011-122012-13

15. Online data entry  General instructions on how to fill checklist online  Options to download checklist in excel format, and in local language  Option to view the approval status Unique access for each city

16. Online data entry  Previous year’s data displayed alongside current year of data entry  Options to save each sub section within a sheet; useful in case of connectivity issues during data entry

17. Indicator Reports  Options to view indicator report  Option to print and save indicator report in PDF

18. Monitoring

19. State level monitoring Options to download  Data for all years  Indicators for all years  Targets in gazette format, excel/ PDF

20. State Profile based on KPIs Option to view year wise information

21. Option to view Thematic State profile State Profile based on KPIs

22. Overview of the city provides general information across all the sectors City level monitoring

23. City Profile gives results based on SLB indicators for each sector City level monitoring

24. Compare Your City helps each city compare itself with another city based on its respective class or state. Comparing cities

25. Analysis of Indicators by Interactive Dashboards Interactive data visualization that provides valuable insight Allows drilling down from indicators, charts and maps to read data better

26. Analysis of Indicators by Interactive Dashboards Interactive data visualization that provides valuable insight Allows drilling down from indicators, charts and maps to read data better Allows combination of multiple data sources to keep track of performance

27. Analysis of Indicators by Interactive Dashboards Interactive data visualization that provides valuable insight Allows drilling down from indicators, charts and maps to read data better Allows combination of multiple data sources to keep track of performance Indicates points of action or decision required

28. Key Lessons from India experience  Scale is critical to demonstrate importance of performance assessment systems  Data quality will not be perfect..but rather than wait for perfect data, it is important to develop a system of benchmarking  Working with governments is essential, but so is patience…..adequate time required to mainstream in government system – 5 to 7 years  Government ownership and regular reviews are essential  Once fully set up, it can be used for both outcome monitoring and rational investment decisions  Benchmarking exercise needs to be linked to performance based funding

29. end