1. CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008

2. Background n Review Panel convened in response to State Water Board’s request for external review of CIWQS n First Panel meeting held in May 2007 n Preliminary Panel report with recommendations issued July 2007 n Water Board staff to consider recommendations and demonstrate progress toward benchmarks through February 2008

3. Findings of Previous Review n Seven essential recommendations: u Reduce the project’s scope u Restructure CIWQS’ project management u Validate the system’s requirements u Rebuild key constituencies u Address data quality issues u Produce key reports u Improve user interface

4. Goals for Phase II n Water Board staff will demonstrate progress by responding to Phase I recommendations u Including specific actions, timelines, and additional performance measures for the coming year n Panel will review progress and prepare final report to: u Judge adequacy of Water Board’s response u Make final recommendations for future direction of the project

5. Overview of Final Review n CIWQS remains essential for the State Water Board to fulfill its mission u Terminating funding would be counterproductive n CIWQS should be funded at an appropriate level n The program’s new direction should lead to success u We are impressed with the remarkable turnaround since May n We are particularly impressed with progress in reorganizing management and communicating with users

6. Overview (cont.) n The redefinition of project scope was appropriate and effective n We have remaining concerns with some aspects of the technical approach, rate of data cleanup, and report production

7. Original Recommendations n Reduce the scope n Restructure management n Validate system requirements n Rebuild constituency n Address data issues n Produce key reports n Improve user interface

8. Reduce the Scope n Evaluation u Achieved significant scope reduction u Increased probability of project success u Achieved buy-in from Steering Committee & users u Phased eSMR implementation plan is reasonable u Implementation through eSMR Level 3 is desirable u We concur with need for a separate federal interface n Recommendation u Consider trimming unneeded parts of the database to meet reduced project scope

9. Restructure Management Restructure Management n Evaluation u The new management structure positions CIWQS for success u Effective new organization with leadership in Water Quality not in DIT ties the project closer to the Board’s mission u Buy-in from senior managers for the project’s importance and new direction

10. Validate System Requirements n Evaluation u Fundamentally improved systems design & implementation u Corrected design and approach based on testing u The underlying structure can be made to work Referential integrity issues are being addressed Referential integrity issues are being addressed u Requirements better reflect user inputs & needs n Recommendations u Demonstrate a disciplined systems engineering process Document team members’ roles, responsibilities, activities Document team members’ roles, responsibilities, activities u Use test cases to demonstrate successful resolution of referential integrity issues

11. Rebuild Constituency n Evaluation u Achieved remarkable cultural change that significantly improved user communication and support u Developed effective user community structure u Users attended Panel meeting and strongly supported the project u Users strongly engaged in most areas u Issues with stormwater group illustrate cultural improvements have further to go n Recommendations u Should apply approach used for eSMR to all groups and issues

12. Address Data Issues n Evaluation u Less was accomplished than we had expected u Acknowledged the need for systematic QA/QC u Database integrity is being addressed u Some progress on correcting existing data errors n Recommendations u Devote more focused effort to QA/QC and data cleanup u Develop coherent plan that prioritizes data cleanup and identifies needed resources and timeframe u Create formal QA/QC mechanisms, including user participation u Make QA/QC an integral part of database administration and operation

13. Produce Key Reports n Evaluation u Created a mechanism to identify and prioritize key reports u Made some progress on developing canned reports n Recommendations u Demonstrate that required reports can be produced from the core CIWQS system u Increase staff’s ability to create ad hoc reports u Accelerate report production

14. Improve User Interfaces n Evaluation u Created a mechanism for input and feedback from users u Created a successful and easy-to-use SSO module u Seem to be heading in the right direction n Recommendations u Create a formal mechanism for validating and testing user interfaces u Create context-specific drop-down menus u Create context-specific online help

15. Additional Recommendations n Build a non-CIWQS interface with ICIS-NPDES n Develop a set of persistent end-to-end tests to demonstrate that the data model and processes work n Develop a deployment plan for the system and data n Fund the system at an appropriate level

16. ICIS-NPDES Issue Must Be Resolved n ICIS-NPDES will not meet State Board’s needs n There are different reporting needs for EPA and for State Board n State system should cross-check violation determinations between state and federal systems to quickly identify erroneous violations n Download ICIS-NPDES data to CIWQS as needed for State Board purposes

17. Evaluate Interface Options w/EPA n Consider discharger submission using u eDMR (file-oriented) u netDMR (interactive) n Develop test plan and case(s) including EPA DMR data download to state system u EPA has agreed to this using currently submitted data; this should be formalized quickly to take advantage of the offer

18. Develop End-to-End Testing n Develop and maintain a library of test cases with known results that can be used to test system and processes as changes are made n Should be implemented as part of all deployment plans on an ongoing basis

19. Develop Deployment Plan n Develop a deployment plan for every system update and release n Should include software, hardware, data, and quality control aspects n End-to-end testing should be integral part of the plan n Each phase of the eSMR development needs a deployment plan

20. Fund CIWQS at Appropriate Level n Panel is encouraged by progress n Believe that changes make it likely the CIWQS project will result in a system that serves the State Board’s needs for accuracy, automation, and efficiency for the future n Encourage State Board and Legislature to fund the development at levels that can maintain and accelerate development

21. Next Steps n Full report of findings and recommendations, with performance metrics, within 2 months n Panel open to revisiting progress in 9 – 12 months if asked

22. SummarySummary n The project has made substantial improvements and is on a path to success u Management restructuring, rebuilding user constituencies, and system validation are clear successes u Additional progress needed on data cleanup, reporting, user interfaces n Additional recommendations include: u Build a non-CIWQS interface with ICIS-NPDES u Develop end-to-end tests to demonstrate all functionality u Develop comprehensive deployment plan u Fund CIWQS at an appropriate level