1. Designing a DTC Verification System Jennifer Mahoney NOAA/ESRL 21 Feb 2007

2. Talk Overview Define verification system Present the key components that will form the DTC verification system framework  Allow flexibility  Serve a variety of users Discuss the complexities of developing the DTC verification system

3. Acknowledgments ESRL/GSD Verification System Engineering Team  Sean Madine  Nick Matheson  Missy Petty  Dan Schaffer

4. What is a Verification System?

5. What is a DTC Verification System?

6. Verification Analyses as a Function of Software Characteristics

7. DTC Verification Analysis Activities

8. Software Characteristics

9. Non-Functional Requirements Important Operational Considerations  Stability  Reliability  Security Important Development Considerations  Support for collaborative interaction  Adaptability to other environments  Extensibility to other scientific problems  Maintainability  Portability to other organizations of the ‘System’

10. Verification Mechanics Aggregate/ Combine to Produce Specific Metrics Pre-process Forecasts Pre-process Observations Create Displays Verification System Components- Historical Perspective Good solution for a small set well defined questions System structure remains consistent, but can expand forecast dataset Solution needs to change when want to add observation datasets, new verification mechanics and assessment techniques, integrate non- meteorological information, and inter-compare a variety of forecast systems

11. Verification System Components: Data Management Structuring System Data Warehouse Finest resolution Forecast/observation pairs + forecast/obs attributes

12. Verification System Components: Data Management Structuring System Data Warehouse Finest resolution Forecast/observation pairs + forecast/obs attributes Data Marts

13. Verification System Components: Data Management Structuring System Data Warehouse Finest resolution Forecast/observation pairs + forecast/obs attributes Data Marts Intercompare Forecast Systems Produce Statistical Information Aggregation driven by specific data relationships

14. Verification Mechanics Pre-process Forecasts Pre-process Observations Displays DTC Verification System Components Store Finest Granularity Verification Elements Apply storage Optimization Strategy Database info Query interrogation to compute score

15. Verification System Complexities Management of complex interdependent relationships between datasets  Event-driven component initiation Eliminate unnecessary time delays between data execution steps Handle abnormal delays in data access, processing, and user access Account for differences between operational deployments  Multi workflow requests Incorporate relationships with the addition of new datasets, verification methodologies, and forecast intercomparisons Provide only meaningful workflow comparisons (i.e., par down list of all possible choices to meet user analysis questions)

16. Verification System Complexities Effective access to the results  Provide tiered optimizations Fast turn around of pre-defined set of analyses Flexibility to define varying set of analyses Support wide-range of user needs  Allow for wide variety of graphical and standard displays Traditional statistical displays Displays that allow the integration of verification information

17. Summary Verification system will mainly support the forecast intercomparison evaluation (e.g., core tests) function of the DTC Scientific conceptualism developed for the toolkit will feed into the DTC verification system DTC requirements for the verification system require complex data management and interrogation strategies Tiered approaches for data access are required to provide maximum system flexibility to end users

18. Future Efforts Gather verification system requirements from the workshop Develop a Functional Requirements document Begin preliminary development of the system concepts by the end of the year

19. Contributors to the DTC System Development NCAR – scientific verification concepts and toolkit capabilities NCEP – bridge from the current NCEP verification system functionality to the DTC verification system framework GSD – verification system framework by extending database management and web access strategies DTC users – functional requirements for the verification system

20. Questions?