1. Interactive Verification Program Training for AWIPS OB8.2 Hank Herr Office of Hydrologic Development RFC Verification Workshop, 08/15/2007

2. 2 Overview  Introduction  IVP Terminology  Software Components  Database Usage  Output Files  Design  Demonstration  Tips  Future Possibilities  Exercises  Discussion

3. 3  Original IVP written around 2000-2001 for old HP based verify software  Verification moved to archive database for OB4 –IVP Batch Program software written –Old IVP GUI still ran, but used pairs files created during batch processing; it did not access the archive database directly  New IVP GUI written for OB7.2 –Provided ability to build verification products via GUI, modify components of a chart and save modifications, create batch file to reconstruct graphics via IVP Batch Program –Accessed archive database directly –Limitations:  Designed for stage data  Accessed only three data tables of the archive database  Limited number of statistics  No quality control of pairs  Inflexibility in building batch files from GUI  JClass Desktop Views used to render charts (limited charting capabilities; upgrade cost too much money) Introduction History of IVP

4. 4 Introduction New for OB8.2  Designed for verification of stage, flow, temperature, and precipitation forecasts –Can do other data types as long as data is stored within accessible tables –Accesses 10 data tables of the archive database  Uses all data tables except pempsep, pemrsep, and pairedvalues  Additional statistics –Moments, CDF Plot, ROC, etc.  Quality control of pairs via GUI –View pairs in a time series plot, in addition to existing scatter plot –Mark pairs as ‘bad’ so that verification ignores them –Change forecast or observed values within a pair and save changes to database  More flexible batch file construction –Batch file can be built that will construct current chart for other combinations of locations and analysis periods  Chart Director used for rendering –Relatively quick, memory efficient chart rendering –Used previously for HydroGen, so a run-time license already exists

5. 5 IVP Terminology  Verification Location –A forecast point or location and a data type for which verification is to be performed –Uniquely specified by location id (lid within database) and SHEF pedtsep code. –Includes following information:  Forecast and observed categories  Observation type (raw or processed)  Active flag (yes or no)  National flag (yes or no)  Sensor preferences (ordered list of observed type source, representing preferences for which observed values to pair with a forecast)

6. 6 IVP Terminology  Verification Group –A set of locations and parameters that define the data to load and other parameters of verification  Locations must have the same number of categories and must be of the same type: stage (height), flow (discharge), temperature, precipitation –Includes following parameters:  Analysis period and intervals  Lead time period and intervals  Issuance time-of-day (0-24 hours) period and intervals  Restrictions on location response time, active flag, and SHEF components (forecast type source, physical element, duration, and extremum)  Whether or not to compute statistics independently for each lid or forecast type source

7. 7 IVP Terminology  Forecast-Observed Data Pair –A forecast value paired with an observed value  Observed value can be “raw” or “processed” depending on if the value was posted to the database by the raw or processed SHEF decoder  Each forecast can have one raw pair and one processed pair associated with it –“Best” candidate observation is chosen for each forecast based on following criteria:  SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order)  sensor preference (observation type source) rank  difference between forecast validtime and observation obstime –A pair can be “good” or “bad”  Bad pairs are not used in verification computations

8. 8 Software Components  Vfyruninfo Editor GUI –Manage vfyruninfo table  IVP –Data quality control and define verification products graphically  IVP Batch Program –Build pairs and verification products via a batch language  IVP Batch Builder –Edit IVP Batch Program batch files graphically  Verify Pairs Ingestor –Ingest pairs directly from text files of specific format into the vfypairs or vfyprocpairs tables  del_bad_pairs –Removes bad pairs from the vfypairs and vfyprocpairs table

9. 9 Database Usage  Verification Location Information –Vfyruninfo table  Defines locations for which verification can be performed.  Individual entries for each verification location and sensor preference component (observed type source)  Edited via Vfyruninfo Editor GUI –Location table  Information used in building labels, includes RFC name, full location name, etc. –Rivercrit table  Flood stages and flows (if location is of appropriate data type).

10. 10 Database Usage  Building Pairs –Tables of data for pairing:  Forecasts: pedfsep, pehfsep, peqfsep, pedcsep  Raw Observations: pecrsep, pedrsep, peoosep  Processed Observations: pedpsep, pehpsep, peqpsep, peoosep –Vaddadjust table  Adjust observations prior to pairing with a forecast  Only affects forecast-observed pairs  Is not dated; there can be only one entry per location –Sensok table  Allows for specifying time ranges during which sensors should not be used (i.e. observations should not be used for pairing)

11. 11 Database Usage  Storing Pairs –Vfypairs table  Stores pairs constructed using raw observations  Field quality_code defines if the pair is good or bad  Can be very large (millions of records) –Vfyprocpairs table  Stores pairs constructed using processed observations  Field quality_code defines if the pair is good or bad  Can be very large, but will likely be a fraction of the size of vfypairs –A forecast can have up to two pairs built for it: one stored in vfypairs and one stored in vfyprocpairs  Only one of the two pairs can be analyzed at one time

12. 12 Output Files  Output Statistics File (batch only) –Verification statistics output in an ASCII tabular format  Pairs File –Pairs used to compute statistics –ASCII tabular format  Output Image File (.png,.jpg) –PNG or JPG files displaying a plot of verification statistics  Image Data File (.dat) –Statistic values comprising an output image plot –ASCII tabular format

13. 13 Design Foundation Archive Database  Archive database provides the following: –Forecasts and observations –Verification location information –Forecast-observed data pairs User

14. 14 Design Vfyruninfo Editor Archive Database Vfyruninfo Editor User Vfyruninfo rows  Vfyruninfo Editor is used to specify verification locations and saves them to the vfyruninfo table of the archive database

15. 15 Pairing Processor Archive Database User pairs Design IVP Batch Program Vfyruninfo Editor Vfyruninfo rows forecasts, observations, location info  IVP Batch Program provides the user the ability to: –Pair forecasts with observed values according to user specified parameters parameters Batch Batch file

16. 16 Design IVP Batch Program Verification Processor Pairing Processor Archive Database User pairs ASCII pairs, location info forecasts, observations, location info Vfyruninfo Editor Vfyruninfo rows pairs, statistics  IVP Batch Program provides the user the ability to: –Calculate statistics and output results to tabular ASCII files parameters Batch Batch file

17. 17 Design IVP Batch Program Verification Processor Pairing Processor Archive Database User ASCII pairs Graphics Processor Graphics statistics params Vfyruninfo Editor Vfyruninfo rows forecasts, observations, location info pairs, location info pairs, statistics  IVP Batch Program provides the user the ability to: –Generate user customizable graphical verification products displaying statistics parameters Batch Batch file

18. 18 params Verification Processor Pairing Processor Archive Database User GUI Graphics ASCII pairs pairs, statistics Design IVP GUI Vfyruninfo Editor Vfyruninfo rows forecasts, observations, location info pairs, location info  IVP provides the user the ability to: –Specify, build, and view graphics Graphics Processor statistics params parameters Batch Batch file

19. 19 params Verification Processor Pairing Processor Archive Database User GUI Graphics ASCII pairs pairs, statistics Design IVP GUI Vfyruninfo Editor Vfyruninfo rows forecasts, observations, location info pairs, location info  IVP provides the user the ability to: –Modify forecast-observed data pairs Graphics Processor statistics params pairs parameters Batch Batch file

20. 20 params Verification Processor Pairing Processor Archive Database User GUI Graphics ASCII pairs parameters pairs, statistics Design IVP GUI Batch Vfyruninfo Editor Vfyruninfo rows forecasts, observations, location info pairs, location info  IVP provides the user the ability to: –Create batch files to produce graphics via IVP Batch Program Graphics Processor statistics params pairs Batch file batch tokens

21. 21 Design IVP Window Layout 1. Load Pairs 2. View and QC Pairs 3. Define Products 4. View Products

22. 22 Demonstration  Feature availability: –Both OB7.2 and OB8.2 –OB8.2 only*

23. 23  Start Vfyruninfo Editor  Filter ingestfilter table contents*  Select locations from ingestfilter table*  Create locations from scratch  Specify location parameters –Active* –National* –Response time –Sensor preferences  Save to archive database Demonstration Vfyruninfo Editor ref: Vfyruninfo Editor User’s Manual

24. 24  Commands: –Time window: START_TIME, END_TIME –Specify locations: LOCATION, DUR*, EXTREMUM*, FCST_TS*, PE* –Pairing window: PAIRING_WINDOW –Observation type: OBS_TYPE (replaces OBS_TABLE)* –Persistence flag: PERSISTENCE  Action: –Build the pairs: BUILD_PAIRS Demonstration Pairing Batch File ref: IVP Batch Program User’s Manual for Pairing

25. 25  Example (buildpairs_template.bat): Demonstration Pairing Batch File # Pair all locations. Use default settings of # ALL for all other location specifying commands location = ALL # Build pairs for a two week window prior to # current system time start_time = "* - 14 days" end_time = "*" # Parameters pairing_window = 1 obs_type = RAW build_pairs = true ref: IVP Batch Program User’s Manual for Pairing

26. 26  Commands: –Time Window: START_TIME, END_TIME –Specify Locations: DUR*, EXTREMUM*, FCST_TS, PE  Action: –Compute statistics: NATLSTATS  Recommendation*: –Set the national flag for all locations that are part of the national program to ‘Y’ (use Vfyruninfo Editor) –Give the NATLSTATS action a value of NATIONAL Demonstration National Statistics Batch File ref: IVP Batch Program User’s Manual for Verification

27. 27  Example (natlstats_template.bat): Demonstration National Statistics Batch File # Compute stats for September, 2003. START_TIME = “2003-09-01 00:00:00” END_TIME = “2003-09-30 23:59:59” # Default Values are ‘ALL’ # FCST_TS = ALL # PE = ALL # DUR = ALL # EXTREMUM = ALL NATLSTATS = NATIONAL ref: IVP Batch Program User’s Manual for Verification

28. 28  Start IVP go vsys_scripts ivp  Specify parameters of a verification group –Analysis period and intervals –Lead time period and intervals –Issuance time period and intervals* –Miscellaneous parameters Demonstration IVP: Verification Group Manager ref: IVP User’s Manual, Section 9

29. 29  Specify verification locations that comprise a group –Filter available locations* –Choose locations –Specify categories –Specify observation type (from which pairs table to acquire data)* Demonstration IVP: Verification Location Manager ref: IVP User’s Manual, Section 13

30. 30  Create IVP Data Display –Loading data progress window  Open the Verification Group Manager and Verification Location Manager  Save the image Demonstration IVP: IVP Data Display ref: IVP User’s Manual, Section 15

31. 31  Examine data through two views: –Scatter Plot View  Select a location* –In OB7.2, use Show button in Verification Location Manager  Display info about a location*  Line x = y –Time Series View*  Specify time series to view  Emphasize a time series Demonstration IVP: IVP Data Display ref: IVP User’s Manual, Sections 15.1, 15.2

32. 32  Draw a selected region rectangle  Zoom and the navigator panel –Zoom in on scatter view –Zoom in on time series view*  Panning through the time series by zooming –Shift the zoomed region via the navigator panel and keyboard* –Zoom out Demonstration IVP: IVP Data Display ref: IVP User’s Manual, Sections 15.6, 15.7

33. 33  View Pairs via the Verify Pairs Viewer –Select pairs –Sort table –Create a pairs file  WARNING: Do not view very large (30,000+) sets of pairs… the table rendering may be too slow Demonstration IVP: IVP Data Display ref: IVP User’s Manual, Section 16

34. 34  Mark pairs as good or bad via IVP Data Display*  Edit pairs via the Verify Pairs Editor* –Select pairs –Mark pairs as good or bad –Edit forecast or observed value manually –Focus on a pair –Edit forecast or observed value graphically  Scatter view  Time series view  Change the mouse mode*  WARNING: Do not select large sets (1000+) of pairs for editing… the table rendering speed may be too slow Demonstration IVP: IVP Data Display ref: IVP User’s Manual, Sections 15.11, 17

35. 35  Open the Plot Definition Manager  Select statistics –IVP Statistic Chooser Manager  Select plot type  Select x-axis variable  Select comparison variable*  Edit statistics parameters*  Edit group parameters* –Spawns a window nearly identical to Verification Group Manager  Note how variable selection affects analysis, lead time, and issuance time-of-day intervals  Note how variable selection affects category choice boxes Demonstration IVP: Verification Plot Definition Manager ref: IVP User’s Manual, Sections 18, 19

36. 36  Errors  Categorical Statistics  Quantiles and Extremes –Computed relative to whichever variable whose category used is set to “Do Not Use”  Moments*  Sample Size –The number of pairs used in the computations –or– –Number of pairs within the category (if x-axis or comparison variable is forecast or observed categories)  Misc.*  Special Plots* Demonstration IVP: The Statistics ref: IVP Batch Builder User’s Manual, Appendix B

37. 37  Categorical Statistics use 3x3 Contingency Table  Examples: –HFAR = D/(D + E + F) –TFAR = (D + F)/(D + E + F) –CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F)  In a 2x2 contingency table –HFAR is always 0 for the lowermost category –TFAR and HFAR are equivalent for uppermost category  POD + UFR + OFR = 1 in all cases Demonstration IVP: The Statistics ref: IVP Batch Builder User’s Manual, Appendix B

38. 38 Demonstration IVP: Verification Plot Definition Manager x-axis primary y-axis comparison variable secondary y-axis Statistic short name (matches output files)

39. 39  Choose categories used –Disabled if x-axis or comparison variable is forecast or observed categories –Select “Do Not Use”  Only one of the two choice boxes can be “Do Not Use” –Impact of selections on how statistics are computed  Category used restricts pairs used in computations to only those within the category  Some statistics are calculated relative to the variable not used to categorize data (e.g. quantiles, cdf plot*, pdf plot*) –Special case: ROC plot*  Category used defines observation threshold Demonstration IVP: Verification Plot Definition Manager ref: IVP User’s Manual, Section 18.7

40. 40  Create IVP Statistic Display –Data is always loaded from memory even if the overridden group parameters imply additional data is needed (such as a wider analysis interval, for example) –Allows for relatively quick plot generation  Save the image  View the data via the IVP Statistics Data Viewer –Create an ASCII format image data file Demonstration IVP: IVP Statistic Display ref: IVP User’s Manual, Section 21

41. 41  Create a batch file –Change the analysis period* –Change the included locations* –Create other groups for which to generate products* –Specify output and template file names  Replacement strings are needed if images are to be produced for multiple groups in order to avoid overwriting output files* –Specify batch file name Demonstration IVP: IVP Statistics Display ref: IVP User’s Manual, Section 23

42. 42  Open up the Chart Property Manager –Click on menu item or tool bar item –Click on plot component*  Change labels  Change axis limits or chart dimensions  Change legend components  Change x-axis labels (for IVP Statistic Display)  Create a template file –Don’t modify manually unless absolutely necessary  Apply changes from a template file Demonstration IVP: Chart Property Manager ref: IVP User’s Manual, Sections 24, Appendix D

43. 43 Demonstration IVP: Chart Property Manager X-Axis Title Y-Axis 1 Title Legend Y-Axis 2 Title Legend Title Tick Mark Font Plot Title

44. 44  Start IVP Batch Builder  Load batch file  Insert, append, delete command lines  Edit command lines  Comment and uncomment command lines  Check for errors  Run batch file Demonstration IVP Batch Builder and the IVP Batch Program ref: IVP Batch Builder User’s Manual

45. 45  Create one batch file to generate products for many locations –@FILE Command to refer to other files –Replacement strings to avoid overwriting output files Demonstration IVP Batch Builder and the IVP Batch Program ref: IVP Batch Builder User’s Manual

46. 46  Organize output by using subdirectories of $(vsys_output)* –If producing graphics for each location, use @ONELID or @ONELIDSHEF to define directories: OUTPUT_FILE = “@ONELID/output.txt,c” –If producing graphics for time periods, use @STARTTIME or @ENDTIME: OUTPUT_FILE = “@STARTTIME{[CCYY]}/output.txt,c” Demonstration IVP Batch Builder and the IVP Batch Program ref: IVP Batch Builder User’s Manual

47. 47  To minimize time spent querying database, load all needed pairs at once –Only specify the overall analysis period and choose the locations within the Verification Group Manager and Verification Location Manager  Specify other parameters via the Plot Definition version of the Verification Group Manager (click on Edit Group Parameters in the Plot Definition Manager) –Important when generating many products  To minimize the chart rendering time for the IVP Data Display, keep the number of pairs small –Do one location at a time for a couple years –Do several locations at a time for one month –Important when quality controlling pairs data  See the Memory Limitations Tips later for recommended numbers of pairs to load Tips: Verification Group Manager Usage

48. 48  Use OBS_TYPE “RAW” for paired observations if… –Observed data was posted to archive database via raw SHEF decoder, AND –First letter of type source is ‘R’.  Use OBS_TYPE “PROCESSED” if… –Observed data was posted to archive database via processed SHEF decoder, OR –First letter of type source is not ‘R’  Example: –If processed observations were posted via raw SHEF decoder (to peoosep table), then specify PROCESSED Tips: Deciding Between PROCESSED or RAW

49. 49  Specifies parameters of graphical user interfaces –Fonts –Window Sizes –Colors –Others  Location –Default: $(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt –Directory can be overridden by exporting environment variable SYSTEM_SETTINGS_FILE_DIR  Can allow for user specific system settings  Format –ESPADP/IVP batch language standard “ = ” format Tips: System Settings Files ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A

50. 50 Tips: Memory Limitations  Recommended number of pairs: –Limits can be changed by using the ivp and ivpbatch script –m and –s options (run “ivp –hh” or “ivpbatch –hh” to read about these options) –Limits above are for IVP GUI; IVP Batch Program can deal with more pairs because not all pairs need to be stored in memory at one time  If IVP GUI runs out of memory, an error message will pop-up and the pairs will be discarded (in most cases) –The software may crash in other cases  Verify Pairs Viewer window is a memory hog –Add about 25% to the recommended value if not using the Viewer ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A

51. 51  Debug problems –Export vsys_debug token prior to execution to see additional output  Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128.  “export vsys_debug=20” to see additional progress output  “export vsys_debug=40” to see database query where clauses and timing results for chart rendering  “export vsys_debug=128” to see everything (lots and lots of output)  Performance –If IVP Data Display chart rendering is slow…  Reduce number of pairs viewed  Zoom in on regions (points outside current zoom area are not rendered)  Run on faster machines (port code to LX and execute software there) –If database query speed is slow…  Turn off SHEF decoders for a short time (if you can)  Reduce number of locations or width of analysis period Tips: Debugging and Performance ref: IVP User’s Manual, Section 25

52. 52 Future Possibilities  Confidence Intervals –Algorithms need to be developed –Ability to produce CI values and plot them needs to be added  GUI redesign –Reduce the number of windows –Reorganize window so that the Verification Group Manager does not need to be in two places  Add ability to handle multiple comparison variables  Add ability to modify other properties of a chart –Tick spacing, line style (solid, dashed, etc), line width, etc. –Requires additions to Chart Property Manager  Create baseline batch files requiring no changes in order to produce standard graphics –Standard graphics need to be determined  Add ability to put changed forecasts and observed values back into original data tables and rebuild all affected pairs on-the-fly –IVP would become a more powerful, general QC tool for archive data

53. 53 Documentation  Available on the web http://www.nws.noaa.gov/oh/hrl/verification/verification_doc_ob82.php

54. 54 Exercises  Described in hand-outs –The data sets are very large, so while the pairs are loading, discuss the plots that are needed in order to answer the exercise questions  What statistics are to be displayed?  What are the x-axis and comparison variables?  Are there any observed or forecast category restrictions?  Group assignments

55. 55 Discussion  Review exercises  Pre-workshop questions –Have they been answered?  General comments on software