2 Draft of proposed metrics for the WEQ BPS to conduct analysis and evaluation of the Parallel Flow Visualization pilot project. (PFV Pilot) Bert Bressers,

3 Overview of the Data currently calculated and presented by IDC and the Data calculated and presented by PFV pilot Summary of the objectives of proposed metrics to analyze and evaluate results of PFV pilot. Summary of proposed metrics Details of the proposed metrics Topics

4 Overview of the Data currently calculated and presented by IDC and the Data calculated and presented by PFV pilot Data currently calculated and presented by IDC Firm and Non Firm Market Flow impact of Markets The impact of Firm and Non Firm Inter BA Tags. (BA – BA TDF) For non-Market BAs: The impact of Generators serving Native Load of the BA. The output of available generators is scaled based on Load and ownership factor. No relation with real time. Data calculated and presented by PFV pilot Market Flow impact of Markets. (all Firm) The impact of Firm and Non Firm Inter BA Tags. (BA – BA TDF) The impact of Non Firm Intra BA Tags (Gen – BA GLDF) For non-Market BAs: The impact of Generators serving Native Load of the BA. The output of the Generators is based on real time adjusted for resource specific Tags sourcing from Generator. Note: Differences highlighted Red. To be confirmed by Nelson.

5 Summary of the objectives of proposed metrics to analyze and evaluate results of PFV pilot.. Validate the PFV calculation logic. Measure the accuracy of the PFV results. Measure the improvements resulting from PFV pilot. (Improvements defined as more accurate impacts calculated by PFV Pilot and therefore more accurate relief assigned to entities by PFV pilot) Measure the accuracy of the input data submitted by BAs, RCs, TOs, TSPs Measure availability of the input data submitted by BAs, RCs, TOs, TSPs

6 Summary of proposed metrics to analyze and evaluate PFV results 1.Breakdown of the real time flow of a flow gate and determination of amount of unreported impacts. 2.Comparison of the NNL results (GTL Calculations) of current IDC and the NNL results of the PFV pilot. 3.Differences in Non Firm curtailments (current IDC versus PFV pilot) for TLR Level 3 4.Differences in Firm curtailments (current IDC versus PFV pilot) for TLR Level 5 5.Accuracy of input data of PFV pilot project. 6.Availability of the input data of PFV pilot project. 7.Accuracy of the Market Flow calculation of PFV pilot.

7 Breakdown of the real time flow of a flow gate and determination of amount of unreported impacts. Calculate breakdown of real time flow: [A] Net Firm and Non Firm Market Flow impact of Markets [B] Net impact of Firm and Non Firm Inter BA Tags. (BA – BA TDF) [C] Net impact of Non Firm Intra BA Tags (Gen – BA GLDF) [D] Net impact of generators serving Native Load of the BA (non Market) [E] Unreported impacts. Real time flow minus { [A] + [B] + [C] + [D] } Calculate unreported impacts for all flowgates once an hour, calculate average value and standard deviation and present statistical results if proposed in geographic overviews and by season, month, on peak, off peak, weekday, weekend day. Analyze flowgates that have large unreported impacts and determine possible reasons for the large unreported impacts. We could go a step further on the statistical analysis and identification of correlations with season (tough one since this will be done only for one year), time of day/day of week, load, weather patterns (cold in the North, hot in the South), etc, and whether we can forecast the error at any point in time and calculate next hour impacts with better certainty. 1 Note: We could consider calculating average impacts and standard deviations of [A], [B], [C], [D] separate for all Months of the Year and present those statistical results for further analysis. To be discussed if this adds value or is just a nice to know. Objective: Measure the accuracy of the PFV results Validate logic of PFV pilot. Note from Nelson: We might want to limit the number of flowgates that we include in metrics. There are 2,000 flowgates in IDC. The results and analysis on all flowgates will be overwhelming and we may end up spending a lot of time on a handful of not-so-important flowgates that have exceptional (poor) results, and lose track of the overall picture. I think we could use the flowgates that are normally in TLR plus a few relevant and strategically located flowgates, perhaps not to exceed 100. We can always adjust the list of flowgates during the pilot.

8 Comparison of the NNL results of current IDC and the NNL results of the PFV pilot. Calculate the impact of generators serving Native Load of the BA based on current IDC logic and compare the results with the results of the PFV pilot. Perform the calculations described above in first bullet for all flowgates once an hour and determine the average difference and standard deviation. Present data for following time periods : Full Year of data Weekdays only Weekend days only For each Month of Year Peak hours of day Off peak hours 2 Objective: Measure the improvements made by PFV pilot. Note: To be discussed if presenting data for listed time periods other than full Year, adds value or is just a nice to know. Note: Improvements defined as more accurate impacts calculated by PFV Pilot and therefore more accurate relief assigned to Entities by PFV pilot)

9 Differences in Non Firm curtailments (current IDC versus PFV pilot) for TLR Level 3 Select a list of flowgates that are frequent in TLR. Issue TLR Level 3 in both current IDC and PFV pilot. Calculate the differences in Non Firm curtailments for: Non Firm Inter BA Tags Non Firm Market impacts. Non Firm Intra BA Tags Non Firm GTL impacts of Non Market BAs Perform a number of calculations and present the results in statistics (average difference, standard deviation of difference). Objective: Measure the improvements made by PFV pilot. Validate logic of PFV pilot. 3 Note: To be discussed if this is a useful metrics if we take into account the fact that PFV considers all Market impacts as Firm. (?) Maybe select flowgates that have no or low Non Firm Market impact. Note: Improvements defined as more accurate impacts calculated by PFV Pilot and therefore more accurate relief assigned to Entities by PFV pilot)

10 Differences in Firm curtailments (current IDC versus PFV pilot) for TLR Level 5 Select a list of flowgates that are frequent in TLR. Issue TLR Level 5 in both current IDC and PFV pilot. Calculate the differences in Non Firm and Firm curtailments for: Non Firm and Firm Inter BA Tags Non Firm and Firm Market impacts. Non Firm (and Firm) Intra BA Tags Non Firm and Firm GTL impacts of Non Market BAs Perform a number of calculations and present the results in statistics (average difference, standard deviation of difference). Objective: Measure the improvements made by PFV pilot. Validate logic of PFV pilot. 4 Note: To be discussed if this is a useful metrics if we take into account the fact that PFV considers all Market impacts as Firm. (?) Maybe select flowgates that have no or low Non Firm Market impact.

11 Accuracy of Input data of PFV pilot project. RCs/BAs submit following data to SDX: Transmission outages and generator outages. Actual BA Load and forecasted BA Load Real time flows of monitored element and contingency element Real time output of all generators Real time flow of all tie lines Priority of all generators (Firm MW, Non Firm MW ) Proposed metrics: Compare Sum of Tie line flows and Sum of the Tags crossing BA boundary. Should be close to 0. Compare Sum of Actual MW of Generators of BA and Sum of the Actual Load plus Sum of tie line flows. Result should be close to 0. Identify generators that have real time output >0 (or threshold TBD) and at same time outage defined. Compare the real time flow gate flow calculated by RC with the real time flow of monitored element submitted to IDC plus real time flow of contingency element times IDC LODF factor. Present the results in statistics (average difference, standard deviation of difference) measured over a defined time period (e.g. Month). Report metrics results on a monthly basis to identify increases or decreases of the accuracy of the input data Objective: Measure the accuracy of the input data submitted by BAs, RCs, TOs, TSPs 5

12 Availability of the Input data of PFV pilot project. RCs/BAs submit following data to SDX: Transmission outages and generator outages. Actual BA Load and forecasted BA Load Real time flows of monitored element and contingency element Real time output of all generators Real time flow of all tie lines Priority of all generators (Firm MW, Non Firm MW ) Proposed metrics: Monitor the change of the time stamp of the data submission and if not changed in a defined time period (e.g. 30 minutes) mark that interval as input data not available. Present the results in statistics (% availability ) measured over a defined time period (e.g. month). Report metrics results on a monthly basis to identify increases or decreases of the accuracy of the input data.. Objective: Measure the availability of the input data submitted by BAs, RCs, TOs, TSPs 6

13 Accuracy of the Market Flow calculation of PFV pilot. Compare the following Market impact data of current IDC with the Market impact data of PFV for all Market Coordinated flowgates : Total Forward Market Impact >0% Total Forward Market Impact >5% Total Reverse Market Impact >0% Total Reverse Market Impact >5% Net Market impact >0% Perform the calculations described above in first bullet for all Market Coordinated flowgates once an hour and determine the average difference and standard deviation. Present metrics data on a monthly basis: 7 Objective: Measure the Accuracy of the Market Flow calculation of PFV pilot. Validate logic of PFV pilot. Note: To be discussed if calculations will be performed for all Market Coordinated flowgates or for a selected set.