1. Cognitive Task Analysis and its Application to Restoring System Security by Robin Podmore, IncSys Frank Greitzer, PNNL

2. Cognitive Task Analysis Background Historically Task Analysis provides a list of tasks and subtasks as well as input, actions, outputs, conditions for performance and proficiency requirements. Because we live in an age of Technological Complexity many jobs involve complex cognitive processing, rather than simply a collection of overt or observable behaviors. Analysts try to develop a model or theory of how experts in a field perform their tasks. This model can then be used to develop system interfaces, decision aids and training programs Cognitive Task Analysis is the extension of traditional task analysis techniques to yield information about the knowledge, thought processes, and goal structures that underlie observable task performance.

3. Respond to Forced Transmission Outage Task List Evaluate system conditions Interpret relay targets, recorder readings, breaker operations, and field observations to identify the location and cause of the event Obtain details of equipment problem including estimated time of return Call out system personnel when appropriate Direct steps needed to sectionalize a transmission line in order to locate fault Implement a plan for restoring the system to a safe operating condition Coordinate response with other affected systems as applicable Implement a plan to restore system security including preparing for the next contingency Complete outage log, and update applicable displays in SCADA/EMS Make all required notifications.

4. CTA Objectives Analysts speak with experts, journeymen, novices or any combination and analyze their behavior to determine: –Concepts and principles, their relationship to each other and their relationship to the tasks –Goals and goal structures, including methods for achieving goals and their triggers –Cognitive skills, rules, strategies and plans –Perceptual learning, pattern recognition, implicit and explicit knowledge –Mental models – how experts represent and run models of the system –Problem models – how experts represent a problem and work within the problem space –How novices move through all of the above in various stages in order to become experts –Difficulties in acquiring domain knowledge and skills –Instructional procedures useful for moving a person from novice to expert

5. Cognitive Task Analysis Framework ODM and Cognitive Task Analysis Mental Model and Story Analysis State and Situation Awareness Level Analysis Object and Action Analysis Conduct and Debrief Scenario – Script from Operator Interview Scenario – Objectives, Base Case, Events, Roles Simulator Environment – Power System Simulator, Control Center Model, Operator Consoles, Projection Map Board

6. Operator Decision Model Real World Short term memory Short term memory Long term memory Long term memory Situation Cues That affect the That let you create That activate Generates Which you Assess by Using your Which you Validate by Using your Time Available? N Mental Simulation Mental Simulation Mental Simulations Mental Simulations Action Script Action Script Story (Patterns) Story (Patterns) Mental Models Mental Models Mental Models Mental Models Y

7. Graphical Contingency Analysis (GCA) Contingency analysis, as a key function in the Energy Management System (EMS), Assesses the ability of the power grid to sustain various combinations of power grid component failures based on state estimates. Presently, most of the industrial software tools installed in control centers use tabular data to represent system status, which is not easy to interpret when the system is stressed

8. NORMAL RESTORATIVE EMERGENCY INSECURE Preventative Action Remedial Action Island Blackout Outage, Load Increase Restoration Action Power System States

9. Contingency Analysis Output - Example

10. Graphical Contingency Analysis Output – Crawford Baker 1 & 2 Outage

11. Contingency Analysis Output

12. Graphical Contingency Analysis Output Farlie – Grange 1 & 2 Outage

13. Scenario Prebrief –System operating at 2:00 PM, Summer peak conditions –Crawford – Oakdale 1 & 2 out for maintenance –Lightning Storm activity in Northeast and Southwest regions Simulation – 20 minutes –Restore system to a secure state based on Contingency Analysis –Operator plays all the roles Debrief – 20 minutes –Walk through recording of screen activity –Interview on operator thought process

14. Participants Experimental Group –Three participants –Using Graphical Contingency Analysis Control Group –Three participants –Using Tabular Contingency Analysis Common Requirements –NERC Certified System Operators –Some prior experience with PowerSimulator and PALCO –Participants blind on Control and Experiment Groups

15. PNNL – Electricity Infrastructure Operations Center Human Factors Test Laboratory

16. Scenario Debrief and Interview

17. Scenario Debrief and Interview Close Up

18. Example of Actions Implemented Decrease Farlie output from 900 MW to 600 MW Increase Doyle output from 357 MW to 550 MW Increase PALCO import from External from 0 MW to 300 MW.

19. Incorrect Critical Assessment Incorrect Critical Assessment Incorrect Critical Assessment Situation Awareness Analysis

20. Security Restoration Requirements Identification of every critical contingency Identification of all critical system constraints Identification of impact of re-dispatch on other critical contingencies and constraints

21. Operator Decision Model - Enhanced Real World Short term memory Short term memory Long term memory Long term memory Situation Cues That affect the That let you create That activate Generates Which you Assess by Using your Which you Validate by Using your Time Available? N Mental and Computer Simulations Mental and Computer Simulations Mental and Computer Simulations Mental and Computer Simulations Action Script Action Script Story (Patterns) Story (Patterns) Mental and Computer Models Mental and Computer Models Mental and Computer Models Mental and Computer Models Y

22. Conclusions Generic PowerSimulator with PALCO system is an effective tool for Cognitive Task Analysis The environment is realistic and creates tension and stress for certified system operators Replay of dynamic screen capture provides basis to capture operator’s thought process. Level of wide area and local situation awareness can be identified at each step of the scenario. Mental models and mental simulations can be identified at each step of the scenario.

23. Cognitive Task Analysis Benefits Improved Curriculum –Need to add Contingency Analysis to PowerSimulator Improved Visualization Tools –Graphical vs. Tabular Contingency Analysis Improved – more specific, precise and constructive feedback to system operators Improved capture of expert knowledge to accelerate the spiral of knowledge Improved ability to learn from mistakes of others Improved power system applications Improved insight into the Operator Decision Model