Cognitive Task Analysis and its Application to Restoring System Security by Robin Podmore, IncSys Frank Greitzer, PNNL.

Slides:



Advertisements
Similar presentations
Curriculum Development and Course Design
Advertisements

Reliability Software1 Reliability Software Minimum requirements & Best practices Frank Macedo - FERC Technical Conference July 14, 2004.
CHANGING ROLES OF THE DIAGNOSTICIAN Consultants to being part of an Early Intervention Team.
Team Training Dr. Steve Training & Development INP6325 * Adapted from Salas & Canon-Bowers.
Chapter 15 Application of Computer Simulation and Modeling.
INDUSTRIAL & SYSTEMS ENGINEERING
Formative and Summative Evaluations
1 Keeping Track: Coordinating Multiple Representations in Programming Pablo Romero, Benedict du Boulay & Rudi Lutz IDEAs Lab, Human Centred Technology.
Change is a Process Organizational Stages Individual Stages (ADKAR) Business Need Concept and Design Implementation Post-Implementation Awareness Desire.
Course Wrap-Up IS 485, Professor Matt Thatcher. 2 C.J. Minard ( )
3 Chapter Needs Assessment.
Principles of High Quality Assessment
RiversidePublicUtilities.com Arts & Innovation RiversidePublicUtilities.com Challenges and Solutions for Large-Scale PV Integration on RPU’s Distribution.
Task analysis 1 © Copyright De Montfort University 1998 All Rights Reserved Task Analysis Preece et al Chapter 7.
Chapter 10.  Basic Functions  Insert Graphics, Audio/Video  Add Text  Create Links  Capture Brainstormed Ideas  Generate Outline  Organize Graphics,
Simulation Exercises Overview Activities designed to assess, enhance and evaluate preparedness.
SCHINDLER Sales Force Training Needs Assessment and Development Project Michael Yurchuk Sales Training Manager, Schindler Elevator Richard Dapra Ph.D.,
Using Situational awareness and decision making
MULTIMEDIA SYSTEM.
2001 South First Street Champaign, Illinois (217) Davis Power Consultants Strategic Location of Renewable Generation Based on Grid Reliability.
MSF Testing Introduction Functional Testing Performance Testing.
Y. Rong June 2008 Modified in Feb  Industrial leaders  Initiation of a project (any project)  Innovative way to do: NABC ◦ Need analysis ◦ Approach.
Chapter 3 Needs Assessment
Course Overview Course Requirements Knowledge and Skills Student Expectations.
ERCOT Compliance Audits Robert Potts Sr. Reliability Analyst March 23, 2005.
Introduction to Systems Analysis and Design Trisha Cummings.
TESTING STRATEGY Requires a focus because there are many possible test areas and different types of testing available for each one of those areas. Because.
TRAINING AND DEVELOPMENT. WHAT IS TRAINING ? The acquisition of knowledge and skills for present tasks. A tool to help individuals contribute to the organizations.
S/W Project Management Software Process Models. Objectives To understand  Software process and process models, including the main characteristics of.
Human-Centered Information Visualization Jiajie Zhang, Kathy Johnson, Jack Smith University of Texas at Houston Jane Malin NASA Johnson Space Center July.
Chapter 7: Memory and Training Slide Template. WORKING MEMORY.
McGraw-Hill/Irwin ©2009 The McGraw-Hill Companies, All Rights Reserved Marketing Research, Primary Data, Secondary Data, Qualitative Research, Quantitative.
Risk Management - the process of identifying and controlling hazards to protect the force.  It’s five steps represent a logical thought process from.
Business Analysis and Essential Competencies
고려대학교 산업경영공학과 IMEN 315 인간공학 18. Selection and Training.
Conducting Scenario-based Simulations Sandrijn van Schaik, MD PhD Jenna Shaw-Batista, CNM PhD Cynthia Belew, CNM MS UCSF Center for Faculty Educators &
Chapter 5 Job Analysis.
A COMPETENCY APPROACH TO HUMAN RESOURCE MANAGEMENT
Development of Indicators for Integrated System Validation Leena Norros & Maaria Nuutinen & Paula Savioja VTT Industrial Systems: Work, Organisation and.
1 Issues in Assessment in Higher Education: Science Higher Education Forum on Scientific Competencies Medellin-Colombia Nov 2-4, 2005 Dr Hans Wagemaker.
Developing Structured Activity Tools. Aligning assessment methods and tools Often used where real work evidence not available / observable Method: Structured.
Social Science Integration. Hurricane Forecasts as an Element of Generalized Risk Management System Public Emergency Planners Forecast Products Public.
INTERACTIVE ANALYSIS OF COMPUTER CRIMES PRESENTED FOR CS-689 ON 10/12/2000 BY NAGAKALYANA ESKALA.
1 Introduction to Software Engineering Lecture 1.
Application and Implementation of State Estimator at Idaho Power Company S. Kincic and M. Papic.
Advanced Controls and Sensors David G. Hansen. Advanced Controls and Sensors Planning Process.
Using a Logic Model to Plan and Evaluate Your Technology Leadership Development Program Chad Green, Program Analyst Lynn McNally, Technology Resource Supervisor.
Training and Development Prof R K Singh AIMA CME.
 For years, corporations have used computer-based simulations with employee-training programs, augmenting traditional on-the-job training with virtual.
Literacy and Numeracy Benchmarks Prepared by SAPDC Learning Facilitator Team.
U SER I NTERFACE L ABORATORY Situation Awareness a state of knowledge, from the processes used to achieve that state (situation assessment) not encompass.
Training Aids Ice breaker. Previously nominated student to introduce an ice breaker Introduction What type of training aids are there. Last lesson. Learning.
Software Quality Assurance SOFTWARE DEFECT. Defect Repair Defect Repair is a process of repairing the defective part or replacing it, as needed. For example,
Chapter 5:User Interface Design Concepts Of UI Interface Model Internal an External Design Evaluation Interaction Information Display Software.
T EST T OOLS U NIT VI This unit contains the overview of the test tools. Also prerequisites for applying these tools, tools selection and implementation.
Design Evaluation Overview Introduction Model for Interface Design Evaluation Types of Evaluation –Conceptual Design –Usability –Learning Outcome.
BRAIN SCAN  Brain scan is an interactive quiz for use as a revision/ learning reinforcement tool that accompanies the theory package.  To answer a question.
CHAPTER 5 Transfer of Training.
1 Design and evaluation methods: Objectives n Design life cycle: HF input and neglect n Levels of system design: Going beyond the interface n Sources of.
3 Chapter Needs Assessment.
SIE 515 Design Evaluation Lecture 7.
Fundamentals of Information Systems, Sixth Edition
ISO New England System R&D Needs
SKILL ASSESSMENT OF SOFTWARE TESTERS Case Study
The Extensible Tool-chain for Evaluation of Architectural Models
Strategies and Techniques
CHAPTER 9 (part a) BASIC INFORMATION SYSTEMS CONCEPTS
TAKS, Inquiry, Standards and Assessment
Some Important Skills Every Software Testers Should Have
18. Selection and Training
Presentation transcript:

Cognitive Task Analysis and its Application to Restoring System Security by Robin Podmore, IncSys Frank Greitzer, PNNL

Cognitive Task Analysis Background Historically Task Analysis provides a list of tasks and subtasks as well as input, actions, outputs, conditions for performance and proficiency requirements. Because we live in an age of Technological Complexity many jobs involve complex cognitive processing, rather than simply a collection of overt or observable behaviors. Analysts try to develop a model or theory of how experts in a field perform their tasks. This model can then be used to develop system interfaces, decision aids and training programs Cognitive Task Analysis is the extension of traditional task analysis techniques to yield information about the knowledge, thought processes, and goal structures that underlie observable task performance.

Respond to Forced Transmission Outage Task List Evaluate system conditions Interpret relay targets, recorder readings, breaker operations, and field observations to identify the location and cause of the event Obtain details of equipment problem including estimated time of return Call out system personnel when appropriate Direct steps needed to sectionalize a transmission line in order to locate fault Implement a plan for restoring the system to a safe operating condition Coordinate response with other affected systems as applicable Implement a plan to restore system security including preparing for the next contingency Complete outage log, and update applicable displays in SCADA/EMS Make all required notifications.

CTA Objectives Analysts speak with experts, journeymen, novices or any combination and analyze their behavior to determine: –Concepts and principles, their relationship to each other and their relationship to the tasks –Goals and goal structures, including methods for achieving goals and their triggers –Cognitive skills, rules, strategies and plans –Perceptual learning, pattern recognition, implicit and explicit knowledge –Mental models – how experts represent and run models of the system –Problem models – how experts represent a problem and work within the problem space –How novices move through all of the above in various stages in order to become experts –Difficulties in acquiring domain knowledge and skills –Instructional procedures useful for moving a person from novice to expert

Cognitive Task Analysis Framework ODM and Cognitive Task Analysis Mental Model and Story Analysis State and Situation Awareness Level Analysis Object and Action Analysis Conduct and Debrief Scenario – Script from Operator Interview Scenario – Objectives, Base Case, Events, Roles Simulator Environment – Power System Simulator, Control Center Model, Operator Consoles, Projection Map Board

Operator Decision Model Real World Short term memory Short term memory Long term memory Long term memory Situation Cues That affect the That let you create That activate Generates Which you Assess by Using your Which you Validate by Using your Time Available? N Mental Simulation Mental Simulation Mental Simulations Mental Simulations Action Script Action Script Story (Patterns) Story (Patterns) Mental Models Mental Models Mental Models Mental Models Y

Graphical Contingency Analysis (GCA) Contingency analysis, as a key function in the Energy Management System (EMS), Assesses the ability of the power grid to sustain various combinations of power grid component failures based on state estimates. Presently, most of the industrial software tools installed in control centers use tabular data to represent system status, which is not easy to interpret when the system is stressed

NORMAL RESTORATIVE EMERGENCY INSECURE Preventative Action Remedial Action Island Blackout Outage, Load Increase Restoration Action Power System States

Contingency Analysis Output - Example

Graphical Contingency Analysis Output – Crawford Baker 1 & 2 Outage

Contingency Analysis Output

Graphical Contingency Analysis Output Farlie – Grange 1 & 2 Outage

Scenario Prebrief –System operating at 2:00 PM, Summer peak conditions –Crawford – Oakdale 1 & 2 out for maintenance –Lightning Storm activity in Northeast and Southwest regions Simulation – 20 minutes –Restore system to a secure state based on Contingency Analysis –Operator plays all the roles Debrief – 20 minutes –Walk through recording of screen activity –Interview on operator thought process

Participants Experimental Group –Three participants –Using Graphical Contingency Analysis Control Group –Three participants –Using Tabular Contingency Analysis Common Requirements –NERC Certified System Operators –Some prior experience with PowerSimulator and PALCO –Participants blind on Control and Experiment Groups

PNNL – Electricity Infrastructure Operations Center Human Factors Test Laboratory

Scenario Debrief and Interview

Scenario Debrief and Interview Close Up

Example of Actions Implemented Decrease Farlie output from 900 MW to 600 MW Increase Doyle output from 357 MW to 550 MW Increase PALCO import from External from 0 MW to 300 MW.

Incorrect Critical Assessment Incorrect Critical Assessment Incorrect Critical Assessment Situation Awareness Analysis

Security Restoration Requirements Identification of every critical contingency Identification of all critical system constraints Identification of impact of re-dispatch on other critical contingencies and constraints

Operator Decision Model - Enhanced Real World Short term memory Short term memory Long term memory Long term memory Situation Cues That affect the That let you create That activate Generates Which you Assess by Using your Which you Validate by Using your Time Available? N Mental and Computer Simulations Mental and Computer Simulations Mental and Computer Simulations Mental and Computer Simulations Action Script Action Script Story (Patterns) Story (Patterns) Mental and Computer Models Mental and Computer Models Mental and Computer Models Mental and Computer Models Y

Conclusions Generic PowerSimulator with PALCO system is an effective tool for Cognitive Task Analysis The environment is realistic and creates tension and stress for certified system operators Replay of dynamic screen capture provides basis to capture operator’s thought process. Level of wide area and local situation awareness can be identified at each step of the scenario. Mental models and mental simulations can be identified at each step of the scenario.

Cognitive Task Analysis Benefits Improved Curriculum –Need to add Contingency Analysis to PowerSimulator Improved Visualization Tools –Graphical vs. Tabular Contingency Analysis Improved – more specific, precise and constructive feedback to system operators Improved capture of expert knowledge to accelerate the spiral of knowledge Improved ability to learn from mistakes of others Improved power system applications Improved insight into the Operator Decision Model