1. © 2008 OSIsoft, Inc. | Company Confidential IT Application Monitoring Using PI AF and Notifications Josiah Thoen - AEP

2. 2 © 2008 OSIsoft, Inc. | Company Confidential AEP – Who We Are – By Assets One of the largest U.S. electricity generators ( ~ 38,000 MWs) with a significant cost advantage in many of our market areas Largest consumer of coal in the Western Hemisphere Operations within four RTO’s PJM SPP ERCOT MISO A leading consumer of natural gas Major wind power developer (#3 in U.S. in 2005) 39,000 miles of transmission 186,000 miles of distribution Coal transportation 7,000 railcars 2,230 barges and 53 towboats 5 million customers

3. 3 © 2008 OSIsoft, Inc. | Company Confidential AEP IT - Service Performance Monitoring AEP IT Assets for over 7 years –Servers Hardware, Operating Systems (Windows and Unix) –Middleware Platforms IIS, BEA WebLogic, etc. –Applications In house custom applications Vendor Applications –BlackBerry Server, Lotus Notes, etc.

4. 4 © 2008 OSIsoft, Inc. | Company Confidential AEP IT - PI Infrastructure PI Server Collective –2 Servers 8 CPU, 8gb RAM, 85gb SAN Drive Space –Software: –PI Server AF & Notification MS SQL Express ACE (Backup) Data Acquisition Node –2 CPU, 2gb RAM, 30gb Local Drive Space –Software: IT Monitor Interfaces Custom PISDK monitoring services Analytics Server –2 CPU, 2gb RAM, 30gb Local Drive Space –Software: ACE (Primary) Stats: –Tags: 94,049 –Modules: 54,076 –AF Elements: ~10,000 –AF Element Templates: ~25 –Notifications: 331 –Data Storage: 1gb every 23days

5. 5 © 2008 OSIsoft, Inc. | Company Confidential AEP IT and PI – the beginning PI IT Monitor Pilot –Customer IT Application Support Teams – Primary IT Operations – Secondary Business Units - Secondary –Purpose Production Server and Application visibility –Result Infrastructure monitoring approach Application Monitoring

6. 6 © 2008 OSIsoft, Inc. | Company Confidential AEP IT and PI - Problems Time Series data is not relational –Convoluted naming conventions and overloaded tag attributes Network > Datacenter > Server > Components > Application > Middleware > Database > and back Maintenance nightmare –Everything is manual –Numerous duplicate displays No Event Management –All this data but no way to knowing when something is broken.

7. 7 © 2008 OSIsoft, Inc. | Company Confidential AEP IT and PI - Solutions Problem: Time Series data is not relational –Solution: AF – Relational Meta Data on top of Time Series data. Problem: Maintenance nightmare –Solution: AF – Template based management of elements and their attributes Problem: No Event Management –Solution: PI Notifications – Template based triggered event delivery

8. 8 © 2008 OSIsoft, Inc. | Company Confidential AF Implementation WebMethods BrokerClient QueueLength monitoring –SoftwareAG WebMethods SOA, B2B messaging platform Pub-Sub Message Brokering –Publisher sends a message to a Broker. –Broker delivers the message to one or more Subscribers of that message. Used in many Tier 1 and 2 application services and business processes –Facilitates communication with external counter parties –Used Internally to send data between applications.

9. 9 © 2008 OSIsoft, Inc. | Company Confidential Business Need Monitor the Broker to make sure that client queues are not getting full. –Publishing clients send messages to the Broker –Broker delivers the message to subscribing clients queue. –Subscribing clients retrieve messages from their queue. –Implications of failure: Client may be unavailable. Broker may be having problems.

10. 10 © 2008 OSIsoft, Inc. | Company Confidential Functional Requirements Store the Broker Client Info data. Alerting: –Warn when queue contains 25+ messages. –Alarm when queue contains 75+ messages. –Alerts must send a Clear message when the alerting condition is gone. Alerting Recipient: –Email notification to those interested. –AEP IT Command Center: 24x7 Support Center. All Events go to Central Event Management System.

11. 11 © 2008 OSIsoft, Inc. | Company Confidential Monitoring Process Data Acquisition – PISDK Custom Monitor –Invoke a service within the WebMethods environment every 60 seconds. Parse data. Do exception testing Send data to PI. –Automatically Add new Broker Clients to AF as they are discovered. –Automatically create tags for Broker Clients if they do not exist.

12. 12 © 2008 OSIsoft, Inc. | Company Confidential AF Element Templates BrokerClientInfo: –Defines the attributes of a BrokerClient as it exists in WebMethods. –Each Attribute references a PI Point. Appname = \\%Server%\BrokerClientInfo_%..\..\Element%_%..\El ement%_%Element%_%Attribute%_Info QueueLength = \\%Server%\BrokerClientInfo_%..\..\Element%_%..\El ement%_%Element%_%Attribute%_Count

13. 13 © 2008 OSIsoft, Inc. | Company Confidential BrokerClientInfo Template

14. 14 © 2008 OSIsoft, Inc. | Company Confidential BrokerClientInfo Element

15. 15 © 2008 OSIsoft, Inc. | Company Confidential AF Element Structure WebMethods –Brokers – » – » – »

16. 16 © 2008 OSIsoft, Inc. | Company Confidential AF Element Structure

17. 17 © 2008 OSIsoft, Inc. | Company Confidential Notification Process - Requirements Notification is based on the QueueLength attribute of a BrokerClientInfo. –Needs to be customizable for each client. –Needs to be disabled for certain clients. –Default Warning = 25. –Default Alarm = 75.

18. 18 © 2008 OSIsoft, Inc. | Company Confidential Notification Process - Template Attribute Template:

19. 19 © 2008 OSIsoft, Inc. | Company Confidential Notification Process – Warn Trigger IF Notification enabled And Warn enabled And QueueLength >= Warn value And QueueLength < Alarm value

20. 20 © 2008 OSIsoft, Inc. | Company Confidential Notification Process – Alarm Trigger IF Notification enabled And Alarm enabled And QueueLength >= Alarm value

21. 21 © 2008 OSIsoft, Inc. | Company Confidential Delivery Channel WebService –Sends Target and Trigger value to our EventManagement Web Service. Target = AFElement Path Value = CRITICAL, WARN, OK Event Management Web Service –Gets additional details from AF using the Target. –Creates an Event Management message and sends it to our Event Management server. –Closes the previous event using the EventId attribute from the AF element. –Sends new EventId back to AF Targets EventId attribute.

22. 22 © 2008 OSIsoft, Inc. | Company Confidential Event Management View BMC Impact Explorer Transpara