1. Zinsight: Understanding Traces from System Z Steve Heisig, Wim De Pauw IBM T.J. Watson Research heisig@us.ibm.com wim@us.ibm.com WASL 2009

2. © Copyright IBM Corporation 2009 2 System Z OS Level Tracing: Starting point: A problem has been detected by software or human analysts. A System Dump contains a snapshot of the Trace Tables ~2 Million, ~100 byte trace events per second per cpu Syscall level events Optional Branch Trace ~50 event types Used for Problem Diagnosis and Performance Tuning Why can’t I saturate the cpu? Who is consuming the cpu?

3. © Copyright IBM Corporation 2009 3 --------------------------------------------------- SYSTEM TRACE TABLE ---------------------------------------------------- PR ASID WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3 PSACLHS- PSALOCAL PASD SASD TIMESTAMP-LOCAL CP UNIQUE-4 UNIQUE-5 UNIQUE-6 PSACLHSE DATE-10/26/2006 003-0038 007C58B0 SSAR... 0038 0038 03-0038 007C58B0 SSAR... 0038 0038 03-0038 007C58B0 PR... 7 0FA3DACE 00_135BB110 0038 03-0038 007C58B0 PC... 8 0479E3E1 02B00 03-0038 007C58B0 PC... 2 01421F34 00343 IeaMQry 03-0038 007C5250 PR... 2 0479E3E0 7F5F29D8 0038 03-0038 007C5250 PC... 2 0479E34A 02B01 03-0038 007C5250 PR... 0 01421F34 012B8336 001E 03-0038 007C5250 PR... 2 0479E34A 7F5F29D8 0038 03-0038 007C5250 PR... 2 11303529 1191F11A 0038 03-0038 007C5250 PC... 8 11C473E1 02D08 03-0038 007C5250 PR... 2 11C473E1 1191F11A 0038 03-0038 007C5250 EXT 1005 078D2400 988FDEAE 00001005 00000000 00000000 0038 0038 12:22:55.911696 14 00000000 03-0038 007C5250 SVC 18 078D2400 CB0B130C 00000000 4B5EB188 4B0B3FCA DevType 12:22:55.911757 14 03-0038 007C5250 SVC 3C 070C0000 854D512E 00000000 00000100 007FD84C Estae 12:22:55.911757 14 03-0038 007C5250 SVCR 3C 070C0000 854D512E 00000000 00000000 007FD84C 12:22:55.911759 14 03-0038 007C5250 SVC 3C 078C0000 854D5334 00000004 00000084 4B0B3FCA Estae 12:22:55.911759 14 03-0038 007C5250 SVCR 3C 078C0000 854D5334 00000000 00000000 4B0B3FCA 12:22:55.911760 14 03-0038 007C5250 SVCR 18 078D2400 CB0B130C 00000004 00000004 00000000 12:22:55.911760 14 03-0038 007C5250 PC... 8 0 4B0B4B6D 00180205 03-0038 007C5250 SSAR... 002C 002C 03-0038 007C5250 SSAR... 0038 0038 03-0038 007C5250 PR... 7 4B0B4B6D 11FA1BC2 0038 03-0038 007C5250 PC... 8 0 4B0B4B6D 00180205 03-0038 007C5250 SSAR... 002C 002C 03-0038 007C5250 SSAR... 0038 0038 03-0038 007C5250 PC... 7 0 0FA3DACE 0018D901 03-0038 007C5250 SSAR... 002E 002E 03-0038 007C5250 SSAR... 0038 0038 03-0038 007C5250 PC... 7 0479E34A 02B01 Raw Trace Entries

4. © Copyright IBM Corporation 2009 4 Graphical representation, navigation, searching and statistics

5. © Copyright IBM Corporation 2009 5 Zinsight extracts execution flows to reduce the amount of data for the user

6. © Copyright IBM Corporation 2009 6 Finding bottlenecks is easy

7. © Copyright IBM Corporation 2009 7 Multiprocessor analysis shows the impact of MP on your applications

8. © Copyright IBM Corporation 2009 8 Zinsight usage

9. © Copyright IBM Corporation 2009 9 Conclusion: Zinsight as a QA tool Pro-active analysis of performance Problem determination for difficult cases Combination of visualization and pattern extraction to handle large complexities Zinsight technology can be applied to different systems

10. © Copyright IBM Corporation 2009 10 Next Steps: Register and overlay traces from other levels of the system Millicode, Hypervisor, IO Assist processor, Hardware Accelerators, Component Traces Larger Traces Not capable of fitting into memory or being processed deterministically Use Temporal Data Mining techniques, sampling to build up a statistical picture Tweaking Hardware to make Tracing cheaper