1. z/TPF Application experiences and tools By Michael Bahlen Applications Development KLM TPFug 21 April 2009 Lincolnshire IL

2. 21-Mar-2009 z/TPF Experiences and Tools 2 Contents Where did we start? Where are we now? Single Source, re-compile and re-assembly 1 st z/TPF system up Testing and the tools used Keeping the sources in sync between TPF4.1 and z/TPF Most common errors Demo Questions

3. 21-Mar-2009 z/TPF Experiences and Tools 3 Where did we start? At the start of the z/TPF project we were (and still are) running TPF4.1 put 19+ Started Single Source Updates in October 2007 Started re-compile and re-assembly in 1 st quarter 2008 Started z/TPF testing 2 nd quarter 2008 Integration testing started December 2008 Acceptance testing started March 2009

4. 21-Mar-2009 z/TPF Experiences and Tools 4 Where are we now? We are in the middle of the acceptance tests, and have at this moment no NO-GO items open All but 1 test-system are converted to z/TPF Application development is already developing future enhancements for z/TPF Planned cut-over on May 16 th

5. 21-Mar-2009 z/TPF Experiences and Tools 5 Single Source, re-compile and re-assembly Single Source was applied to the software in about 3 months by 4 programmers, testing and loading to TPF4.1 took over a year. Re-compile/re-assembly took 2 weeks, solving the problems of this step took another month. During the process of testing and applying new APARS we had to re-compile/re-assemble several times Note: For information on Single Source and re-compile/re-assembly see my presentations on previous TPFug’s.

6. 21-Mar-2009 z/TPF Experiences and Tools 6 1 st z/TPF system up Around May-June 2008 the 1 st test-system running with z/TPF based on our own builds came up. At first this was a vanilla system, but after a short time it included most of the KLM modifications and all applications Since this project was also a nice way of doing a clean-up of things no longer used, we started of with many snap-63’s. It took a few weeks to solve most of these problems

7. 21-Mar-2009 z/TPF Experiences and Tools 7 Testing and the tools used What did we use for testing before z/TPF? –SST –Debugger –ODF (Online Dump Facility) –ERREPA (Sort of snap-dump) –Stress tests What was available for testing on z/TPF? –Debugger Was this enough? –NO! But we had to start with only the debugger. How to go on from here? –Convert the tools from TPF4.1 to z/TPF! This excluded SST, its not a KLM product and it’s too much work.

8. 21-Mar-2009 z/TPF Experiences and Tools 8 Testing and the tools used So before the tools were converted we had to do with the IBM dump list display (very short). Post processing dumps (time consuming). Use the debugger, which was highly unstable; we had to apply many APARS before it started to be useful. Later ODF and Stress-test came along and testing became easier

9. 21-Mar-2009 z/TPF Experiences and Tools 9 Stress-test What is the KLM stress-test and how does it work? It is based on another tool KLM built into TPF called PMC (Permanent Message Collector) It writes all incoming and outgoing messages to RTA This is the base for stress testing What else is needed for stress-testing? Full capture of the production database and restore onto a test-system. This database, with a special extraction of PMC messages from RTA are the base to perform a stress- test The database is a certain point in time of the production system, sending in the input PMC messages on a high speed basis into this system is a stress-test

10. 21-Mar-2009 z/TPF Experiences and Tools 10 Stress-test Results: –(Snap) Dumps –Other Error messages –Console logging –RTA tape with PMC messages (can be used to compare output on same entry, but is a lot of work) At first we ran a few 100,000 messages and had a lot of things to look into (also stress-test related) Now we are processing between 3mln and 5mln messages with almost no problems.

11. 21-Mar-2009 z/TPF Experiences and Tools 11 Online Dump Facility During the 1 st tests, ODF came along with more and more details of dumps. At 1 st it gave a full report on all (snap) dumps that happened on the test system. Secondly more and more details of the data dumped came into the displays, helping the programmers with direct information of what happened. For example: Displacement of the dump in the causing program, the program itself, data blocks attached to the dump Last but not least the Macro trace, which provides much more information in z/TPF.

12. 21-Mar-2009 z/TPF Experiences and Tools 12 ERREPA ERREPA is KLM’s version of a snap-dump and was created (1989) before the snap-dump existed. This was the 1 st KLM tool completely converted. Some parts had to be re-written to 64-bit code due to the fact that lots of C-programs passed constant strings. Many applications contain ERREPA code, so a lot of these type of reports were investigated. Later with ODF available, we could use the full-dump option on the ERREPA, to get more information out of these error situations.

13. 21-Mar-2009 z/TPF Experiences and Tools 13 Debug view in toolkit Before ODF was available on z/TPF we used the TPF- Toolkit dump viewer. Shows a lot of details about the dump, but since the Toolkit – z/TPF relation was still highly unstable it was causing dumps as well Also the view shows a list of dumps that are only sequence numbers, finding your dump was trial and error.

14. 21-Mar-2009 z/TPF Experiences and Tools 14 Scripts Some tests where performed through scripting (called run-files in KLM) These scripts log every in and output to file, running the script on a TPF4.1 system and a z/TPF system gives 2 output files. Comparing these files resulted in finding several problems.

15. 21-Mar-2009 z/TPF Experiences and Tools 15 Keeping the sources in sync between TPF4.1 and z/TPF To understand how we did this a little look into our development environment is needed. TPF4.1 environment USS based Repository in DB2 under z/OS Idefix Package IdefixGui on PC NFS maestro for PC – USS connection z/TPF environment z/Linux based Repository in DB2 under z/OS z/Idefix Package (ported and extended Idefix) TPFfix on PC NFS maestro for PC – z/Linux connection TPFfix was started by me as a replacement for the IdefixGui as IdefixGui had too many limitations TPFfix stands for: The Professional Fantastic File Interface eXtension (Yes it was started as a joke)

16. 21-Mar-2009 z/TPF Experiences and Tools 16 Keeping the sources in sync between TPF4.1 and z/TPF TPFfix was extended to be able to work with both Idefix versions, so through this interface a coupling was possible. TPFfix is and was extended by many special functions to co-op with transporting TPF source from Idefix to z/Idefix Also functions are available to keep both program bases in sync. How is this done?

17. 21-Mar-2009 z/TPF Experiences and Tools 17 Keeping the sources in sync between TPF4.1 and z/TPF In Idefix every component has a name, version, status, language and platform. In z/Idefix only the version and status are different. Idefix and z/Idefix allow a comment in the metadata. The copy and update functions follow these metadata comments and the version and status in Idefix. As soon as a component is copied, the metadata comment will show ‘Copy41 version:9AOPR’ At each update round, this comment is compared to the ‘new’ version in Idefix, if there is a difference, a copy is made and the metadata comment is also updated.

18. 21-Mar-2009 z/TPF Experiences and Tools 18 Keeping the sources in sync between TPF4.1 and z/TPF To keep track of updates made for the z/TPF system, this comment can also have the following syntax: –Updated!9AOPR –Differs!9AOPR The “Updated!” means that the component has updates specially for z/TPF which are possible on TPF4.1 but not (yet) applied on TPF4.1 The “Differs!” means that the component has updates specially for z/TPF, that are not possible on TPF4.1, or specific conditional code is needed. In these cases, an upgrade from TPF4.1 involves manual labor!

19. 21-Mar-2009 z/TPF Experiences and Tools 19 Most common errors C-language –String arrays too short –Passing wrong type of pointers to assembler (stack programs, KLM specific) Assembler language –Use of R14, R15 after IBM macro’s –Pointers passed to C (pointer to pointer type) –CPROC created for assembler program –No CPROC created for C-program Most curious one: LTR R0,R0 FLIPC D2,D5 BZ SOMEWHERE Of course this was dinasour code

20. 21-Mar-2009 z/TPF Experiences and Tools 20 Demo Demo includes: Some ODF displays Some ERREPA displays Some TPFfix views Note: The following slides will show some examples of the live demo

21. 21-Mar-2009 z/TPF Experiences and Tools 21 ODF examples (report) UODF0001I 18.10.16 Online Dump Error Report * CpuID-D Date Time IS Seq. Type Addr Prog Trc Dups RTA/RTL Volume: C00036 16MAR 17.16.27 02 SN-000065 SNP-S0BDBD110 C0014F YLID 2 RTA/RTL Volume: C00035 16MAR 12.10.30 01 SE-002361 OPR-I000004 E99A00 CP CPSE 1 16MAR 12.10.30 01 SE-002360 OPR-I000004 E99A00 KQT90A KQT9 1 16MAR 11.57.45 02 SE-002359 OPR-SDBCB20 D703FA Y77E1A Y77E 8 16MAR 11.43.04 *** System Up *** 16MAR 11.43.00 01 SE-002358 CTL-I007856 8 RTA/RTL Volume: C00007 16MAR 11.43.14 02 SE-002258 CTL-I0000E2 010000 CP NSTA 16MAR 11.43.14 03 SN-000064 SNP-S0BDBD11A C05812 YLID 6 16MAR 11.43.14 02 SN-****** SNP-S0BDBD110 C0BEA3 YLID 10 16MAR 11.42.48 *** System Down *** End Of Display

22. 21-Mar-2009 z/TPF Experiences and Tools 22 ODF examples (Normal dump display) UODF0007I 18.14.01 Online Mini Dump Display SE-002360 Date - 16MAR Time - 12.10.30 OPR - I000004 Terminal - E99A00 Program - KQT90A Trace - KQT9 Offset - 0000095C Loadset - BASE Object - kqt9 SS - BSS SSU - BSS CpuId - D IS - 01 ADSpace - EVM *** Program Interrupt Code - 0038 - *** ASCE-type Exception *** General Registers At Dumptime *** R0 275CEE0A R1 275CEE3C R2 275CEE0D R3 275CEE17 R4 275CEE1B R5 275CEE25 R6 275CEE2E R7 C9 R8 5514078 R9 25300000 R10 1 R11 1000 R12 2000 R13 2750F9C0 R14 2750F780 R15 F20000002750F9C0 Continued

23. 21-Mar-2009 z/TPF Experiences and Tools 23 ODF examples continuation *** Current PSW At Time Of Dump *** 64PU1 0715000180000000 000000000551495C *** Data Areas *** ECB At 25459000 For 12288 Bytes D0 At 2530C000 For 4096 Bytes D1 At 25306BE0 For 1056 Bytes CSTATIC At 381317000 For 4096 Bytes RCT At 500C7D008 For 36 Bytes RLT At 500C76E50 For 104 Bytes TRACES At 2B800000 For 11648 Bytes REGAREAS At 24675200 For 1408 Bytes HEAP At 275BA100 For 168 Bytes HEAP At 275CEBE0 For 8008 Bytes HEAP At 275D0B28 For 32767 Bytes *** DECB Areas *** DECB FRAME 1 At 25308000 For 4096 Bytes DECB 001 ATTACHED BLOCK At 25312000 For 4096 Bytes DECB 002 ATTACHED BLOCK At 2530A000 For 4096 Bytes End Of Display

24. 21-Mar-2009 z/TPF Experiences and Tools 24 ODF examples (DECB) UODF0007I 18.19.08 Online Mini Dump Display SE-002360 DISPLAY IN USE DECB DATA LEVELS INDEX NAME FARW CBRW DET 001 IDFSLEVELDD@ 00000000 00000000 00000000 25312000 4 000 002 IDFFRW1FKL. C16F8F01 00000000 FFFFFFFF 2530A000 4 000 End Of Display

25. 21-Mar-2009 z/TPF Experiences and Tools 25 ODF examples (DECB/DF block) UODF0007I 18.20.58 Online Mini Dump Display SE-002360 DECB ATTACHED CORE BLOCK FRAME 25308000 DECB 002 BLOCK 2530A000 FFF Bytes TPFDF Formatted Filename - RW1FKL Record Content And Current Pointer *---* 0001 0000001A 2530A01A 000F80 G E N1F1D 1C H W P 000000 0002 00000029 2530A029 003A80 U N B I A T B 1 K L 1 E T 00000000 1 A P P C000000 00 0 9 9 1 E 1 3 0 0 0 0 A 0 E00 00000000 00000000 00000000 0000 0003 00000063 2530A063 004280 U N H 200 00000000 00000000 00000000 H W P R E S 9 6 2 I A 0 9 9 1 E 1 3 0 0 0 0 A 0 E61 E 9 9 A 0 0 00000000 00000000 00000000 0000 0004 000000A5 2530A0A5 005180 P O S0000 1 A40 M U C4040 1 B40 S G4040 S I N4040 0000 A 3 7 9 2 0 0 1 340 404040 B 2 1 0 0 E80 S I N 1 B 2 1 0 000 2680 R C I 1 A00 3 5 5 6 2 W0000 00000000 00 0005 000000F6 2530A0F6 000D80 M S G0000 0000 300 7 0006 00000103 2530A103 002680 R C I 1 A 00 3 5 5 6 2 W00 00000000 00000000 00000000 00 K L00 T K X G H D 0007 00000129 2530A129 001480 U N T 200 00000000 00000000 00000000 0008 0000013D 2530A13D 001580 U N Z 1 0 9 9 1 E 1 3 0 0 0 0 A 0 E End Of Display

26. 21-Mar-2009 z/TPF Experiences and Tools 26 ODF examples (Trace) UODF0007I 18.28.09 Online Mini Dump Display SE-002360 *** Macro Trace *** Psw Macro Prog Obj Disp Lsn Time of macro IS KLMAPPL : 64SM0 KJVE0A kjve.0a F9C 01:57:13.014048 01 KJVE IZTPFDF : 64SM0 CTDF7A ufge.3a 6E0 01:57:13.014047 01 __dbcls 31PU1 BACKC UTDF3J uad0.2a 1AE BASE 01:57:13.014047 01 R0 F0 R1 2750E720 R2 381314522 R3 0 R4 0 R5 0 R6 275CEE25 R7 3DDA14060 R8 2750E3A8 R14 A185A1B2 R15 2750E3A8 31PU1 DECBC UTDF3J ubc0.3a 11FE BASE 01:57:13.014047 01 Type-Release AD-253084A0 R0 0 R1 2750E720 R2 325314000 R3 275D1500 R4 25308020 R5 253084A0 R6 275C5192 R7 3A1887CF0 R8 21883380 R14 253084A0 R15 1 31PU1 EHEAPC UTDF3J ubc0.3a 1192 BASE 01:57:13.014046 01 R0 0 R1 2750E720 R2 325314000 R3 275D1500 R4 25308020 R5 253084A0 R6 275C5192 R7 3A1887CF0 R8 21883380 R14 275C5130 R15 1 31PU1 EHEAPC UTDF3J ubc0.3a 1180 BASE 01:57:13.014045 01 R0 1 R1 2750E720 R2 325314000 R3 275D1500 R4 25308020 R5 253084A0 R6 275C5184 R7 3A1887CF0 R8 21883380 R14 275B8000 R15 1

27. 21-Mar-2009 z/TPF Experiences and Tools 27 ERREPA (report) ERREP OVERVIEW FOR PROJECT : RES DATE: 15MAR 6149 03.06 KOTZ 0604BE 2 6155 11.40 KOTZ 0604BE 1 6156 16.46 T910 06013A 19 6157 17.02 KGRN 060233 1 6158 17.08 KGRM 060567 1 6159 21.21 TIV2 060511 2 6160 22.15 T12D 06012C 5 DATE: 16MAR 6164 16.10 T910 06013A 3 6165 17.03 KGRN 060233 1 6166 17.10 KGRM 060567 1 6167 18.10 TAC4 06054A 1 6168 18.45 KSBA 06FFFD 2

28. 21-Mar-2009 z/TPF Experiences and Tools 28 ERREPA (display) 16MAR09 17.03.43 060233 RES PGM:KGRN SYS:600 SEQNR:6165 000000 *** PROGRAM NESTING AREA *** LVL BASE NSI PAT PBI PGM 0 00000000069C4078 00000000869C41C4 0000000023A50E60 0000 KGRN0A 1 00000000069BC078 00000000869BC898 0000000023A51140 0000 KGRO0A 2 00000000069B4078 00000000869B4412 0000000023A502E0 0000 KGRB0A 3 00000000069A6078 00000000869A62FE 0000000023A505C0 0000 KGRC0A 4 0000000000000000 0000000000006133 0000000000000000 0000 GIF CROSSCHECK NUMBER IN PARTY GROUP DOESN'T MATCH NUMBER IN PARTY OF PNR'S GROUPID: 325077

29. 21-Mar-2009 z/TPF Experiences and Tools 29 TPFfix (normal view)

30. 21-Mar-2009 z/TPF Experiences and Tools 30 TPFfix (actions on components)

31. 21-Mar-2009 z/TPF Experiences and Tools 31 TPFfix (Xref-view)

32. 21-Mar-2009 z/TPF Experiences and Tools 32 TPFfix (label search results)

33. z/TPF Application experiences and tools Questions? michael.bahlen@klm.com