1. The Use of HDS’s and EMC’s DASD Copy Services at Gary Spencer Worldspan

2. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan Terminology Target volume Source volume Native test system Periodic refreshed test system Daily test system Read/write VM test system

3. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan External Control Unit Copy “Remote copy” Runs external of host Full volume copy “Sync’ed” up Synchronous (semi-sync, adaptive copy) Suspend/split Point in time copy Incremental refresh (copy) Restore capabilities Higher Impact on Host Source Ctl Unit 100 To Host Fiber connection Target Ctl Unit 101 Maybe To Host # 2 xxx yyy

4. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan Internal Control Unit Copy “Local Copy” Runs external of host Very fast – low host impact Full volume copy “sync’ed” up Suspend/split Point in time copy Incremental refresh (copy) Restore capabilities Ctl Unit 100 101 To Host Target volume are “generally” not addressable to the same host as the Source volumes. (IOCP’s are generally used at Worldspan to control access) Maybe To Host # 2 xxx yyy

5. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan Vendor Specific Comments Product Names TrueCopy - (HDS’s remote copy) Symmetrix Remote Data Facility - SRDF (EMC’s remote copy) Shadow Image - (HDS’s local copy) TimeFinder - (EMC’s local copy) Target/Source pairing TrueCopy and Shadow Image may be defined in the host or by the CE in the control unit SRDF source/target pairs may be defined in the host and must be defined by the CE in the control units BIN file. TimeFinder pairs are call Standard volumes/BCVs (Business Continuance Volume) Must be defined in the control unit as “Standard” volumes or as “BCV’s” Source/Target (standard/BCV) pairs are defined in user controlled host tables Copies may be from any “standard” volume to any “BCV”.

6. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan Worldspan’s Three Production TPF systems OSS - Delta Airline’s flight control and flight operations system Deltamatic (RES) - Delta Airline’s reservation system GDS - Worldspan’s agency reservation system Northwest Airline’s reservation system

7. Use of HDS’s Copy Services on Worldspan’s GDS Initial Goal – speed up test system build First use of copy services to build test systems First use of “local” copy

8. Use of HDS’s Copy Services on Worldspan’s GDS The Production GDS System GDS is a 8 way loosely couple TPF system GDS is Worldspan’s agency reservation system GDS is Northwest Airline’s reservation system Statistics GDS has 640 prime mods (1280 active mods) Each physical control unit has 45 or 46 active modules The I/O rate is approximately 250,000 + I/O’s per second Fire Wall 28 HDS 7700E’s 3 LCU’s per CU HDS 7700E

9. Use of HDS’s Copy Services on Worldspan’s GDS Prior GDS Test Environment “Large Test System” Prior GDS test environment- ltst 3 VM Systems Tape Restore Large Test (LTST) 4 7700E’s HDS 7700E Any GDS production processor Large Test System runs read/write to VM during the week – this read/write system is a customer system. Down Time is considered an impact. System is refreshed each Sunday from Saturday capture tapes. Native testing was restricted to weekends before the refresh (some exceptions allowed) The LTST DASD has connectivity to each of the GDS production processors.

10. Use of HDS’s Copy Services on Worldspan’s GDS Current GDS Test Environment Large Test HDS LTST 7700E - 1 4 physical CU’s 2 copies of the production database TrueCopy’s target from the production system rotates each week. Shadow copy the new TrueCopy’ed test system to other address range Unique VSN for each address range HDS 7700E Addr range xx00-xx3f Addr range xx40-xx7f

11. Use of HDS’s Copy Services on Worldspan’s GDS Current GDS Test Environment Large Test Current GDS test environment - ltst 3 VM Systems 4 7700E’s 1 set of shadow volumes HDS 7700E Any GDS production processor Or VM LPAR Fire Wall HDS 7700E 14 7700E’s 1 set of shadow volumes Shadow volumes Rotating TrueCopy Targets Low addresses High addresses Prod volumes Native or VM read/write testing use the non-ltst volumes Step one – shadow production Step two – TrueCopy to LTST Step three – build LTST Step four – IPL new database Step five – shadow LTST

12. Use of HDS’s Copy Services on Worldspan’s GDS Prior GDS Disaster Recovery Prior GDS GDS Disaster Recovery Tape Restore

13. Use of HDS’s Copy Services on Worldspan’s GDS Current GDS Disaster Recovery Current GDS Disaster Recovery Fire Wall 28 HDS 7700E’s 3 LCU’s per CU HDS 7700E A shadow image copy of the GDS database each night. System may be restored from the shadow’ed volumes. Groundwork for a no tape capture shadow volume Production volume

14. Use of HDS’s Copy Services on Worldspan’s GDS TPF Software Use TPF software use First use of HDS’s TPF software for TrueCopy and Shadow Image Originally, Shadow Image and TrueCopy activities were done manually by CE Production control improvements TPF Coverage staff controls the activity Automation is able to run the “copy” procedures. The TPF software eliminated the need for logging tapes during the database split TPF software now controls all TrueCopy and Shadow Image activities for test system builds and DR support Software was new

15. Use of HDS’s Copy Services on Worldspan’s GDS Benefits to GDS GDS payoffs 1.Test System benefits a.Improved LTST down/build time (18 plus hours to less than 2 hours) b.Removing tape restore from LTST build eliminated tape prep/sort/errors time. c.Better and More Native Testing capabilities i.No down time for native testing ii.Native testing at any time – no impact on ltst database iii.Native test time setup takes about 30 minutes (using some of VM’s LPAR’s ) d.Read/write ltst system under VM e.Quick copies of small test systems (8 mod test system) 2.Disaster Recovery benefits. a.Laid groundwork to eliminate tape capture b.Production database can be restored (quickly) from the shadow image copy. 3.New DASD is often formatted using the control unit’s copy services. 4.Proved TrueCopy and Shadow Image technology for TPF use at Worldspan 5.Proved TrueCopy and Shadow Image TPF software controls for use at Worldspan

16. Use of HDS’s Copy Services on Worldspan’s GDS GDS Current Development Current development Working with HDS to use shadow image to split the production database on a nightly basis Working with HDS and with Worldspan’s coverage, production control staffs to improve automation and production control issues Training Worldspan Coverage Staff Possible Future GDS Development Another set of shadow volumes on the large test system (allow to create a daily test system) More shadow volumes on the production database (allow multiple backup copies of the production database) Continue to work towards a capture without tapes.

17. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Most ambitious goals Most work done initially with TPF software EMC provided an onsite consultant First Use of Timefinder SRDF was already in use on OSS

18. Use of EMC’s Copy Services on Worldspan’s Deltamatic System The Production Deltamatic (RES) System Fire Wall Deltamatic is a 6 way loosely coupled TPF system Deltamatic is Delta Airline’s reservation system EMC 8430 Statistics Deltamatic 592 prime mods (1184 active mods) Each EMC 8430 has 74 active modules Each EMC 5830 has 53 or 54 active mods The I/O rate is approximately 115,000 I/O’s per second EMC 8430 EMC 5830 11 EMC 5830’s 2 LCU per CU 8 EMC 8430’s 2 LCU per CU

19. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Prior Deltamatic Test Environment Prior Deltamatic test environment 2 VM Systems PMR Tape Restore Native testing restricted to quarterly DR test weekends No daily test system Intermediate step to replace PMR with 5430’s and manual SRDF Brought in the 8730 to upgrade the testing facilities Deltamatic Test System (RTS) Fire Wall EMC 5830 11 EMC 5830’s 2 LCU per CU EMC 8730 SRDF LINK EMC 5430

20. Use of EMC’s Copy Services on Worldspan’s Deltamatic System The EMC 8730 Control Unit for Three Test Systems EMC 8730 - 1 10 LCU’s 3 copies of the production database Each LCU has three 64 address ranges Unique VSN for each address range 11 production control units (22 LCUs) mapped to 10 test system LCUs. EMC 8730 Addr range xx00-xx3f Addr range xx40-xx7f Addr range xx80-xxbf LCU 1 Addr range xx00-xx3f Addr range xx40-xx7f Addr range xx80-xxbf LCU 10 …

21. Use of EMC’s Copy Services on Worldspan’s Deltamatic System The EMC 8730 Control Unit for Three Test Systems EMC 8730 - 2 First address range in each LCU The daily test system, a base for VM VPAR systems Target volumes for SRDF from the production system (refreshed nightly) TimeFinder’s standard volumes Second address range in each LCU The periodic test system, a base for VM VPAR systems These devices are mirrored TimeFinder’s BCV group 1 Third address range in each LCU The native test system or a read/write test system for VM TimeFinder’s BCV group 2 Addr range xx00-xx3f TimeFinder std volumes Addr range xx40-xx7f TimeFinder BCV group 1 Addr range xx80-xxbf TimeFinder BCV group 2 8730’s LCU x

22. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Current Deltamatic Test Environment Current Deltamatic test environment 2 VM Systems SRDF links Std to std Fire Wall EMC 5830 11 EMC 5830’s 2 LCU per CU 1 std, 2 bcv’s each volume EMC 8730 2 Deltamatic production processors standard BCV 1 BCV 2 Loosely coupled Native testing VM VPARS Daily test system VM VPARS Periodic test system No more tape restore VM read/write test system

23. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Prior Deltamatic Disaster Recovery Prior Deltamatic Disaster Recovery Fire Wall EMC 8430 8 EMC 8430’s 2 LCU per CU … EMC 8430 8 EMC 8430’s 2 LCU per CU SRDF links Continuous, semisync Worldspan Data Center DR Site Break SRDF links for … DASD for quarterly DR test exercise Quarterly native test DASD Selective database recovery from capture or logging tapes

24. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Current Deltamatic Disaster Recovery Current Deltamatic Disaster Recovery Production EMC 5830 and 8430 Standard volumes (production active volumes) BCV 1 Copy to from standard BCV 2 Copy to from standard Alternate TimeFinder copy from standard volumes to BCVs

25. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Current Deltamatic Disaster Recovery Current Deltamatic Disaster Recovery Fire Wall EMC 8430 8 EMC 8430’s 2 LCU per CU EMC 8430 8 EMC 8430’s 2 LCU per CU SRDF links Continuous, semisync Worldspan Data Center DR Site Control DR SRDF stop/start with TPF software In production, we can “capture” the database rotating between two BCV’s. At the DR site, we can “capture” the database rotating between two BCV’s. Selective database recovery from the daily test system Production system may be restored from the BCV’s EMC 5830 11 EMC 5830’s 2 LCU per CU Each standard has 2 BCV’s

26. Use of EMC’s Copy Services on Worldspan’s Deltamatic System TPF Software Use TPF software use First use of EMC’s TimeFinder software EMC’s TPF SRDF software had been used on OSS Similar advantages as the GDS TPF Coverage Staff controls the activity, software controls are automated No tapes (including logging tapes) required for test system refresh The TPF software controls all SRDF and TimeFinder activities for test system builds and DR activities Working to use remote TimeFinder for DR copies. TimeFinder must be started with host software (not a routine task for a CE) The software can be used to restore the production database from the BCVs..

27. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Benefits to Deltamatic Deltamatic payoffs 1.Test System benefits a.Native Testing capabilities on request (no longer must wait for a quarterly DR test) b.Daily Test System i.Improved problem analysis from 6 wk system c.A read/write VM system for high write testing d.Periodic Test System build much easier, faster (no tapes) 2.Disaster Recovery benefits. a.Coverage controls DR SRDF activity with TPF software b.Laid groundwork to eliminate tape capture c.Have multiple days of copies of the database on DASD d.Daily Test System i.Selective database restore ii.Eliminate the need for continual logging tapes (although not done yet) 3.Proved TimeFinder and remote TimeFinder technology for TPF use at Worldspan’s 4.Proved EMC’s TPF TimeFinder software for use at Worldspan’s

28. Use of EMC’s Copy Services on Worldspan’s Deltamatic System Deltamatic Current Development Current and future development Completing the implementation in using BCV’s for database capture/restore Working with EMC to complete implementation of remote TimeFinder use Working with EMC to use TimeFinder in the production system Working with EMC to improve automation and production control issues Training Worldspan Coverage Staff Work towards a no tape capture solution More BCV’s for more “back” days of capture Possible Future Deltamatic Development

29. Use of EMC’s Copy Services on Worldspan’s OSS Limited Goals due to older Technology used for this system EMC 5230’s – no TimeFinder Primary goal was to set up a native test environment Initial use of EMC’s TPF software Work similar to the Deltamatic work

30. Use of EMC’s Copy Services on Worldspan’s OSS The Production OSS System Split Site (Disaster Recovery Location) (Worldspan Data Center) OSS is a Uni-Processor TPF system OSS is Delta Airline’s flight control and flight operations system EMC 5230 Statistics OSS has 72 prime mods (144 active mods) Each physical control unit has 36 active modules The I/O rate is approximately 3500 I/O’s per second 4 EMC 52300’s 2 LCU per CU

31. Use of EMC’s Copy Services on Worldspan’s OSS Benefits to OSS OSS payoffs Similar to the Deltamatic benefits. By adding a new control unit for testing – and using EMC’s copy services - 1.Test system has native testing and a daily test system 2.Disaster Recovery gained from the daily test system and easier setup for DR testing. Also, the groundwork was started to help eliminate tape capture 3.Proved SRDF technology and TPF software controls for TPF use at Worldpan. With new technology - OSS could make use of TimeFinder in similar ways as Deltamatic.

32. Use of EMC’s Copy Services on Worldspan’s OSS Recap of the OSS work Split Site (Disaster Recovery Location) (Worldspan Datacenter) recap of oss work Production OSS Processor EMC 5230 DR OSS Processor RES Processor PMR EMC 5230 SRDF links Tape Restore 2 VM Systems Daily test system Native test system SRDF links Manual process AMOD UP DR duplicate DASD DASD for DR test exercise Quarterly native test DASD

33. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan Implementation Experiences Common Experiences between Vendors or between TPF Systems  Production I/O rate impacted copy services  Most copy service problems were found during production use  Stalled mods  Had to switch (or will switch) to use local copy only on production. Then remote copy from that local production copy.  The TPF software commands were done manually until the concept was proven. At that time the software commands to control the copy activity were automated  Many Automation/Production Control issues  Coverage education. Specifically awareness that this activity runs outside of TPF ’s control.  CE’s education.  Outstanding Vendor and Vendor’s programmer support. Code was high quality - errors found were generally due to configuration issues. Vendor’s would provide off hour telephone or onsite support.  Building source/target tables required vendor help.  OCO code delayed testing, especially during production testing (examples : waiting for programmers to resolve problems, waiting for new code).

34. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan Implementation Experiences Experiences with the Delta Systems and EMC  Need for rebuilding BIN files during implementation and testing  EMC provided a full time onsite consultant for the TPF software implementation and to help set up the use of TimeFinder Experiences with the GDS and HDS  TPF software was new

35. The Use of HDS’s and EMC’s DASD Copy Services at Worldspan What Worldspan Gained Test Systems Easier access to native testing on all three systems A “daily” test system on both of the Delta Airline TPF systems Faster “Periodic” Test system build for GDS and Deltamatic Test systems built using no tapes (reduced tape activities) Less test system down time (“or” test systems are more available) Daily test systems, native test systems are increasing the quality of testing Disaster Recovery Easier (and faster) to set up a DR test for Delta Airlines. Quick access to a 24 hour old database for selective database restores All three systems have a DASD copy of the database to restore from Started Groundwork for a no tape capture More ? As the programming groups become experienced using these test environments, we have seen them exploit this tool and continue to come up with new ways to use these systems. Today on GDS we are seeing extensive recoup/capture testing.