1. Single Point Failure: The case study of RBS CS/SE 6361 FINAL TERM PAPER Shahed Shuman

2. What was the issue? On the day of June 19 2012 The RBS payment processing system suddenly stopped, None of the overnight job running the Bank balance calculation happend, all of the transactions processing were disrupted, ATM machines were not operational, online bank balance was showing incorrect balance! The day before the Incident, an incorrect upgrade was Installed to the RBS CA-7 Software, CA-7 is a job scheduling / workflow automation software package sold by CA technologies. Resulting in total stoppage of Payment processing jobs. RBS moved the CA7 maintenance and support to offshore.RBS also cut down their onshore CA-7 resources As a result,no CA-7 expert was left in RBS who were experienced enough to diagnose the issue and revert the corrupted updates

3. Root cause 1.Proper CA-7 update requirements were not followed by the offshore team. 2.Systems upgrade happened on a weekday without proper testing, it caused major business disruption 3.The issue was not found until it was already in the RBS production server. 4.RBS didn’t have any requirement available to tackle these kinds of technical emergency 5.RBS did not have resources to implement any corrective measure 6.The delay to implement the fix and subsequent media coverage caused the issue to be longer as CA technologies took longer time to avoid another blunder. the error was made when backing out of an upgrade from CA-7 v11.1 to v11.3. The CA-7 upgrade took place at the weekend of 16/17th June and a problem was noticed on Monday which prompted a back-out from the upgrade on Tuesday night. In the back-out, an "inexperienced operator" made the wrong move and the day's data was wiped from the system. This created the backlog(http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_offshoring_software_glitch/)

4. Problem Interdependency Graph

5. Solutions – FRNFR R1)Programmer must follow the update procedure who shall keep track if the proper procedure was followed R2)New Programmers will not make any updates to mission critical systems Who maintains and validates the programmer capabilities R3) A response team should have fixed this issue How fast it must be done? R4) There must be backups to mission critical systems How will the Reliability, security and performance must be maintained

6. Requirements Prioritization AHP R1R2R3R4 R111/342 R23135 R31/41/314 R41/21/51/41 R1R2R3R4 R1 0.2105260.1785710.4848480.230769 R2 0.6315790.5357140.3636360.384615 R3 0.0526320.1785710.1212120.307692 R4 0.1052630.1071430.0303030.076923 Normalized Requirement Priority SUMSUM/4 R1 1.1047140.276179 R2 1.9155440.478886 R3 0.6601070.165027 R4 0.3196320.079908

7. Questions?