Presentation is loading. Please wait.

Presentation is loading. Please wait.

Gray & Reuter FT 2: 1 Dependable Computing Systems Jim Gray Microsoft, Microsoft.com Andreas Reuter International University,

Similar presentations


Presentation on theme: "Gray & Reuter FT 2: 1 Dependable Computing Systems Jim Gray Microsoft, Microsoft.com Andreas Reuter International University,"— Presentation transcript:

1 Gray & Reuter FT 2: 1 Dependable Computing Systems Jim Gray Microsoft, Microsoft.com Andreas Reuter International University, 9:00 11:00 1:30 3:30 7:00 Overview Faults Tolerance T Models Party TP mons Lock Theory Lock Techniq Queues Workflow Log ResMgr CICS & Inet Adv TM Cyberbrick Files &Buffers COM+ Corba Replication Party B-tree Access Paths Groupware Benchmark MonTueWedThurFri

2 Gray & Reuter FT 2: 2 High Speed Network ( 10 Gb/s) The Airplane Rule A two engine airplane has twice as many engine problems. A thousand-engine airplane has thousands of engine problems Fault Tolerance is KEY! Mask and repair faults Internet: Node fails every 2 weeks Vendors: Disk fails every 40 years Here: node fails every 20 minutes disk fails every 2 weeks.

3 Gray & Reuter FT 2: 3 Outline Does fault tolerance work?Does fault tolerance work? General methods to mask faults.General methods to mask faults. Software-fault toleranceSoftware-fault tolerance SummarySummary

4 Gray & Reuter FT 2: 4 DEPENDABILITY: The 3 ITIES Reliability / Integrity: Does the right thing (also large MTTF)Reliability / Integrity: Does the right thing (also large MTTF) Availability: Does it now. (also large MTTF MTTF+MTTR System Availability: If 90% of terminals up & 99% of DB up? (=>89% of transactions are serviced on time).Availability: Does it now. (also large MTTF MTTF+MTTR System Availability: If 90% of terminals up & 99% of DB up? (=>89% of transactions are serviced on time). Holistic vs Reductionist viewHolistic vs Reductionist view Security Integrity / Reliability Availability Security Integrity / Reliability Availability

5 Gray & Reuter FT 2: 5 High Availability System Classes Goal: Build Class 6 Systems System Type Unmanaged Managed Well Managed Fault Tolerant High-Availability Very-High-Availability Ultra-Availability Unavailable (min/year) 50,000 5, Availability 90.% 99.% 99.9% 99.99% % % % Availability Class

6 Gray & Reuter FT 2: 6 Sources of Failures MTTFMTTR Power Failure :2000 hr 1 hr Phone Lines Soft >.1 hr.1 hr Hard4000 hr10 hr Hardware Modules :100,000hr10hr Hardware Modules :100,000hr10hr (many are transient) Software : 1 Bug/1000 Lines Of Code (after vendor-user testing) => Thousands of bugs in System! Most software failures are transient: dump & restart system. Most software failures are transient: dump & restart system. Useful fact: 8,760 hrs/year ~ 10k hr/year

7 Gray & Reuter FT 2: 7 Case Studies - Japan "Survey on Computer Security", Japan Info Dev Corp., March (trans: Eiichi Watanabe). Vendor (hardware and software) 5 Months Application software 9 Months Communications lines1.5 Years Operations 2 Years Environment 2 Years 10 Weeks 10 Weeks 1,383 institutions reported (6/84 - 7/85) 7,517 outages, MTTF ~ 10 weeks, avg duration ~ 90 MINUTES To get 10 year mttf must attack all these problems

8 Gray & Reuter FT 2: 8 Case Studies -Tandem Outage Reports to Vendor Totals: More than 7,000 Customer years More than 30,000 System years More than 80,000 Processor years More than 200,000 Disc Years Summary Tandem EWR Data Customers EWR Customers??267 Outage Customers Systems Processors7,00015,00025,500 Discs16,00046,00074,000 Cases Reports Faults Outages System MTTF8 years20 years21 years Systematic Under-reporting But ratios & trends interesting

9 Gray & Reuter FT 2: 9 Case Studies - Tandem Trends MTTF improved: WOW! Outages per millennium. Shiftfrom Hardware & Maintenance to from 50% to 10% toSoftware (62%) & Operations (15%) NOTE: Systematic under-reporting ofEnvironment Operations errors Application Software

10 Gray & Reuter FT 2: 10 Case Studies - Tandem Trends Reported MTTF by Component SOFTWARE Years HARDWARE Years MAINTENANCE Years OPERATIONS Years ENVIRONMENT Years SYSTEM82021Years Remember Systematic Under-reporting

11 Gray & Reuter FT 2: 11 Summary Current Situation: ~4-year MTTF => Fault Tolerance Works. Hardware is GREAT (maintenance and MTTF). Software masks most hardware faults. Many hidden software outages in operations: New System Software. New Application Software. Utilities. Must make all software ONLINE. Software seems to define a 30-year MTTF ceiling. Reasonable Goal: 100-year MTTF. class 4 today => class 6 tomorrow.

12 Gray & Reuter FT 2: 12 Outline Does fault tolerance work?Does fault tolerance work? General methods to mask faults.General methods to mask faults. Software-fault toleranceSoftware-fault tolerance SummarySummary

13 Gray & Reuter FT 2: 13 Key Idea ArchitectureHardware Faults ArchitectureHardware Faults Software MasksEnvironmental Faults Software MasksEnvironmental Faults DistributionMaintenance DistributionMaintenance Software automates / eliminates operatorsSoftware automates / eliminates operatorsSo, In the limit there are only software & design faults. Software-fault tolerance is the key to dependability. INVENT IT!In the limit there are only software & design faults. Software-fault tolerance is the key to dependability. INVENT IT! }{ } {

14 Gray & Reuter FT 2: 14 Fault Tolerance Techniques FAIL FAST MODULES: work or stop SPARE MODULES : repair time. SPARE MODULES : instant repair time. INDEPENDENT MODULE FAILS by design MTTF Pair ~ MTTF 2 / MTTR ( so want tiny MTTR ) MESSAGE BASED OS: Fault Isolation MESSAGE BASED OS: Fault Isolation software has no shared memory. SESSION-ORIENTED COMM: Reliable messages SESSION-ORIENTED COMM: Reliable messages detect lost/duplicate messages coordinate messages with commit PROCESS PAIRS : PROCESS PAIRS :Mask Hardware & Software Faults TRANSACTIONS: give A.C.I.D. (simple fault model)

15 Gray & Reuter FT 2: 15 Example: the FT Bank Modularity & Repair are KEY: vonNeumann needed 20,000x redundancy in wires and switches vonNeumann needed 20,000x redundancy in wires and switches We use 2x redundancy. Redundant hardware can support peak loads (so not redundant)

16 Gray & Reuter FT 2: 16 Fail-Fast is Good, Repair is Needed Improving either MTTR or MTTF gives benefit Simple redundancy does not help much. FaultDetect Repair Lifecycle of a module fail-fast gives short fault latency High Availability is low UN-Availability is low UN-Availability Unavailability ­ MTTR MTTF MTTF return

17 Gray & Reuter FT 2: 17 Hardware Reliability/Availability (how to make it fail fast) Comparitor Strategies: Duplex: Fail-Fast: fail if either fails (e.g. duplexed cpus) vs Fail-Soft: fail if both fail (e.g. disc, atm,...) Note: in recursive pairs, parent knows which is bad. Triplex:Fail-Fast: fail if 2 fail (triplexed cpus) Fail-Soft: fail if 3 fail (triplexed FailFast cpus) Fail-Soft: fail if 3 fail (triplexed FailFast cpus) Basic FailFast Designs Pair Triplex Recursive Availability Designs Pair & Spare ++ Triple Modular Redundancy

18 Gray & Reuter FT 2: 18 Redundant Designs Have Worse MTTF! THIS IS NOT GOOD: Variance is lower but MTTF is worse Simple redundancy does not improve MTTF (sometimes hurts). This is just an example of the airplane rule. This is just an example of the airplane rule. mttf/1 2 work 1 0 mttf/2 1.5*mttf Duplex: fail soft mttf/1 3 work mttf/3 mttf/2 11/6*mttf TMR: fail soft mttf/1 3 work mttf/2 3/4*mttf Pair & Spare: fail fast 4 work mttf/4 mttf 3 work mttf/2 ~2.1*mttf Pair & Spare: fail soft 4 work mttf/4 mttf/3 2 work 1 0 mttf/2 : Duplex fail fast mttf/1 3 work mttf/3 mttf/2 5/6*mttf TMR: fail fast mttf/1

19 Gray & Reuter FT 2: 19 Add Repair: Get 10 4 Improvement 2 work 1 0 mtbf/2 Duplex: fail fast: mttf/2 mttr mttf/1 3 work mttf/3mttf/2 mttf/1 2 work 1 0 mttr 10 mttf 5 TMR: fail soft Duplex: fail soft 10 mttf 4 3 work mttf/3 TMR: fail fast mttr 10 mttf 4 mttf/2

20 Gray & Reuter FT 2: 20 When To Repair? Chances Of Tolerating A Fault are 1000:1 (class 3) A 1995 study: Processor & Disc Rated At ~ 10khr MTTF Computed Single Observed FailuresDouble Fails Ratio 10k Processor Fails14 Double ~ 1000 : 1 10k Processor Fails14 Double ~ 1000 : 1 40k Disc Fails,26 Double ~ 1000 : 1 40k Disc Fails,26 Double ~ 1000 : 1 Hardware Maintenance: On-Line Maintenance "Works" 999 Times Out Of The chance a duplexed disc will fail during maintenance ~ 1:1000 Risk Is 30x Higher During Maintenance => Do It Off Peak Hour Software Maintenance: Repair Only Virulent Bugs Wait For Next Release To Fix Benign Bugs

21 Gray & Reuter FT 2: 21 OK: So Far Hardware fail-fast is easy Redundancy plus Repair is great (Class 7 availability) Hardware redundancy & repair is via modules. How can we get instant software repair? We Know How To Get Reliable Storage RAID Or Dumps And Transaction Logs. We Know How To Get Available Storage Fail Soft Duplexed Discs (RAID 1...N). ? HOW DO WE GET RELIABLE EXECUTION? ? HOW DO WE GET AVAILABLE EXECUTION?

22 Gray & Reuter FT 2: 22 Outline Does fault tolerance work?Does fault tolerance work? General methods to mask faults.General methods to mask faults. Software-fault toleranceSoftware-fault tolerance SummarySummary

23 Gray & Reuter FT 2: 23 Software Techniques: Learning from Hardware Most outages in Fault Tolerant Systems are SOFTWARE Fault Avoidance Techniques : Good & Correct design. After that: Software Fault Tolerance Techniques: Modularity (isolation, fault containment) Design diversity N-Version Programming: N-different implementations Defensive Programming: Check parameters and data Auditors: Check data structures in background Transactions: to clean up state after a failure Paradox: Need Fail-Fast Software

24 Gray & Reuter FT 2: 24 Fail-Fast and High-Availability Execution Software N-Plexing: Design Diversity N-Version Programming Write the same program N-Times (N > 3) Compare outputs of all programs and take majority vote Process Pairs: Instant restart (repair) Use Defensive programming to make a process fail-fast Have restarted process ready in separate environment Second process takes over if primary faults Transaction mechanism can clean up distributed state if takeover in middle of computation.

25 Gray & Reuter FT 2: 25 What Is MTTF of N-Version Program? First fails after MTTF/N Second fails after MTTF/(N-1),... so MTTF(1/N + 1/(N-1) /2) harmonic series goes to infinity, but VERY slowly for example 100-version programming gives ~4 MTTF of 1-version programming Reduces variance N-Version Programming Needs REPAIR If a program fails, must reset its state from other programs. => programs have common data/state representation. How does this work for Database Systems? Operating Systems? Network Systems? Answer: I dont know.

26 Gray & Reuter FT 2: 26 Why Process Pairs Mask Faults Many Software Faults are Soft After Design Review Code Inspection Alpha Test Beta Test 10k Hrs Of Gamma Test (Production) Most Software Faults Are Transient MVS Functional Recovery Routines 5:1 Tandem Spooler 100:1 Adams >100:1 Terminology: Heisenbug: Works On Retry Bohrbug: Faults Again On Retry Adams: "Optimizing Preventative Service of Software Products", IBM J R&D,28.1,1984 Gray: "Why Do Computers Stop", Tandem TR85.7, 1985 Mourad: "The Reliability of the IBM/XA Operating System", 15 ISFTCS, 1985.

27 Gray & Reuter FT 2: 27 Process Pair Repair Strategy If software fault (bug) is a Bohrbug, then there is no repair wait for the next release or get an emergency bug fix or get a new vendor If software fault is a Heisenbug, then repair is reboot and retry or switch to backup process (instant restart) PROCESS PAIRS Tolerate Hardware Faults PROCESS PAIRS Tolerate Hardware FaultsHeisenbugs Repair time is seconds, could be mili-seconds if time is critical Flavors Of Process Pair: Lockstep Automatic State Checkpointing Delta Checkpointing Persistent

28 Gray & Reuter FT 2: 28 How Takeover Masks Failures Server Resets At Takeover But What About Application State? Database State? Network State? Answer: Use Transactions To Reset State! Abort Transaction If Process Fails. Keeps Network "Up" Keeps System "Up" Reprocesses Some Transactions On Failure

29 Gray & Reuter FT 2: 29 PROCESS PAIRS - SUMMARY Transactions Give Reliability Process Pairs Give Availability Process Pairs Are Expensive & Hard To Program Transactions + Persistent Process Pairs => Fault TolerantSessions & Execution When Tandem Converted To This Style Saved 3x Messages Saved 5x Message Bytes Made Programming Easier

30 Gray & Reuter FT 2: 30 SYSTEM PAIRS FOR HIGH AVAILABILITY Programs, Data, Processes Replicated at two sites. Pair looks like a single system. System becomes logical concept Like Process Pairs: System Pairs. Backup receives transaction log (spooled if backup down). If primary fails or operator Switches, backup offers service.

31 Gray & Reuter FT 2: 31 SYSTEM PAIR CONFIGURATION OPTIONS Primary Backup Mutual Backup: each has 1/2 of Database & Application each has 1/2 of Database & Application Primary Hub: One site acts as backup for many others One site acts as backup for many others In General can be any directed graph Primary Backup Copy Stale replicas: Lazy replication Backup Primary Copy

32 Gray & Reuter FT 2: 32 SYSTEM PAIRS FOR: SOFTWARE MAINTENANCE (Primary) (Backup) V1 V1 Step 1: Both systems are running V1. (Primary) (Backup) V1 V2 Step 2: Backup is cold-loaded as V2. (Backup) (Primary) V1 V2 Step 3: SWITCH to Backup. V2 (Backup) (Primary) V2 Step 4: Backup is cold-loaded as V2 D30. Similar ideas apply to: Database Reorganization Hardware modification (e.g. add discs, processors,...) Hardware maintenance Environmental changes (rewire, new air conditioning) Move primary or backup to new location.

33 Gray & Reuter FT 2: 33 SYSTEM PAIR BENEFITS Protects against ENVIRONMENT: different sites weatherutilitiessabotage Protects against OPERATOR FAILURE: two sites, two sets of operators Protects against MAINTENANCE OUTAGES work on backup software/hardware install/upgrade/move... Protects against HARDWARE FAILURES backup takes over Protects against TRANSIENT SOFTWARE ERRORS Commercial systems: Digital's Remote Transaction Router (RTR) Tandem's Remote Database Facility (RDF) IBM's Cross Recovery XRF( both in same campus) Oracle, Sybase, Informix, Microsoft... replication

34 Gray & Reuter FT 2: 34 SUMMARY FT systems fail for the conventional reasons Environmentmostly Peoplesometimes Softwaremostly HardwareRarely MTTF of FT SYSTEMS ~ 50X conventional ~ years vs weeks ~ years vs weeks Fail-Fast Modules + Reconfiguration + Repair => Good Hardware Fault Tolerance Transactions + Process Pairs => Good Software Fault Tolerance (Repair) System Pairs Hide Many Faults Challenge: Tolerate Human Errors (make system simpler to manage, operate, and maintain) (make system simpler to manage, operate, and maintain)

35 Gray & Reuter FT 2: 35 Key Idea ArchitectureHardware Faults ArchitectureHardware Faults Software MasksEnvironmental Faults Software MasksEnvironmental Faults DistributionMaintenance DistributionMaintenance Software automates / eliminates operatorsSoftware automates / eliminates operatorsSo, In the limit there are only software & design faults. Software-fault tolerance is the key to dependability. INVENT IT!In the limit there are only software & design faults. Software-fault tolerance is the key to dependability. INVENT IT! }{ } {

36 Gray & Reuter FT 2: 36 References Adams, E. (1984). Optimizing Preventative Service of Software Products. IBM Journal of Research and Development. 28(1): Anderson, T. and B. Randell. (1979). Computing Systems Reliability. Garcia-Molina, H. and C. A. Polyzois. (1990). Issues in Disaster Recovery. 35th IEEE Compcon Gray, J. (1986). Why Do Computers Stop and What Can We Do About It. 5th Symposium on Reliability in Distributed Software and Database Systems Gray, J. (1990). A Census of Tandem System Availability between 1985 and IEEE Transactions on Reliability. 39(4): Gray, J. N., Reuter, A. (1993). Transaction Processing Concepts and Techniques. San Mateo, Morgan Kaufmann. Lampson, B. W. (1981). Atomic Transactions. Distributed Systems -- Architecture and Implementation: An Advanced Course. ACM, Springer-Verlag. Laprie, J. C. (1985). Dependable Computing and Fault Tolerance: Concepts and Terminology. 15th FTCS Long, D.D., J. L. Carroll, and C.J. Park (1991). A study of the reliability of Internet sites. Proc 10th Symposium on Reliable Distributed Systems, pp , Pisa, September 1991.


Download ppt "Gray & Reuter FT 2: 1 Dependable Computing Systems Jim Gray Microsoft, Microsoft.com Andreas Reuter International University,"

Similar presentations


Ads by Google