1. Maintenance Windows CPTE 433 John Beckett

2. Phases of Maint. Window Preparation –Schedule –Pick flight director –Prepare change proposals –Build master plan, notify Execution –Notify people –Disable access –Shutdown sequence –Execute plan –Perform testing Resolution –Announce completion –Enable access –Have a visible presence –Be prepared for problems This is Table 20.1 Page 474

3. How To Sell Benefit to Organization –Reduce disruption (unplanned downtime) –Increase planned downtime ? –Perhaps: With a planning program in process, you can achieve less downtime of all kinds! No sale: benefit to SAs

4. Know Your Business Cycle “McBee Baking” - bread –Product has limited shelf-life –Tight constraints on profitability Spoilage is an issue –1,470 reps in the field Entering orders daily –Total retail $2.2 million Factory value $1.2 million

5. 1,470 Orders Per Day! 3PM 6PM 9PM Traffic

6. Maintenance Window Time 3PM 6PM 9PM Traffic Maintenance Window

7. The Flight Director “On top” of the plan –Knows the plan –Creates the plan –Call shots –Obtains power “legitimately” at first –Maintains power by earning respect Accepts proposals for changes to be made Chairs decisions to make changes

8. Communications (During) The components involved in the change may not be in the same room. Will your communications be disrupted by the changes you are making? –Or someone else’s maint window? Radios –Check reception Cell phones - increasingly the choice –May not reach server room VoIP 802.11 –Do they depend on the infrastructure you are rearranging?

9. The Dry Run Write up your procedures. Go through them one step at a time. –Taking notes Make sure cables you plan to use… –Are where you need them –Are long enough –Have the right TE/CE mode –Have the right connectors (SC versus ST)

10. Be There During planning phase –Planning is an agreement. If you weren’t there for it, you weren’t a “player.” During dress rehearsal During event, of course At critical points…

11. When to Be There 3PM 6PM 9PM Traffic Maintenance Window Traffic Begins Heavy Traffic Next Morning

12. Post-Mortem All players should be involved Techie talk in one meeting Clients in a second meeting –Formal minutes Follow-up next time around to see if your response to the challenges worked –…Or what problems those “cures” caused

13. Co-Pilot Heir apparent to the flight director’s slot Increasingly involved in “actually doing it” Ready to take over when the flight director’s talents are recognized by management –Or if flight director is sick –Or if flight director needs to use technical skills for solving a non-management problem

14. Partial Availability? Take it all the way down: + “Cleaner” process + Expectations more likely to be met - You don’t get as “live” a test Part-way: Some availability - Not as “clean” - Not as understandable + You get user feedback all along the way

15. Change Proposal Questions What changes are going to be made? What machines will you be working on? What are the pre-maintenance window dependencies, and due dates? What needs to be up for the change to happen? (…and who needs to be there for support and/or testing) What will be affected by the change? Who is performing the work How long, and what additional helpers needed? What test procedures? What equipment & systems? What is the back-out plan and how long will that take?

16. How Long Was That? Keep records of downtime required for specific tasks. Use those numbers to predict the future. Use those records to refine your process. Log information is very useful for communicating with the boss.

17. Conclusion Plan your work Work your plan Analyze the results

18. Paper: The Cost of Downtime by David Patterson

19. Citation A Simple Way to Estimate the Cost of Downtime David A. Patterson – University of California at Berkeley Presented at 2002 LISA XVI – November 3-8, 2002 – Philadelphia, PA pp. 185-188

20. Thesis The cost of downtime can be calculated. Uses direct and indirect costs.

21. Cost of One Hour of Downtime Brokerage Operations$6,450,000 Credit card auth2,600,000 eBay225,000 Amazon.com180,000 Package shipping service150,000 Home shopping channel113,000 Catalog sales center90,000 Airline reservation center89,000 Cellular service activation41,000 On-line network fees25,000 ATM service fees14,000 Source: Contingency Planning Research, 2000

22. What Causes Failures? Public Switched Telephone Network Overload – 11% Software – 8% Hardware – 22% Operator – 59% Internet sites Overload – 0% Software – 34% Hardware – 15% Operator – 51%

23. Points Made Downtime is expensive. The greatest single contributor to downtime is operator error. Kill that problem, and downtime is cut by half. Operator-induced downtime is probably under- reported because the operators are busy un- doing their mistakes instead of logging downtime. –Besides, the system didn’t go down on its own anyhow!

24. Points Missed Your customers are only a click away from the competition. If you’re down, your customers are GONE. To get them back: –You’ll have to bribe them (affects profit margin) or –The competition will have to make a mistake Once they’re gone, your customers may very well never come back because they are happy wherever they went. Hence: Downtime contributes to loss of market share, which has long-term effects.

25. The Moat Around Your Castle What keeps your customers from fleeing? –Trust in you that makes them patient –They know their way around your Website –Conversion cost (maybe they’re used to working with you) –The competition may be worse –Each of these will keep customers in the short term, but presents an opportunity for the competition to steal your thunder

26. The Bottom Line Downtime probably costs you a lot more than you recognize. Downtime impacts you to the extent that your company depends on the system. Dependence tends to grow without being noticed, so it’s easy to stay in denial about it.

27. Step 1 1.Measure your downtime. 2.Identify the cause of each downtime. 3.Work on the problem. 4.Go To step 1.